Phi 2.0

Phi 2.0 Model Details

Phi-2 Description: It's a 2.7 billion parameter Transformer model trained with various NLP synthetic texts and filtered websites.

It's designed to address common sense, language understanding, and logical reasoning.

Intention: The model is not fine-tuned with reinforcement learning from human feedback. It is intended for research, especially in safety challenges like reducing toxicity, understanding biases, etc.

Formats: Phi-2 is optimised for QA format, chat format, and code format.

Sample Code

• Execution Modes: Four modes are detailed, depending on whether you use CUDA or CPU and the floating-point precision (FP16 or FP32).

• Recommended Usage: A Python example is provided to demonstrate how to load and use the model, including tokenizing inputs and generating outputs.

The generic configuration file for training Phi 2.0 from axolotl

Source: axolotl/examples/phi/phi2-ft.yml

base_model: microsoft/phi-2
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: garage-bAInd/Open-Platypus
    type: alpaca

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./phi-sft-out

sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true

adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_torch
adam_beta2: 0.95
adam_epsilon: 0.00001
max_grad_norm: 1.0
lr_scheduler: cosine
learning_rate: 0.000003

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: True
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
resize_token_embeddings_to_32x: true
special_tokens:
  pad_token: "<|endoftext|>"

We will now continue to populate the training configuration file.