# Training Configuration

Last updated

This documentation is for the Axolotl community

Last updated

Field Name | Explanation |
---|---|

gradient_accumulation_steps

`gradient_accumulation_steps`

controls the number of forward and backward passes to skip before updating the model's weights. It is useful when the batch size is too large to fit into GPU memory at once. Gradients are accumulated over the specified number of steps before performing weight updates.

micro_batch_size

`micro_batch_size`

specifies the number of samples included in each batch sent to each GPU. It helps in determining the mini-batch size for training.

eval_batch_size

`eval_batch_size`

sets the batch size for evaluation. It determines how many samples are processed in each evaluation step.

num_epochs

`num_epochs`

defines the number of training epochs, which represent the number of times the entire dataset is processed during training.

warmup_steps

`warmup_steps`

specifies the number of warm-up steps for the learning rate scheduler. During the warm-up phase, the learning rate gradually increases to its full value.

learning_rate

`learning_rate`

sets the initial learning rate for training. It is a critical hyperparameter that determines the step size for weight updates during optimization.

lr_quadratic_warmup

`lr_quadratic_warmup`

is a field related to learning rate scheduling. It is used to specify a quadratic warm-up schedule for the learning rate, which can be beneficial for certain training scenarios.

logging_steps

`logging_steps`

sets the frequency at which training logs are generated. It controls how often training progress is reported.

save_strategy

`save_strategy`

determines when model checkpoints are saved during training. Setting it to 'no' skips checkpoint saves, while other options control the timing of saves.

save_steps

`save_steps`

specifies the frequency at which model checkpoints are saved. You can leave it empty to save at each epoch or specify a different number of steps.

eval_steps

`eval_steps`

controls the frequency of model evaluation during training. It can be specified as an integer for every N steps or as a decimal for a fraction of total steps.

save_total_limit

`save_total_limit`

limits the maximum number of checkpoints saved at a time. Older checkpoints are deleted to keep the total number within this limit.

max_steps

`max_steps`

defines the maximum number of iterations to train for. It takes precedence over `num_epochs`

. For example, if you set `max_steps`

to 100, the training will stop after 100 steps, regardless of the number of epochs.

eval_table_size

`eval_table_size`

specifies the approximate number of predictions sent to wandb (Weights and Biases) depending on the batch size. This field is enabled above 0 and is useful for tracking evaluation metrics.

eval_table_max_new_tokens

`eval_table_max_new_tokens`

sets the total number of tokens generated for predictions sent to wandb. It helps control the amount of data sent for monitoring.

save_safetensors

When specified, `save_safetensors`

indicates saving the model as safetensors, requiring the safetensors package for compatibility.

train_on_inputs

`train_on_inputs`

determines whether to mask out or include the human's prompt from the training labels. Setting it to 'false' omits the prompt from training.

group_by_length

When set to 'true,' `group_by_length`

groups data with similar sequence lengths together to minimize padding. This can help improve training efficiency but may lead to an oscillating training loss.

gradient_checkpointing

`gradient_checkpointing`

controls whether to use gradient checkpointing, a technique that can reduce memory consumption during training. When enabled, it trades off computation for memory.

early_stopping_patience

`early_stopping_patience`

determines when to stop training if evaluation losses increase consecutively for a specified number of times. It helps prevent overfitting.

lr_scheduler

`lr_scheduler`

specifies the learning rate scheduler to use during training. Options include 'one_cycle,' 'log_sweep,' or leaving it empty for cosine scheduling.

lr_scheduler_kwargs

`lr_scheduler_kwargs`

can be used to provide additional arguments to the learning rate scheduler, depending on the chosen scheduler type.

lr_div_factor

For 'one_cycle' optimizer, `lr_div_factor`

determines the learning rate division factor during the one-cycle learning rate schedule.

log_sweep_min_lr

For 'log_sweep' optimizer, `log_sweep_min_lr`

sets the minimum learning rate for the logarithmic learning rate sweep.

log_sweep_max_lr

For 'log_sweep' optimizer, `log_sweep_max_lr`

sets the maximum learning rate for the logarithmic learning rate sweep.

optimizer

`optimizer`

specifies the optimizer to use for training. There are various optimizer options available, and the choice depends on the model and use case.

weight_decay

`weight_decay`

determines the weight decay applied during optimization. It is a regularization term that prevents overfitting by penalizing large model weights.

adam_beta1

For 'adamw' optimizer, `adam_beta1`

sets the beta1 hyperparameter for the Adam optimizer. It controls the exponential moving average of past gradients.

adam_beta2

For 'adamw' optimizer, `adam_beta2`

sets the beta2 hyperparameter for the Adam optimizer. It controls the exponential moving average of past squared gradients.

adam_epsilon

For 'adamw' optimizer, `adam_epsilon`

sets the epsilon value added to the denominator to prevent division by zero.

max_grad_norm

`max_grad_norm`

specifies the maximum gradient norm value. Gradients are clipped to this value during training to prevent exploding gradients.