# Training Configuration
Field NameExplanation
gradient_accumulation_stepsgradient_accumulation_steps controls the number of forward and backward passes to skip before updating the model's weights. It is useful when the batch size is too large to fit into GPU memory at once. Gradients are accumulated over the specified number of steps before performing weight updates.
micro_batch_sizemicro_batch_size specifies the number of samples included in each batch sent to each GPU. It helps in determining the mini-batch size for training.
eval_batch_sizeeval_batch_size sets the batch size for evaluation. It determines how many samples are processed in each evaluation step.
num_epochsnum_epochs defines the number of training epochs, which represent the number of times the entire dataset is processed during training.
warmup_stepswarmup_steps specifies the number of warm-up steps for the learning rate scheduler. During the warm-up phase, the learning rate gradually increases to its full value.
learning_ratelearning_rate sets the initial learning rate for training. It is a critical hyperparameter that determines the step size for weight updates during optimization.
lr_quadratic_warmuplr_quadratic_warmup is a field related to learning rate scheduling. It is used to specify a quadratic warm-up schedule for the learning rate, which can be beneficial for certain training scenarios.
logging_stepslogging_steps sets the frequency at which training logs are generated. It controls how often training progress is reported.
save_strategysave_strategy determines when model checkpoints are saved during training. Setting it to 'no' skips checkpoint saves, while other options control the timing of saves.
save_stepssave_steps specifies the frequency at which model checkpoints are saved. You can leave it empty to save at each epoch or specify a different number of steps.
eval_stepseval_steps controls the frequency of model evaluation during training. It can be specified as an integer for every N steps or as a decimal for a fraction of total steps.
save_total_limitsave_total_limit limits the maximum number of checkpoints saved at a time. Older checkpoints are deleted to keep the total number within this limit.
max_stepsmax_steps defines the maximum number of iterations to train for. It takes precedence over num_epochs. For example, if you set max_steps to 100, the training will stop after 100 steps, regardless of the number of epochs.
eval_table_sizeeval_table_size specifies the approximate number of predictions sent to wandb (Weights and Biases) depending on the batch size. This field is enabled above 0 and is useful for tracking evaluation metrics.
eval_table_max_new_tokenseval_table_max_new_tokens sets the total number of tokens generated for predictions sent to wandb. It helps control the amount of data sent for monitoring.
save_safetensorsWhen specified, save_safetensors indicates saving the model as safetensors, requiring the safetensors package for compatibility.
train_on_inputstrain_on_inputs determines whether to mask out or include the human's prompt from the training labels. Setting it to 'false' omits the prompt from training.
group_by_lengthWhen set to 'true,' group_by_length groups data with similar sequence lengths together to minimize padding. This can help improve training efficiency but may lead to an oscillating training loss.
gradient_checkpointinggradient_checkpointing controls whether to use gradient checkpointing, a technique that can reduce memory consumption during training. When enabled, it trades off computation for memory.
early_stopping_patienceearly_stopping_patience determines when to stop training if evaluation losses increase consecutively for a specified number of times. It helps prevent overfitting.
lr_schedulerlr_scheduler specifies the learning rate scheduler to use during training. Options include 'one_cycle,' 'log_sweep,' or leaving it empty for cosine scheduling.
lr_scheduler_kwargslr_scheduler_kwargs can be used to provide additional arguments to the learning rate scheduler, depending on the chosen scheduler type.
lr_div_factorFor 'one_cycle' optimizer, lr_div_factor determines the learning rate division factor during the one-cycle learning rate schedule.
log_sweep_min_lrFor 'log_sweep' optimizer, log_sweep_min_lr sets the minimum learning rate for the logarithmic learning rate sweep.
log_sweep_max_lrFor 'log_sweep' optimizer, log_sweep_max_lr sets the maximum learning rate for the logarithmic learning rate sweep.
optimizeroptimizer specifies the optimizer to use for training. There are various optimizer options available, and the choice depends on the model and use case.
weight_decayweight_decay determines the weight decay applied during optimization. It is a regularization term that prevents overfitting by penalizing large model weights.
adam_beta1For 'adamw' optimizer, adam_beta1 sets the beta1 hyperparameter for the Adam optimizer. It controls the exponential moving average of past gradients.
adam_beta2For 'adamw' optimizer, adam_beta2 sets the beta2 hyperparameter for the Adam optimizer. It controls the exponential moving average of past squared gradients.
adam_epsilonFor 'adamw' optimizer, adam_epsilon sets the epsilon value added to the denominator to prevent division by zero.
max_grad_normmax_grad_norm specifies the maximum gradient norm value. Gradients are clipped to this value during training to prevent exploding gradients.