LogoLogo
Continuum Knowledge BankContinuum Applications
  • Introduction
  • Creation of Environment
    • Platform Installation
    • Axolotl Dependencies
    • setup.py objectives
      • script analysis
  • Huggingface Hub
  • Download the dataset
    • Types of Dataset Structures
    • Structuring Datasets for Fine-Tuning Large Language Models
    • Downloading Huggingface Datasets
    • Use Git to download dataset
    • Popular Datasets
    • Download cleaned Alpaca dataset
    • Template-free prompt construction
  • Downloading models
    • Phi 2.0 details
    • Downloading Phi 2.0
    • Available Models
  • Configuration for Training
  • Datasets
  • Model Selection - General
  • Phi 2.0
    • Phi 2.0 - Model Configuration
    • Phi 2.0 - Model Quantization
    • Phi 2.0 - Data Loading and Paths
    • Phi 2.0 - Sequence Configuration
    • Phi 2.0 - Lora Configuration
    • Phi 2.0 - Logging
    • Phi 2.0 - Training Configuration
    • Phi 2.0 - Data and Precision
    • Phi 2.0 - Optimisations
    • Phi 2.0 - Extra Hyperparameters
    • Phi 2.0 - All Configurations
    • Phi 2.0 - Preprocessing
    • Phi 2.0 - Training
    • Uploading Models
  • Llama2
    • Llama2 - Model Configuration
    • Llama2 - Model Quantization
    • Llama2 - Data Loading and Paths
    • Llama2 - Sequence Configuration
    • Llama2 - Lora Configuration
    • Llama2 - Logging
    • Llama2 - Training Configuration
    • Llama2 - Data and Precision
    • Llama2 - Optimisations
    • Llama2 - Extra Hyperparameters
    • Llama2- All Configurations
    • Llama2 - Training Configuration
    • Llama2 - Preprocessing
    • Llama2 - Training
  • Llama3
    • Downloading the model
    • Analysis of model files
      • Model Analysis - Configuration Parameters
      • Model Analysis - Safetensors
      • Tokenizer Configuration Files
        • Model Analysis - tokenizer.json
        • Model Analysis - Special Tokens
    • Llama3 - Model Configuration
    • Llama3 - Model Quantization
    • Llama3 - Data Loading and Paths
    • Llama3 - Sequence Configuration
    • Llama3 - Lora Configuration
    • Llama3 - Logging
    • Llama3 - Training Configuration
    • Llama3 - Data and Precision
    • Llama3 - Optimisations
    • Llama3 - Extra Hyperparameters
    • Llama3- All Configurations
    • Llama3 - Preprocessing
    • Llama3 - Training
    • Full Fine Tune
  • Special Tokens
  • Prompt Construction for Fine-Tuning Large Language Models
  • Memory-Efficient Fine-Tuning Techniques for Large Language Models
  • Training Ideas around Hyperparameters
    • Hugging Face documentation on loading PEFT
  • After fine tuning LLama3
  • Merging Model Weights
  • Merge Lora Instructions
  • Axolotl Configuration Files
    • Configuration Options
    • Model Configuration
    • Data Loading and Processing
    • Sequence Configuration
    • Lora Configuration
    • Logging
    • Training Configuration
    • Augmentation Techniques
  • Axolotl Fine-Tuning Tips & Tricks: A Comprehensive Guide
  • Axolotl debugging guide
  • Hugging Face Hub API
  • NCCL
  • Training Phi 1.5 - Youtube
  • JSON (JavaScript Object Notation)
  • General Tips
  • Datasets
Powered by GitBook
LogoLogo

This documentation is for the Axolotl community

On this page

Was this helpful?

  1. Axolotl Configuration Files

Lora Configuration

Field Name
Explanation

adapter

The adapter field specifies whether to use 'lora' for fine-tuning or leave it blank to train all parameters in the original model. Using 'lora' allows for more fine-grained control over certain model parameters during training.

lora_model_dir

If you already have a trained LoRA model that you want to load, you can specify the directory path here. This is useful for testing the model after training. Ensure this value matches the path to the saved LoRA model.

lora_r

lora_r is a hyperparameter that determines the number of repetitions or hops for the LoRA mechanism. It controls how many times the adapter parameters are applied to the input sequence. A higher value can capture longer dependencies but may require more resources.

lora_alpha

lora_alpha is another hyperparameter that controls the strength of the LoRA mechanism. It determines how much influence the adapter parameters have on the original model's output. Higher values make the adapter more influential.

lora_dropout

lora_dropout represents the dropout rate applied to the adapter's internal layers. Dropout is a regularization technique that helps prevent overfitting during training by randomly dropping some neurons.

lora_target_modules

lora_target_modules is a list of module names within the transformer architecture that the LoRA adapter parameters should be applied to. In this example, it targets 'q_proj' and 'v_proj' modules. You can uncomment and add more modules as needed.

lora_target_linear

If set to 'true,' the LoRA adapter will target all linear layers within the specified lora_target_modules. This option can be used for a more comprehensive adaptation of linear layers.

lora_modules_to_save

lora_modules_to_save specifies which LoRA modules should be saved. It's essential when adding new tokens to the tokenizer because certain LoRA modules need to be aware of these new tokens. In this example, the modules are commented out, but you should specify the modules to save as needed.

lora_out_dir

After completing training, the model will be saved to the directory specified by lora_out_dir. If you merge the adapter to the base model, a subdirectory named 'merged' will be created under this directory. Ensure that lora_model_dir points to this directory if you intend to use the trained model.

lora_fan_in_fan_out

When set to 'false,' this option disables the use of fan-in/fan-out scaling in the LoRA mechanism. Fan-in/fan-out scaling is a technique that helps stabilize training by scaling the adapter's parameters based on the input and output dimensions of the targeted modules.

relora_steps

relora_steps specifies the number of steps per ReLoRA (Restartable LoRA) restart. ReLoRA is a technique used to restart the LoRA mechanism during training to improve convergence. This field sets the frequency of restarts.

relora_warmup_steps

relora_warmup_steps sets the number of warm-up steps for each ReLoRA restart. Warm-up steps allow the LoRA mechanism to adapt gradually during a restart, which can improve training stability.

relora_cpu_offload

When set to 'true,' this option performs LoRA weight merges on the CPU during restarts, potentially saving GPU memory. It's useful for situations where GPU memory is a limiting factor during training.

PreviousSequence ConfigurationNextLogging

Last updated 1 year ago

Was this helpful?

Page cover image