LogoLogo
Continuum Knowledge BankContinuum Applications
  • Introduction
  • Creation of Environment
    • Platform Installation
    • Axolotl Dependencies
    • setup.py objectives
      • script analysis
  • Huggingface Hub
  • Download the dataset
    • Types of Dataset Structures
    • Structuring Datasets for Fine-Tuning Large Language Models
    • Downloading Huggingface Datasets
    • Use Git to download dataset
    • Popular Datasets
    • Download cleaned Alpaca dataset
    • Template-free prompt construction
  • Downloading models
    • Phi 2.0 details
    • Downloading Phi 2.0
    • Available Models
  • Configuration for Training
  • Datasets
  • Model Selection - General
  • Phi 2.0
    • Phi 2.0 - Model Configuration
    • Phi 2.0 - Model Quantization
    • Phi 2.0 - Data Loading and Paths
    • Phi 2.0 - Sequence Configuration
    • Phi 2.0 - Lora Configuration
    • Phi 2.0 - Logging
    • Phi 2.0 - Training Configuration
    • Phi 2.0 - Data and Precision
    • Phi 2.0 - Optimisations
    • Phi 2.0 - Extra Hyperparameters
    • Phi 2.0 - All Configurations
    • Phi 2.0 - Preprocessing
    • Phi 2.0 - Training
    • Uploading Models
  • Llama2
    • Llama2 - Model Configuration
    • Llama2 - Model Quantization
    • Llama2 - Data Loading and Paths
    • Llama2 - Sequence Configuration
    • Llama2 - Lora Configuration
    • Llama2 - Logging
    • Llama2 - Training Configuration
    • Llama2 - Data and Precision
    • Llama2 - Optimisations
    • Llama2 - Extra Hyperparameters
    • Llama2- All Configurations
    • Llama2 - Training Configuration
    • Llama2 - Preprocessing
    • Llama2 - Training
  • Llama3
    • Downloading the model
    • Analysis of model files
      • Model Analysis - Configuration Parameters
      • Model Analysis - Safetensors
      • Tokenizer Configuration Files
        • Model Analysis - tokenizer.json
        • Model Analysis - Special Tokens
    • Llama3 - Model Configuration
    • Llama3 - Model Quantization
    • Llama3 - Data Loading and Paths
    • Llama3 - Sequence Configuration
    • Llama3 - Lora Configuration
    • Llama3 - Logging
    • Llama3 - Training Configuration
    • Llama3 - Data and Precision
    • Llama3 - Optimisations
    • Llama3 - Extra Hyperparameters
    • Llama3- All Configurations
    • Llama3 - Preprocessing
    • Llama3 - Training
    • Full Fine Tune
  • Special Tokens
  • Prompt Construction for Fine-Tuning Large Language Models
  • Memory-Efficient Fine-Tuning Techniques for Large Language Models
  • Training Ideas around Hyperparameters
    • Hugging Face documentation on loading PEFT
  • After fine tuning LLama3
  • Merging Model Weights
  • Merge Lora Instructions
  • Axolotl Configuration Files
    • Configuration Options
    • Model Configuration
    • Data Loading and Processing
    • Sequence Configuration
    • Lora Configuration
    • Logging
    • Training Configuration
    • Augmentation Techniques
  • Axolotl Fine-Tuning Tips & Tricks: A Comprehensive Guide
  • Axolotl debugging guide
  • Hugging Face Hub API
  • NCCL
  • Training Phi 1.5 - Youtube
  • JSON (JavaScript Object Notation)
  • General Tips
  • Datasets
Powered by GitBook
LogoLogo

This documentation is for the Axolotl community

On this page

Was this helpful?

  1. Llama2

Llama2 - Lora Configuration

This is the default Lora Configuration.

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

lora_r

This parameter determines the rank of the low-rank matrices used in LoRA.

It controls the capacity and expressiveness of the LoRA adaptation. A higher value of lora_r allows for more fine-grained adaptations but also increases the number of trainable parameters.

In this configuration, lora_r is set to 32.

lora_alpha

This parameter controls the scaling factor applied to the LoRA adaptation.

It determines the contribution of the LoRA matrices to the original model's weights.

A higher value of lora_alpha gives more importance to the LoRA adaptation. In your configuration, lora_alpha is set to 16.

lora_dropout

This parameter specifies the dropout rate applied to the LoRA matrices during training.

Dropout is a regularization technique that helps prevent overfitting.

A value of 0.05 means that 5% of the elements in the LoRA matrices will be randomly set to zero during training.

lora_target_modules

This parameter specifies the names of the modules in the model architecture where LoRA will be applied.

In this configuration, LoRA is applied to the q_proj and v_proj modules, which are likely the query and value projection matrices in the attention mechanism.

We have commented out other potential target modules like k_proj, o_proj, gate_proj, down_proj, and up_proj.

lora_target_linear

This parameter is not set in your configuration.

If set to true, LoRA will be applied to all linear modules in the model.

peft_layers_to_transform

This parameter allows you to specify the indices of the layers to which LoRA should be applied.

If not specified, LoRA will be applied to all layers by default.

lora_modules_to_save

This parameter is relevant when you have added new tokens to the tokenizer. In such cases, you may need to save certain LoRA modules that are aware of the new tokens.

For LLaMA and Mistral models, you typically need to save embed_tokens and lm_head modules. embed_tokens converts tokens to embeddings, and lm_head converts embeddings to token probabilities.

In this configuration, these modules are commented out.

lora_fan_in_fan_out

This parameter determines the structure of the LoRA matrices. If set to true, it uses a more efficient implementation of LoRA that reduces the number of additional parameters. In your configuration, it is set to false.

These hyperparameters allow you to control various aspects of the LoRA adaptation during fine-tuning.

The optimal values for these hyperparameters may vary depending on your specific task, dataset, and model architecture.

It's recommended to experiment with different configurations and monitor the performance to find the best settings for your use case.

PreviousLlama2 - Sequence ConfigurationNextLlama2 - Logging

Last updated 1 year ago

Was this helpful?

Page cover image