# Llama3 - Lora Configuration

This is the default Lora Configuration for Llama3

```yaml
adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
```

#### <mark style="color:yellow;">`lora_r`</mark>

This parameter determines the rank of the low-rank matrices used in LoRA.&#x20;

It controls the capacity and expressiveness of the LoRA adaptation. A higher value of `lora_r` allows for more fine-grained adaptations but also increases the number of trainable parameters.&#x20;

In this configuration, <mark style="color:yellow;">**`lora_r`**</mark> is set to 32.

#### <mark style="color:yellow;">`lora_alpha`</mark>

This parameter controls the scaling factor applied to the LoRA adaptation.&#x20;

It determines the contribution of the LoRA matrices to the original model's weights.

A higher value of <mark style="color:yellow;">**`lora_alpha`**</mark> gives more importance to the LoRA adaptation. In our configuration, <mark style="color:yellow;">**`lora_alpha`**</mark> is set to 16.

#### <mark style="color:yellow;">`lora_dropout`</mark>

This parameter specifies the dropout rate applied to the LoRA matrices during training.&#x20;

Dropout is a regularization technique that helps prevent overfitting.&#x20;

A value of 0.05 means that 5% of the elements in the LoRA matrices will be randomly set to zero during training.

#### <mark style="color:yellow;">`lora_target_modules`</mark>

This parameter specifies the names of the modules in the model architecture where LoRA will be applied.&#x20;

In this  configuration, LoRA is applied to the `q_proj` and `v_proj` modules, which are likely the query and value projection matrices in the attention mechanism.

We have commented out other potential target modules like `k_proj`, `o_proj`, `gate_proj`, `down_proj`, and `up_proj`.

#### <mark style="color:yellow;">`lora_target_linear`</mark>

This parameter is not set in your configuration.&#x20;

If set to `true`, LoRA will be applied to all linear modules in the model.

#### <mark style="color:yellow;">`peft_layers_to_transform`</mark>

This parameter allows you to specify the indices of the layers to which LoRA should be applied.

If not specified, LoRA will be applied to all layers by default.

#### <mark style="color:yellow;">`lora_modules_to_save`</mark>

This parameter is relevant when you have added new tokens to the tokenizer. In such cases, you may need to save certain LoRA modules that are aware of the new tokens.&#x20;

For LLaMA and Mistral models, you typically need to save `embed_tokens` and `lm_head` modules. `embed_tokens` converts tokens to embeddings, and `lm_head` converts embeddings to token probabilities.&#x20;

In this configuration, these modules are commented out.

#### <mark style="color:yellow;">`lora_fan_in_fan_out`</mark>

This parameter determines the structure of the LoRA matrices. If set to <mark style="color:yellow;">**`true`**</mark>, it uses a more efficient implementation of LoRA that reduces the number of additional parameters. In your configuration, it is set to <mark style="color:yellow;">**`false`**</mark>.

These hyperparameters allow you to control various aspects of the LoRA adaptation during fine-tuning.&#x20;

The optimal values for these hyperparameters may vary depending on your specific task, dataset, and model architecture. It's recommended to experiment with different configurations and monitor the performance to find the best settings for your use case.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://axolotl.continuumlabs.pro/llama3/llama3-lora-configuration.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.