# Introduction

This is Continuum's documentation for training large language models.

This training platform has been put together by a dedicated group of people, whose generosity has allowed a community to form and become able to fine tune a variety of large language models.

{% embed url="<https://github.com/OpenAccess-AI-Collective/axolotl>" %}
Link to Axolotl GitHub Repository
{% endembed %}

### <mark style="color:blue;">Background</mark>

The GitHub repository "Axolotl" provides a versatile tool designed for the fine-tuning of various AI models, specifically targeting ease of use and flexibility in handling different model configurations and architectures.&#x20;

### <mark style="color:blue;">Core Purpose</mark>

Axolotl is aimed at streamlining the fine-tuning process of AI models, offering compatibility with a wide range of Huggingface models and fine-tuning techniques.

### <mark style="color:blue;">Supported Features</mark>

* **Model Support**: It supports training with various Huggingface models
* **Fine-tuning Techniques**: The tool supports several fine-tuning methods&#x20;
* **Configuration Flexibility**: Users can customise configurations using a YAML file or override settings via the CLI.
* **Dataset Compatibility**: Axolotl can load different dataset formats, support custom formats, or handle user-provided tokenized datasets.
* **Integration with Advanced Tools**: The tool integrates with xformer, flash attention, rope scaling, and multipacking for enhanced model performance and efficiency.
* **Multi-GPU Support**: It facilitates training on single or multiple GPUs using FSDP or Deepspeed.
* **Docker Support**: Axolotl can be easily run with Docker, either locally or on the cloud.
* **Experiment Tracking**: It allows logging results and optionally checkpoints to WandB or MLflow.


# Creation of Environment

This documentation has been prepared to assist the implementation of the Axolotl training and fine tuning platform

## <mark style="color:blue;">**Environment Creation**</mark>

The first step in the process is ensuring our machine is optimised to use the NVIDIA platform and GPUs

## <mark style="color:blue;">Base Distribution: Ubuntu 20.04</mark>

We are going to use <mark style="color:purple;">**Ubuntu 20.04**</mark> as the base image because:<br>

-This version is known for its long-term support (LTS)

-Ubuntu is well-supported by NVIDIA

-Works well as remote instance so we can access high powered GPUs

-These remote environments allows all team members to access our platform

## <mark style="color:blue;">Integrated Development Environment (IDE)</mark>&#x20;

We recommend using Visual Studio Code (VS Code)  for our <mark style="color:yellow;">Integrated Development Environment</mark>

1. <mark style="color:green;">**Support for Remote Development**</mark><mark style="color:green;">:</mark> VS Code allows remote development support, crucial for accessing and managing virtual machines with powerful GPUs
2. <mark style="color:green;">**Integrated Terminal and Docker Support**</mark><mark style="color:green;">:</mark> The integrated terminal in VS Code enables direct interaction with command-line tools, essential for managing Docker containers and executing model training scripts.&#x20;
3. <mark style="color:green;">**Extensive Language Support**</mark><mark style="color:green;">:</mark> Large language model development often involves multiple programming languages (like Python, C++).  VS Code supports a wide range of languages and their specific tooling, which is critical for such multifaceted development.
4. <mark style="color:green;">**Version Control Integration**</mark><mark style="color:green;">:</mark> With built-in Git support, VS Code makes it easier to track and manage changes in code

## <mark style="color:blue;">Virtual Machine Requirements:</mark>

-Docker\*

-CUDA (Version <mark style="color:yellow;">12.1</mark>)\*: parallel computing platform and programming model

-NVIDIA NGC: NVIDIA Container toolkit for access to NVIDIA Docker Container

-NVIDIA CUDA Toolkit\*:  <mark style="color:yellow;">compiler for CUDA</mark>, translates CUDA code into executable programs

-GCC:  the <mark style="color:blue;">**compiler**</mark> required for development using the CUDA Toolkit

-GLIBC: the GNU Project's implementation of the C standard library.  Includes facilities for basic file I/O, string manipulation, mathematical functions, and various other standard utilities.

#### \*Please note, Continuum's base virtual machine installation script <mark style="color:purple;">installs Docker, the NVIDIA Container Toolkit and CUDA Driver 12.1</mark> as well <mark style="color:purple;">as the</mark> <mark style="color:yellow;">NVIDIA Container Toolkit</mark>

## <mark style="color:blue;">Check the virtual machine is ready</mark>&#x20;

To ensure that the virtual machine is set up for the training and fine tuning of large language models, <mark style="color:purple;">follow the instructions below:</mark>

## <mark style="color:blue;">Check the installation of the</mark> <mark style="color:yellow;">NVIDIA CUDA Toolkit</mark>

<mark style="color:green;">What is the CUDA Toolkit?</mark>

The NVIDIA CUDA Toolkit provides a <mark style="color:yellow;">development environment</mark> for creating high performance GPU-accelerated applications.&#x20;

With the CUDA Toolkit, you can develop, optimize, and deploy your applications on GPU-accelerated systems.

The toolkit includes GPU-accelerated libraries, debugging and optimization tools, a C/C++ compiler, and a runtime library to deploy your application.

### We will be installing the NVIDA CUDA Toolkit, <mark style="color:yellow;">version 12.1</mark>

First, check to see if the CUDA Toolkit is installed, we can check to see whether the core compiler is installed, the NVIDIA CUDA Compiler (NVCC)

<mark style="color:blue;">**Nvidia CUDA Compiler (NVCC)**</mark> is a part of the CUDA Toolkit.  It is the <mark style="color:yellow;">compiler for CUDA</mark>, responsible for translating CUDA code into executable programs.  &#x20;

NVCC takes high-level CUDA code and turns it into a form that can be understood and executed by the GPU.  It handles the partitioning of code into segments that can be run on either the CPU or GPU, and manages the compilation of the GPU parts of the code.

First, to check if <mark style="color:yellow;">NVCC is installed</mark> and its version, run

```bash
nvcc --version
```

If the NVIDIA CUDA Toolkit has been installed, this will be the output:

<pre class="language-bash"><code class="lang-bash">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 12.1, V11.8.89
<a data-footnote-ref href="#user-content-fn-1">Build cuda_</a>12.1.r11.8/compiler.31833905_0
</code></pre>

You should see <mark style="color:yellow;">release 12.1</mark> - which indicates the <mark style="color:yellow;">CUDA Toolkit Version 12.1</mark> has been successfully installed.

If NVCC is not installed, then go ahead an install the CUDA Toolkit 12.1. &#x20;

The CUDA Toolkit download website is located here:

{% embed url="<https://developer.nvidia.com/cuda-12-1-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local>" %}
CUDA 12.1 Download
{% endembed %}

The web application at this site will ask you to define your installation set up. &#x20;

For our base virtual machine this will be:

| Variable         |     Parameter    |
| ---------------- | :--------------: |
| Operation System |       Linux      |
| Architecture     | x86\_64 (64 bit) |
| Distribution     |      Ubuntu      |
| Version          |       20.04      |
| Installer Type   |    deb (local)   |

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2F68VYV9F3sfrjEKXEDKOJ%2FScreenshot%202023-12-06%20122715.png?alt=media&amp;token=49f72359-3756-4299-8656-7a2d55924379" alt=""><figcaption><p>This website will provide you the instructions for installing the CUDA Toolkit 12.1</p></figcaption></figure>

The detail instructions and explanation of the process is below:

<details>

<summary>Reference: <mark style="color:green;">Installation of CUDA Toolkit 12.1</mark></summary>

### <mark style="color:blue;">Using the local installation approach</mark>

<mark style="color:blue;">**deb (local)**</mark>

* This installer is a Debian package file that contains all the necessary CUDA files.
* It's a large file because it includes everything needed for the installation.

#### <mark style="color:purple;">Step 1: Download the CUDA Repository Pin</mark>

Open your terminal and execute the following command. This will <mark style="color:yellow;">download a pin file</mark> for the CUDA repository:

{% code overflow="wrap" %}

```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
```

{% endcode %}

#### <mark style="color:purple;">Step 2: Move the Pin File</mark>

Move the downloaded pin file to `/etc/apt/preferences.d/`. This will prioritize the CUDA repository over others:

```bash
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
```

#### <mark style="color:purple;">Step 3: Download the CUDA Repository Package</mark>

Now, download the Debian package for the CUDA repository:

{% code overflow="wrap" %}

```bash
wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb
```

{% endcode %}

#### <mark style="color:purple;">Step 4: Install the CUDA Repository Package</mark>

Install the downloaded package using `dpkg`:

```bash
sudo dpkg -i cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb
```

#### <mark style="color:purple;">Step 5: Copy the GPG Keyring</mark>

Copy the GPG keyring file to the `/usr/share/keyrings/` directory. This step is necessary for the authentication of the CUDA repository:

```bash
sudo cp /var/cuda-repo-ubuntu2004-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/
```

#### <mark style="color:purple;">Step 6: Update the Package Lists</mark>

Update the package lists to include packages from the newly added CUDA repository:

```bash
sudo apt-get update
```

#### <mark style="color:purple;">Step 7: Install CUDA</mark>

Finally, install CUDA:

```bash
sudo apt-get -y install cuda
```

### Post-Installation

After the installation is complete, you will need to <mark style="color:yellow;">set up the environment variables for CUDA.</mark>&#x20;

Add the following lines to your `.bashrc` or `.bash_profile` to set `PATH` and `LD_LIBRARY_PATH`:

```bash
export PATH=/usr/local/cuda-12.1/bin${PATH:+:${PATH}}
```

```bash
export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```

After editing, apply the changes:

```bash
source ~/.bashrc
```

#### Verify Installation

To verify the installation, you can check the CUDA version:

```bash
nvcc --version
```

</details>

Post installation, to check if <mark style="color:yellow;">NVCC has been installed successfully</mark> and check its version, run

```bash
nvcc --version
```

If the NVIDA CUDA Toolkit has been installed, this will be the output:

<pre class="language-bash"><code class="lang-bash">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 12.1, V11.8.89
<a data-footnote-ref href="#user-content-fn-1">Build cuda_11.8</a>.r11.8/compiler.31833905_0
</code></pre>

{% hint style="warning" %}
If NVCC is pointing to an older version of CUDA despite upgrading to 12.1 you will need to follow the instructions below.  CUDA must be on PATH!
{% endhint %}

<details>

<summary><mark style="color:yellow;">Make sure CUDA 12.1 is on PATH</mark></summary>

If you've followed the CUDA installation instructions and still find that an old version of NVCC (NVIDIA CUDA Compiler) is installed, there are several steps you can take to troubleshoot and resolve the issue:

### <mark style="color:blue;">**Verify the Installation**</mark><mark style="color:blue;">:</mark>

Ensure that CUDA was installed correctly without errors. You can check the installation logs for any errors or warnings that might have occurred during the installation process.

Use the command to <mark style="color:yellow;">list all CUDA-related packages</mark> installed and their versions.  Enter into the terminal:

```bash
dpkg -l | grep cuda
```

This will provide a list of all CUDA related packages:

Below are some example files that this command highlights, for example:

<mark style="color:blue;">**cuda, cuda-12-1**</mark><mark style="color:blue;">:</mark> These are meta-packages for CUDA version 12.1. Installing a meta-package <mark style="color:yellow;">installs all the components of CUDA.</mark>

<mark style="color:blue;">**cuda-cccl-12-1**</mark><mark style="color:blue;">:</mark> CUDA CCCL (CUDA C++ Core Library) is part of the CUDA Toolkit and provides <mark style="color:yellow;">essential libraries for CUDA C++ development.</mark>

<mark style="color:blue;">**cuda-command-line-tools-12-1**</mark><mark style="color:blue;">:</mark> This package includes <mark style="color:yellow;">command-line tools for CUDA,</mark> such as `nvcc` (NVIDIA CUDA Compiler), which is crucial for compiling CUDA code.

<mark style="color:blue;">**cuda-compiler-12-1**</mark><mark style="color:blue;">:</mark> This package includes the <mark style="color:yellow;">CUDA compiler,</mark> which is essential for converting CUDA code into code that can run on Nvidia GPUs.

<mark style="color:blue;">**cuda-demo-suite-12-1**</mark><mark style="color:blue;">:</mark> This package <mark style="color:yellow;">contains demos showcasing the capabilities</mark> and features of CUDA.

<mark style="color:blue;">**cuda-documentation-12-1**</mark><mark style="color:blue;">:</mark> Provides the <mark style="color:yellow;">documentation for CUDA,</mark> useful for developers to understand and use CUDA APIs.

<mark style="color:blue;">**cuda-driver-dev-12-1**</mark><mark style="color:blue;">:</mark> Includes <mark style="color:yellow;">development resources for the CUDA driver,</mark> such as headers and stub libraries.

<mark style="color:blue;">**cuda-libraries-12-1, cuda-libraries-dev-12-1**</mark><mark style="color:blue;">:</mark> These meta-packages include <mark style="color:yellow;">libraries necessary for CUDA development and their development counterparts.</mark>

<mark style="color:blue;">**cuda-nvcc-12-1**</mark><mark style="color:blue;">:</mark> NVIDIA CUDA Compiler (NVCC) is a tool for compiling CUDA code.

<mark style="color:blue;">**cuda-repo-ubuntu2004-12-1-local**</mark><mark style="color:blue;">:</mark> Contains repository configuration files for the CUDA toolkit

### <mark style="color:blue;">**Check Environment Variables**</mark><mark style="color:blue;">:</mark>

* Ensure that your environment variables are pointing to the new CUDA installation. Specifically, check the <mark style="color:yellow;">`PATH`</mark> and <mark style="color:yellow;">`LD_LIBRARY_PATH`</mark> environment variables via the following commands:

```bash
echo $PATH
```

```bash
echo $LD_LIBRARY_PATH
```

Update these variables if they are pointing to an older CUDA version.&#x20;

If CUDA is not on PATH:

### <mark style="color:blue;">**Export PATH**</mark>

```bash
export PATH=/usr/local/cuda-12.4/bin${PATH:+:${PATH}}
```

This command adds <mark style="color:yellow;">`/usr/local/cuda-12.1/bin`</mark> to the `PATH` environment variable.

<mark style="color:yellow;">`PATH`</mark> is a list of directories the shell searches for executable files. Adding CUDA's bin directory makes it possible to run CUDA tools and compilers directly from the command line without specifying their full path.

`${PATH:+:${PATH}}` is a <mark style="color:yellow;">shell parameter expansion pattern</mark> that appends the existing `PATH` variable to the new path. If <mark style="color:yellow;">`PATH`</mark> is unset, nothing is appended.

### <mark style="color:blue;">LD\_LIBRARY\_PATH Variable Setup</mark>

**Export LD\_LIBRARY\_PATH for 64-bit OS**:

{% code overflow="wrap" %}

```bash
export LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```

{% endcode %}

<mark style="color:yellow;">`LD_LIBRARY_PATH`</mark> is an environment variable specifying directories where libraries are searched for first, before the standard set of directories.

This command <mark style="color:yellow;">adds the</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`lib64`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">directory of the CUDA installation to</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`LD_LIBRARY_PATH`</mark>, which is necessary for 64-bit systems. It ensures that the system can find and use the CUDA libraries.

### <mark style="color:blue;">**Verify NVCC Version**</mark><mark style="color:blue;">:</mark>

After updating the environment variables, check the NVCC version again using:

```
nvcc --version
```

This should reflect the new version if the environment variables are set correctly.

**Update Alternatives**:

* Sometimes, multiple versions of CUDA can coexist, <mark style="color:yellow;">and the system may still use the old version.</mark> Use `update-alternatives` to configure the default CUDA version.
* Run `sudo update-alternatives --config nvcc` to choose the correct version of NVCC.

```bash
sudo update-alternatives --config
```

</details>

<details>

<summary>Reference: <mark style="color:green;">What is included in the CUDA Toolkit?</mark></summary>

\
CUDA contains  components and tools designed to facilitate development, optimisation, and deployment of GPU-accelerated applications.&#x20;

These components cater to various aspects of GPU computing, from low-level programming to high-level library support.&#x20;

Here's an overview of the contents CUDA 12.1 toolkit:

1. <mark style="color:blue;">**NVCC (NVIDIA CUDA Compiler)**</mark><mark style="color:blue;">:</mark> NVCC is the primary compiler that converts CUDA code into binary executable form.
2. <mark style="color:blue;">**CUDA Libraries**</mark><mark style="color:blue;">:</mark> CUDA 12.1 includes a range of GPU-accelerated libraries for different types of computing and data processing tasks. These include:
   * <mark style="color:blue;">**cuBLAS**</mark><mark style="color:blue;">:</mark> A GPU-accelerated implementation of the <mark style="color:yellow;">Basic Linear Algebra</mark> Subprograms (BLAS) library.
   * <mark style="color:blue;">**cuFFT**</mark><mark style="color:blue;">:</mark> A library for performing <mark style="color:yellow;">Fast Fourier Transforms (FFT)</mark> on the GPU.
   * <mark style="color:blue;">**cuRAND**</mark><mark style="color:blue;">:</mark> A library for <mark style="color:yellow;">generating random numbers</mark> on the GPU.
   * <mark style="color:blue;">**cuDNN**</mark>: A GPU-accelerated library for <mark style="color:yellow;">deep neural networks.</mark>
   * <mark style="color:blue;">**NVIDIA NPP (NVIDIA Performance Primitives)**</mark><mark style="color:blue;">:</mark> A collection of GPU-accelerated image, video, and signal processing functions.
   * <mark style="color:blue;">**Thrust**</mark><mark style="color:blue;">:</mark> A C++ parallel programming library resembling the C++ Standard Library.
3. <mark style="color:blue;">**CUDA Runtime and Driver APIs**</mark><mark style="color:blue;">:</mark> These APIs allow applications to manage devices, memory, and program executions on GPUs. The runtime API is designed for high-level programming, while the driver API provides lower-level control.
4. <mark style="color:blue;">**CUDA Debugger (cuda-gdb)**</mark><mark style="color:blue;">:</mark> A tool to debug CUDA applications running on the GPU.
5. <mark style="color:blue;">**CUDA Profiler (nvprof and Nsight Systems)**</mark><mark style="color:blue;">:</mark> Tools for profiling the performance of CUDA applications. These profilers help in identifying performance bottlenecks.
6. <mark style="color:blue;">**Nsight Eclipse Edition**</mark><mark style="color:blue;">:</mark> An integrated development environment (IDE) for developing and debugging CUDA applications on Linux and macOS.
7. <mark style="color:blue;">**CUDA Samples and Documentation**</mark><mark style="color:blue;">:</mark> A set of sample code and comprehensive documentation to help developers understand and use various features of the CUDA toolkit.
8. <mark style="color:blue;">**CUDA Toolkit SDK**</mark><mark style="color:blue;">:</mark> Contains additional libraries, code samples, and resources for software development.
9. <mark style="color:blue;">**Support for Various GPU Architectures**</mark><mark style="color:blue;">:</mark> Each CUDA version supports specific Nvidia GPU architectures. CUDA 12.1 would include support for the latest architectures available at its release.
10. <mark style="color:blue;">**Multi-Language Support**</mark>: While CUDA is primarily used with C and C++, it also supports other languages like Python (via libraries like PyCUDA).
11. <mark style="color:blue;">**GPU-accelerated Machine Learning and AI Libraries**</mark><mark style="color:blue;">:</mark> Integration with libraries and frameworks for AI and machine learning, such as TensorFlow and PyTorch, which can leverage CUDA for GPU acceleration.

</details>

## <mark style="color:green;">CUDA</mark> <mark style="color:yellow;">Reference Materials\*</mark>

Please find the NVIDIA CUDA Installation Guide for Linux

{% embed url="<https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html>" %}
Reference Materials for CUDA Installation for Linux distributions
{% endembed %}

<details>

<summary><mark style="color:green;">NVIDIA CUDA Installation Guide for Linux</mark></summary>

\
The NVIDIA CUDA Installation Guide for Linux outlines the process for installing the CUDA Toolkit on a Linux system.&#x20;

1. **System Requirements**:
   * Lists the prerequisites for using NVIDIA CUDA, including a CUDA-capable GPU, a supported version of Linux with gcc compiler and toolchain, and the CUDA Toolkit.
   * Provides information about compatible Linux distributions and kernel versions, including links to resources for specific Linux distributions like RHEL, SLES, Ubuntu LTS, and L4T.
2. **OS Support Policy**:
   * Details the support policy for various Linux distributions like Ubuntu, RHEL, CentOS, Rocky Linux, SUSE SLES, OpenSUSE Leap, Debian, Fedora, and KylinOS, including their end-of-support life (EOSS) timelines.
3. **Host Compiler Support Policy**:
   * Describes the supported compilers for different architectures (x86\_64, Arm64 sbsa, POWER 9), including GCC, Clang, NVHPC, XLC, ArmC/C++, and ICC.
   * Emphasizes the importance of using a compatible host compiler for compiling CPU host code in CUDA sources.
4. **Pre-installation Actions**:
   * Outlines steps to prepare for CUDA Toolkit and Driver installation, including verifying the presence of a CUDA-capable GPU, a supported version of Linux, gcc installation, correct kernel headers and development packages, and considering conflicting installation methods.
   * Notes the possibility of overriding install-time prerequisite checks using the `-override` flag.
5. **Verification Steps**:

   * Provides commands to verify the presence of a CUDA-capable GPU&#x20;

   (`lspci | grep -i nvidia`),
6. he supported version of Linux (`uname -m && cat /etc/*release`), and the gcc installation (`gcc --version`).
   * Explains how to check for the correct kernel headers and development packages.
7. **Ubuntu-Specific Installation Instructions**:
   * Details steps to prepare Ubuntu for CUDA installation, including installing kernel headers and removing outdated signing keys.
   * Describes two installation methods for Ubuntu: local repo and network repo, including steps for each method.
   * For both methods, provides common instructions to update the Apt repository cache and install the CUDA SDK, and includes a reminder to reboot the system after installation.

</details>

If you are interested, you can familiarise yourself with <mark style="color:green;">**CUDA Best Practices:**</mark>

{% embed url="<https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#cuda-compatibility-and-upgrades>" %}
CUDA Best Practices Guide
{% endembed %}

### <mark style="color:purple;">Below is a summary of CUDA Programming</mark>

<details>

<summary><mark style="color:green;">CUDA Tutorial - CUDA Programming is difficult!</mark></summary>

<mark style="color:blue;">**CUDA's Role in Machine Learning**</mark>

CUDA, NVIDIA's parallel computing platform, is particularly important in machine learning because many <mark style="color:yellow;">machine learning tasks involve linear algebra, like matrix multiplications and vector operations.</mark>  CUDA is optimised for these kinds of operations, offering significant performance improvements over traditional CPU processing.

<mark style="color:blue;">**Installation and Setup Requirements**</mark>

To use CUDA, you need to install the CUDA toolkit and have an <mark style="color:blue;">NVIDIA GPU</mark>, as CUDA does not work with other types of GPUs like AMD. The setup process varies depending on your operating system (Linux, Windows, or potentially Mac).

<mark style="color:blue;">**Prerequisite Knowledge**</mark>

a basic understanding of C or C++ programming is necessary to work with CUDA.  Concepts like memory allocation (`malloc`) and freeing memory are used without detailed explanations. If you're only familiar with Python, you might find following the CUDA examples challenging.

<mark style="color:blue;">**CUDA Programming Basics**</mark>

CUDA code is similar to C/C++ but includes additional functions and data types for parallel computing.  Understanding the transition from C to CUDA is critical for grasping how parallelisation is implemented in CUDA.

<mark style="color:blue;">**Understand CUDA's Grid and Block Model**</mark>

The explanation of CUDA's grid and block model is crucial. It's important to understand <mark style="color:yellow;">how to configure blocks and threads within a grid to effectively parallelize tasks.</mark> When defining grid and block dimensions, remember that they directly influence the number of threads that will execute your code and how they are organized. Incorrect configurations can lead to inefficient use of GPU resources or even cause your program to behave unexpectedly.

<mark style="color:blue;">**Memory Management in CUDA**</mark>

In CUDA, memory allocation and management are crucial, especially since you're dealing with both host (CPU) and device (GPU) memory. Incorrect memory handling can lead to crashes or incorrect program results.

<mark style="color:blue;">**Mapping Problem Domain to CUDA's Architecture**</mark>

The idea of <mark style="color:yellow;">mapping a matrix's shape to CUDA's grid shape illustrates a key concept in CUDA programming</mark>: effectively mapping your problem domain to CUDA's architecture. This can be challenging, as it requires a good <mark style="color:yellow;">understanding of both your application's requirements and CUDA's parallel execution model.</mark>

<mark style="color:blue;">**Performance Considerations**</mark>

One of the objectives of using CUDA is to enhance performance. It's important to note that not all problems will see a dramatic performance increase with CUDA, and sometimes the overhead of managing GPU resources can outweigh the benefits for smaller or less complex tasks.

<mark style="color:blue;">**Initialisation and Memory Representation**</mark>

It's important to understand how multi-dimensional data structures like matrices are represented in memory, especially since CUDA typically deals with flattened arrays. This understanding is crucial for correctly indexing elements during calculations.

<mark style="color:blue;">**Understanding the Mapping of Computation to CUDA Threads**</mark>

Each thread in CUDA is assigned a specific part of the computation, like a segment of a matrix-vector multiplication. This mapping is critical for efficient parallel computation. It's important to correctly calculate the indices each thread will work on, taking into account both block and thread indices.

<mark style="color:blue;">**Boundary Conditions in CUDA Computations**</mark>

When working with CUDA, it’s important to <mark style="color:yellow;">handle boundary conditions carefully.</mark>  If the dimensions of your data do not exactly match the grid and block dimensions you've configured, you must ensure that your code correctly handles these edge cases to avoid out-of-bounds memory access, which can lead to incorrect results or crashes.

<mark style="color:blue;">**Host-Device Memory Management**</mark>

The tutorial <mark style="color:yellow;">highlights the complexity of managing memory between the host (CPU) and the device (GPU)</mark>.  Data must be explicitly allocated and transferred between the host and device memories. This process adds an extra layer of complexity to CUDA programming and requires careful handling to ensure data integrity and to avoid memory leaks.

<mark style="color:blue;">**Grid and Block Dimension Calculations**</mark>

Calculating the dimensions of grids and blocks is a critical aspect of CUDA programming.  The dimensions influence how many threads are launched and how they are organised. Misconfigurations here can lead to inefficient use of GPU resources or even failure to execute the program correctly.

</details>

<details>

<summary><mark style="color:green;">CUDA - Version Control</mark></summary>

Managing CUDA versions and resolving conflicts can be critical.

### <mark style="color:purple;">Understanding CUDA and its Interaction with Libraries</mark>

<mark style="color:green;">**Compatibility:**</mark>

* <mark style="color:blue;">**CUDA and GPU Compatibility**</mark><mark style="color:blue;">:</mark> Each GPU model supports certain CUDA versions. Ensure your GPU is compatible with the CUDA version you plan to use.
* <mark style="color:blue;">**Library Dependencies**</mark><mark style="color:blue;">:</mark> Machine learning libraries like TensorFlow, PyTorch, etc., are *<mark style="color:yellow;">built against specific CUDA and cuDNN versions</mark>*<mark style="color:yellow;">.</mark> Using incompatible versions can lead to errors.

<mark style="color:green;">**CUDA Toolkit vs. CUDA Runtime**</mark><mark style="color:green;">:</mark>

* <mark style="color:blue;">**CUDA Toolkit**</mark><mark style="color:blue;">:</mark> Includes the CUDA runtime and development environment (compiler, debugger, etc.). Needed for compiling CUDA-enabled applications.
* <mark style="color:blue;">**CUDA Runtime**</mark><mark style="color:blue;">:</mark> Required to run CUDA-enabled applications. Often, libraries like PyTorch bundle the necessary CUDA runtime components, so you might not need a system-wide CUDA installation.

### <mark style="color:purple;">Continuum - managing CUDA Versions and Conflicts</mark>

1. **Isolate Environments**:
   * Use separate Python environments using Anaconda for different projects. This isolation allows you to install different versions of libraries (and their corresponding CUDA versions) without conflict.
   * Example: Have one Conda environment for TensorFlow with CUDA 11.0 and another for PyTorch with CUDA 10.2.
2. **Containerization**:
   * Use Docker or similar containerisation tools. They encapsulate the entire runtime environment, including the specific versions of CUDA, cuDNN, and other dependencies.
   * Nvidia provides Docker images (NGC Containers) with TensorFlow, PyTorch, etc., pre-installed with the required CUDA and cuDNN versions.
3. **Local CUDA Installation**:
   * If you need to compile CUDA code, install the CUDA Toolkit locally.
   * Be mindful of the CUDA version if you have multiple projects requiring different versions.
4. **Check Compatibility Before Installation**:
   * Before installing a deep learning library, check its compatibility with your CUDA version.
   * Use the installation commands tailored to specific CUDA versions (as seen with PyTorch&#x20;

</details>

## <mark style="color:blue;">Check the installation of the</mark> <mark style="color:yellow;">NVIDIA Container Toolkit</mark>

The NVIDIA Container Toolkit enables users to build and run GPU-accelerated containers.&#x20;

The toolkit includes a container runtime [library](https://github.com/NVIDIA/libnvidia-container) and utilities to automatically configure containers to leverage NVIDIA GPUs.

This allows you to use NVIDIA Container in Docker.

### <mark style="color:blue;">Background and Explanation</mark>

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FkMOq0fYrencY00wPXmMj%2F5b208976-b632-11e5-8406-38d379ec46aa.png?alt=media&amp;token=1fa21fd0-75fa-4c14-8364-45bdef3e5702" alt=""><figcaption></figcaption></figure>

The <mark style="color:blue;">**NVIDIA Container Toolkit**</mark> is designed to integrate NVIDIA GPUs into containerised applications. It's compatible with various container runtimes and consists of several components:

1. <mark style="color:blue;">**NVIDIA Container Runtime (nvidia-container-runtime):**</mark> An OCI-compliant runtime for Docker or containerd, enabling the use of NVIDIA GPUs in containers.
2. <mark style="color:blue;">**NVIDIA Container Runtime Hook (nvidia-container-toolkit / nvidia-container-runtime-hook):**</mark> A component executing prestart scripts to configure GPU access in containers.
3. <mark style="color:blue;">**NVIDIA Container Library and CLI (libnvidia-container1, nvidia-container-cli):**</mark> These provide a library and CLI for automatically configuring containers with NVIDIA GPU support, independent of the container runtime.

The toolkit's architecture allows for integration with various container runtimes like Docker, containerd, cri-o, and lxc. Notably, the NVIDIA Container Runtime is not required for cri-o and lxc.

The toolkit comprises main packages: nvidia-container-toolkit, nvidia-container-toolkit-base, libnvidia-container-tools, and libnvidia-container1, with specific dependencies between them.&#x20;

Older packages like nvidia-docker2 and nvidia-container-runtime are now deprecated and merged into the nvidia-container-toolkit.

### <mark style="color:blue;">The Architecture</mark>

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2F6iu9ymNCzKdfLmjX9BTn%2Fruntime-architecture.png?alt=media&amp;token=ce6bac03-5c12-47bc-bd35-6efd3a9fecb8" alt=""><figcaption></figcaption></figure>

### <mark style="color:blue;">Key functionalities</mark>

* <mark style="color:blue;">**NVIDIA Container Runtime Hook:**</mark> Implements a runC prestart hook, configuring GPU devices in containers based on the container's `config.json`.
* <mark style="color:blue;">**NVIDIA Container Runtime:**</mark> A wrapper around runC, modifying the OCI runtime spec for GPU support.
* <mark style="color:blue;">**NVIDIA Container Toolkit CLI:**</mark> Offers utilities for configuring runtimes and generating Container Device Interface (CDI) specifications.

For installation, the <mark style="color:yellow;">`nvidia-container-toolkit`</mark> package is generally sufficient.&#x20;

The toolkit's packages are available on GitHub, useful for both online and air-gapped installations. The repository also hosts experimental releases of the software.

### <mark style="color:blue;">Check for Docker</mark>

Before installing ensure you have Docker installed

```bash
docker --version
```

The output should be as below:

```bash
Docker version 26.0.2, build 3c863ff
```

To install the NVIDIA Container Toolkit, follow the instructions below:

<details>

<summary><mark style="color:green;">Installation of the NVIDIA Container Toolkit</mark></summary>

Configure the repository

{% code overflow="wrap" fullWidth="true" %}

```bash
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
```

{% endcode %}

Update the packages list from the repository:

```bash
sudo apt-get update
```

Install the NVIDIA Container Toolkit packages:

```bash
sudo apt-get install -y nvidia-container-toolkit
```

#### Configuring Docker

Configure the container runtime by using the `nvidia-ctk` command:

```bash
sudo nvidia-ctk runtime configure --runtime=docker
```

The `nvidia-ctk` command modifies the `/etc/docker/daemon.json` file on the host. The file is updated so that Docker can use the NVIDIA Container Runtime.

Restart the Docker daemon:

```bash
sudo systemctl restart docker
```

</details>

Post installation, you can check to see if the Container Toolkit is working:

```bash
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
```

The output should look like this:

```
---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-80GB           On | 00000000:00:05.0 Off |                    0 |
| N/A   35C    P0               55W / 400W|      5MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
```

### <mark style="color:blue;">Compatibility Testing</mark>

The CUDA development environment <mark style="color:yellow;">relies on tight integration with the host development environment</mark>, ***including the host compiler and C runtime libraries***, and is therefore only supported on Ubuntu versions that have been qualified for the CUDA Toolkit release.

Now that we have installed the NVIDIA CUDA Toolkit and the NVIDIA Container Toolkit, we need to ensure our virtual machine is compatible with these installations.

{% hint style="danger" %} <mark style="color:orange;">Compatibility is critical</mark>
{% endhint %}

The material below provides instructions on how to <mark style="color:yellow;">ensure the NVIDIA Drivers are compatible with the host system.</mark>

With the NVIDIA CUDA Toolkit installed, we need to ensure the host machine is compatible with this Toolkit.

### <mark style="color:blue;">Compatibility between CUDA 12.1 and the host development environment</mark>

This table lists the kernel versions, default <mark style="color:purple;">**GCC (GNU Compiler Collection)**</mark> versions, and <mark style="color:purple;">**GLIBC (GNU C Library) versions**</mark> for two different LTS (Long-Term Support) releases of Ubuntu.

<table data-full-width="false"><thead><tr><th>Distribution</th><th align="center">Kernel</th><th align="center">Default GCC</th><th align="center">GLIBC</th></tr></thead><tbody><tr><td>Ubuntu 22.04 LTS</td><td align="center">5.15.0-43</td><td align="center">11.2.0</td><td align="center">2.35</td></tr><tr><td><mark style="color:green;">Ubuntu 20.04 LTS</mark></td><td align="center"><mark style="color:red;">5.13.0-46</mark></td><td align="center"><mark style="color:red;">9.3.0</mark></td><td align="center"><mark style="color:red;">2.31</mark></td></tr></tbody></table>

### <mark style="color:blue;">Check the Kernel compatibility</mark>

To check the <mark style="color:blue;">**kernel version**</mark> of your Ubuntu 20.04 system, you can use the <mark style="color:yellow;">`uname`</mark> command in the terminal. The <mark style="color:yellow;">`uname`</mark> command with different options provides various system information, including the kernel version. Here's how you can do it:

**Run the&#x20;**<mark style="color:yellow;">**`uname`**</mark>**&#x20;command t**o get the kernel version by typing the following command and press Enter:

```bash
uname -r
```

The output should be this on a typical <mark style="color:yellow;">Ubuntu WSL2 distribution:</mark>

```bash
5.15.133.1-microsoft-standard-WSL2
```

or this on a <mark style="color:yellow;">typical Ubuntu 20.04</mark> virtual machine

```bash
5.4.0-167-generic
```

As you can see here the first  <mark style="color:blue;">Linux kernel is 5.15.133.1 -</mark> which is compatible with the CUDA Toolkit installed (range of 5.13.0 to 5.13.46). &#x20;

The second Linux kernel is also compatible at 5.4 (range of 5.13 to 5.46)

<details>

<summary><mark style="color:green;">What is a kernel?</mark></summary>

<mark style="color:yellow;">A kernel is the core component of an operating system (OS)</mark>.&#x20;

It acts as a bridge between applications and the actual data processing done at the hardware level<mark style="color:yellow;">.</mark>&#x20;

The kernel's responsibilities include managing the system's resources and allowing multiple programs to run and use these resources efficiently. Here are some key aspects of a kernel:

<mark style="color:blue;">**Resource Management**</mark>

The kernel <mark style="color:yellow;">manages hardware resources like the CPU, memory, and disk space</mark>. It allocates resources to various processes, ensuring that each process receives enough resources to function effectively while maintaining overall system efficiency.

<mark style="color:blue;">**Process Management**</mark>

It handles the creation, scheduling, and termination of processes. The kernel decides which processes should run when and for how long, a process known as <mark style="color:yellow;">scheduling.</mark> This is critical in multi-tasking environments where multiple processes require CPU attention.

<mark style="color:blue;">**Memory Management**</mark>

The kernel controls how <mark style="color:yellow;">memory is allocated to various processes and manages memory access</mark>, ensuring that each process has access to the memory it needs without interfering with other processes. It also manages virtual memory, allowing the system to use disk space as an extension of RAM.

<mark style="color:blue;">**Device Management**</mark>

It acts as an <mark style="color:yellow;">intermediary between the hardware and software of a computer.</mark>  For instance, when a program needs to read a file from a disk, it requests this service from the kernel, which then communicates with the disk drive’s hardware to read the data.

<mark style="color:blue;">**Security and Access Control**</mark>

The kernel enforces access control policies, preventing unauthorised access to the system and its resources. It manages user permissions and ensures that processes have the required privileges to execute their tasks.

<mark style="color:blue;">**System Calls**</mark>

These are the mechanisms through which user-space applications interact with the kernel.  For example, when an application needs to open a file, it makes a system call, which is handled by the kernel.

<mark style="color:blue;">**Types of Kernels**</mark>

* **Monolithic Kernels**: These kernels include various services like the filesystem, device drivers, network interfaces, etc., within one large kernel. Example: Linux.
* **Microkernels**: These kernels focus on minimal functionality, providing only basic services like process and memory management. Other components like device drivers are run in user space. Example: Minix.
* **Hybrid Kernels**: These are a mix of monolithic and microkernel architectures. Example: Windows NT kernel.

<mark style="color:blue;">Examples of Kernels</mark>

* **Linux Kernel**: Used in Linux distributions.
* **Windows NT Kernel**: Used in various versions of Microsoft Windows.
* **XNU Kernel**: Used in macOS and iOS.

</details>

### <mark style="color:blue;">Check GNU Compiler Compatibility</mark>

NVIDIA CUDA Libraries work in conjunction with <mark style="color:blue;">**GCC (GNU Compiler Collection)**</mark> on Linux systems.&#x20;

GCC is commonly used for compiling the <mark style="color:blue;">**host (CPU)**</mark> part of the code, while CUDA tools like nvcc (NVIDIA CUDA Compiler) are used for compiling the <mark style="color:blue;">**device (GPU)**</mark> part of the code.

The CUDA Toolkit includes wrappers and libraries that *<mark style="color:yellow;">**facilitate the integration between the CPU and GPU parts of the code.**</mark>* &#x20;

NVIDIA provides compatibility information for specific versions of GCC, especially on Linux systems where GCC is a common choice for compiling the host code.

The CUDA runtime libraries, which are installed separately, are sufficient for running CUDA applications on systems with compatible NVIDIA GPUs.

{% hint style="warning" %}
The <mark style="color:blue;">**gcc compiler**</mark> is required for development using the CUDA Toolkit
{% endhint %}

To reiterate - when developing applications that use both CPU and GPU, <mark style="color:yellow;">developers might use GCC for compiling the CPU part of the code</mark>, while <mark style="color:yellow;">CUDA tools (like nvcc - NVIDIA CUDA Compiler) are used for compiling the GPU part</mark>.

The CUDA toolkit often includes compatibility information with specific versions of GCC, especially on Linux systems, where GCC is a common choice for compiling the host code.

Run the following command to check the <mark style="color:blue;">installed version of GCC:</mark>

```bash
gcc --version
```

The first line of the output will show the version number. Ensure it matches the default GCC version listed in your table for your Ubuntu version.

<pre><code><strong>gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 
</strong>9.4.0  &#x3C;--- This is the GCC version number
</code></pre>

&#x20;If you <mark style="color:yellow;">**do not**</mark> have GCC installed, execute the following:

<details>

<summary><mark style="color:green;">Installation of GCC via installing 'build essentials'</mark></summary>

The <mark style="color:yellow;">`build-essential`</mark> meta-package in Ubuntu is a collection of tools and packages needed for <mark style="color:yellow;">compiling and building software.</mark>&#x20;

This package is particularly useful for developers and those compiling software from source. Here's a detailed summary of each package included in `build-essential` and information about where these packages are typically stored on Ubuntu 20.04:

### <mark style="color:purple;">**dpkg-dev**</mark>

* **Purpose**: This package is a collection of <mark style="color:yellow;">development tools required to handle Debian (.deb) packages.</mark> It includes utilities to unpack, build, and upload Debian source packages, making it an essential tool for packaging software for Debian-based systems like Ubuntu.
* **Storage Location**: The tools and scripts from <mark style="color:yellow;">`dpkg-dev`</mark> are usually stored in <mark style="color:yellow;">`/usr/bin/`</mark> and <mark style="color:yellow;">`/usr/share/dpkg/`</mark><mark style="color:yellow;">.</mark>

### <mark style="color:purple;">**make**</mark>

* **Purpose**: `make` is a build automation tool that automatically builds executable programs and libraries from source code by reading files called Makefiles.  It's crucial for compiling large programs where it manages dependencies and only recompiles parts of the program that have changed.
* **Storage Location**: The `make` executable is typically found in <mark style="color:yellow;">`/usr/bin/make`</mark><mark style="color:yellow;">.</mark>

### <mark style="color:purple;">**libc6-dev**</mark>

* **Purpose**: This package contains the development libraries and header files for the GNU C Library. It's essential for <mark style="color:yellow;">compiling C and C++ programs</mark>, as it includes standard libraries and headers.
* **Storage Location**: The headers and libraries are generally located in `/usr/include/` and `/usr/lib/` respectively.

### <mark style="color:purple;">**gcc/g++**</mark>

* **Purpose**: These are the GNU Compiler Collection for C and C++ languages. <mark style="color:yellow;">`gcc`</mark> is for compiling C programs, while <mark style="color:yellow;">`g++`</mark> is used for C++ programs. They are fundamental for software development in these languages.
* **Storage Location**: The compilers are usually found in <mark style="color:yellow;">`/usr/bin/`</mark><mark style="color:yellow;">.</mark>

When you install the <mark style="color:yellow;">`build-essential`</mark> package on Ubuntu, it automatically installs these components and their dependencies. This package streamlines the setup process for a development environment by bundling these critical tools together.

To install <mark style="color:yellow;">`build-essential`</mark> on Ubuntu 20.04, you can use the following command in the terminal:

```bash
sudo apt update
sudo apt install build-essential
```

This command will download and install the <mark style="color:yellow;">`build-essential`</mark> package along with its dependencies. The packages are typically stored in the locations mentioned above, following the standard file system hierarchy of Linux systems. This structure helps in maintaining a standardized path for binaries, libraries, and other files, making it easier for users and other software to locate them.

</details>

Post installation of build essentials, check the GCC version you have:

```
gcc --version
```

The output should <mark style="color:blue;">now prove you have GCC installed:</mark>

<pre class="language-bash"><code class="lang-bash">gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) <a data-footnote-ref href="#user-content-fn-2">9.4.0</a>
</code></pre>

This version 9.4 should work with CUDA 12.1, which requires at least 9.3.   GCC is considered 'backward compatible', so this version of 9.4 should be fine.&#x20;

<details>

<summary><mark style="color:green;">What is GCC and why is the version important?</mark></summary>

GCC is a <mark style="color:purple;">**collection of compilers for various programming languages**</mark>**.**&#x20;

Although it started primarily for C (hence the original name GNU C Compiler), it now supports C++, Objective-C, Fortran, Ada, Go, and D.

<mark style="color:blue;">**Cross-Platform Compatibility**</mark>

GCC can be used on many different types of operating systems and hardware architectures. This cross-platform capability makes it a versatile tool for developers who work in diverse environments.

<mark style="color:blue;">**Optimization and Portability**</mark>

GCC offers a wide range of options for code optimization, making it possible to <mark style="color:purple;">tune performance for specific hardware or application requirements</mark>. It also emphasizes portability, enabling developers to compile their code on one machine and run it on another without modification.

<mark style="color:blue;">**Standard Compliance**</mark>

GCC strives to adhere closely to various programming language standards, including those for C and C++. This compliance ensures that code written and compiled with GCC is compatible with other compilers following the same standards.

<mark style="color:blue;">**Debugging and Error Reporting**</mark>

GCC is known for its helpful debugging features and detailed error reporting, which are invaluable for developers in identifying and fixing code issues.

<mark style="color:blue;">**Integration with Development Tools**</mark>

GCC easily integrates with various development tools and environments. It's commonly used in combination with IDEs, debuggers, and other tools, forming a complete development ecosystem.

</details>

### <mark style="color:blue;">Check GLIBC Compatibility</mark>

The GNU C Library, commonly known as glibc, is an important component of GNU systems and Linux distributions.&#x20;

GLIBC is the GNU Project's implementation of the C standard library.   It provides the system's core libraries. This includes facilities for basic file I/O, string manipulation, mathematical functions, and various other standard utilities.

To check the <mark style="color:blue;">GLIBC version:</mark>

```bash
ldd --version
```

The first line of the output will show the version number. For example:

```bash
ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31
```

The output should be:

```bash
ldd (Ubuntu GLIBC 2.31-0ubuntu9.12) 2.31  <--This is the version
```

Compare this with the GLIBC version in your table.  &#x20;

The <mark style="color:yellow;">GLIBC version of 2.31</mark> is the same as the version required for the NVIDIA CUDA Toolkit

<details>

<summary><mark style="color:green;">What is GLIBC?</mark></summary>

1. **Definition**: GLIBC is the <mark style="color:yellow;">GNU Project's implementation of the C standard library</mark>. Despite its name, it now also directly supports C++ (and indirectly other programming languages).
2. **Purpose**: It provides the system's core libraries. This includes facilities for basic file I/O, string manipulation, mathematical functions, and various other standard utilities.
3. **Compatibility**: It's designed to be compatible with the POSIX standard, the Single UNIX Specification, and several other open standards, while also extending them in various ways.
4. **System Calls and Kernel**: glibc serves as a wrapper for system calls to the Linux kernel and other essential functions. This means that most applications on a Linux system depend on glibc to interact with the underlying kernel.
5. **Portability**: It's used in systems that range from embedded systems to servers and supercomputers, providing a consistent and reliable base across various hardware architectures.

<mark style="color:blue;">**Checking GLIBC Version**</mark>

To check the version of glibc on a Linux system, you can use the <mark style="color:yellow;">`ldd`</mark> command, which prints the shared library dependencies. The version of glibc will be displayed as part of this output. Here's how to do it:

**Run the Command**: Type the following command and press Enter:

```bash
ldd --version
```

The first line of the output will typically show the glibc version. For example, it might say `ldd (Ubuntu GLIBC 2.31-0ubuntu9.2) 2.31`, where "2.31" is the version of glibc.

#### <mark style="color:blue;">Importance in Development</mark>

1. **Compatibility**: When developing software for Linux, it's crucial to know the version of glibc your application will be running against, as different versions may have different features and behaviors.
2. **Portability**: For applications intended to run on multiple Linux distributions, understanding glibc compatibility is key to ensuring broad compatibility.
3. **System-Level Programming**: For low-level system programming, knowledge of glibc is essential as it provides the interface to many kernel-level services and system resources.
4. **Debugging**: Understanding glibc can be crucial for debugging, especially for complex applications that perform a lot of system-level operations.

</details>

#### With the NVIDIA CUDA Toolkit's compatibility with host installations, the next step is to do a check for compatibility

### <mark style="color:blue;">Process for checking installations have been successful</mark>

First, <mark style="color:yellow;">check your Ubuntu version</mark>. Ensure it matches <mark style="color:blue;">Ubuntu 20.04,</mark> which is our designated Linux operating system

```bash
lsb_release -a
```

Then, verify that your system is based on the x86\_64 architecture. Run:

```bash
uname -m
```

The output should be:

```
x86_64
```

To check if your system has a <mark style="color:blue;">**CUDA-capable NVIDIA GPU**</mark>**,** run

```bash
nvidia-smi
```

You should see an output like this, which details the <mark style="color:blue;">NVIDIA Drivers installed</mark> and the <mark style="color:blue;">CUDA Version</mark>.

<pre><code>+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    <a data-footnote-ref href="#user-content-fn-3">CUDA Version: 12.1</a>     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  <a data-footnote-ref href="#user-content-fn-4">NVIDIA A100-SXM4-80GB</a>           On | 00000000:00:05.0 Off |                    0 |
| N/A   36C    P0               56W / 400W|      4MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1314      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+
</code></pre>

If this output is not visible, we must <mark style="color:blue;">**install the NVIDIA Drivers**</mark>

### <mark style="color:blue;">A full analysis</mark>

To do this all at once...

If you would like a full printout of your system features, enter this command into the terminal:

```bash
echo "Machine Architecture: $(uname -m)" && \
echo "Kernel Name: $(uname -s)" && \
echo "Kernel Release: $(uname -r)" && \
echo "Kernel Version: $(uname -v)" && \
echo "Hostname: $(uname -n)" && \
echo "Operating System: $(uname -o)" && \
echo "----" && \
cat /proc/version && \
echo "----" && \
echo "CPU Information:" && cat /proc/cpuinfo | grep 'model name' | uniq && \
echo "----" && \
echo "Memory Information:" && cat /proc/meminfo | grep 'MemTotal' && \
echo "----" && \
lsb_release -a 2>/dev/null && \
echo "----" && \
echo "NVCC Version:" && nvcc --version
```

The output from the terminal will provide you all the information necessary to check system information for compatibility.

<details>

<summary><mark style="color:green;">Typical analysis of the output from a A-100 80GB instance</mark></summary>

<mark style="color:blue;">**Machine Architecture: x86\_64**</mark>

* Your system uses the 64-bit version of the x86 architecture. This is a standard architecture for modern desktops and servers, supporting more memory and larger data sizes compared to 32-bit systems.

<mark style="color:blue;">**Kernel Details**</mark>

* **Kernel Name**: Linux, indicating that your operating system is based on the Linux kernel.
* **Kernel Release**: 5.4.0-167-generic. This specifies the version of the Linux kernel you are running. 'Generic' here implies a standard kernel version that is versatile for various hardware setups.
* **Kernel Version**: #184-Ubuntu SMP. This shows a specific build of the kernel, compiled with Symmetric Multi-Processing (SMP) support, allowing efficient use of multi-core processors. The timestamp shows the build date.

<mark style="color:blue;">**Hostname**</mark><mark style="color:blue;">:</mark> ps1rgbvhl

* This is the network identifier for your machine, used to distinguish it in a network environment.

<mark style="color:blue;">**Operating System**</mark><mark style="color:blue;">:</mark> GNU/Linux

* This indicates that you're using a GNU/Linux distribution, a combination of the Linux kernel with GNU software.

<mark style="color:blue;">**Detailed Kernel Version**</mark>

* This reiterates your kernel version and build details. It also mentions the GCC version used for building the kernel (9.4.0), which affects compatibility with certain software.

<mark style="color:blue;">**CPU Information**</mark><mark style="color:blue;">:</mark> Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz

* The system is powered by an Intel Xeon Gold 6342 processor, which is a high-performance, server-grade CPU. The 2.80 GHz frequency indicates its base clock speed.

<mark style="color:blue;">**Memory Information**</mark><mark style="color:blue;">:</mark> MemTotal: 92679772 kB

* The system has a substantial amount of RAM (approximately 92.68 GB). This is a significant size, suitable for memory-intensive applications and multitasking.

<mark style="color:blue;">**Ubuntu Distribution Information**</mark>

* **Distributor ID**: Ubuntu. This shows the Linux distribution you're using.
* **Description**: Ubuntu 20.04.6 LTS, indicating the specific version and that it's a Long-Term Support (LTS) release.
* **Release**: 20.04, the version number.
* **Codename**: focal, the internal codename for this Ubuntu release.

<mark style="color:blue;">**NVCC Version**</mark>

* The output details the version of the NVIDIA CUDA Compiler (NVCC) as 12.1, built in February 2023. NVCC is a key component for compiling CUDA code, essential for developing applications that leverage NVIDIA GPUs for parallel processing tasks.

In summary, the output paints a picture of a powerful, 64-bit Linux system with a high-performance CPU and a significant amount of RAM, running an LTS version of Ubuntu.&#x20;

The presence of the NVCC with CUDA version 12.1 indicates readiness for CUDA-based development, particularly in fields like data science, machine learning, or any computationally intensive tasks that can benefit from GPU acceleration.

</details>

### <mark style="color:blue;">Installation of .NET SDK -  required for Polyglot Notebooks</mark>

<details>

<summary><mark style="color:green;">Installation of .NET</mark></summary>

.NET is a free, open-source, and cross-platform framework developed by Microsoft.&#x20;

It is used for building various types of applications, including web applications, desktop applications, cloud-based services, and more. .NET provides a rich set of libraries and tools for developers to create robust and scalable software solutions.

#### <mark style="color:green;">Add the Microsoft package repository</mark> <a href="#add-the-microsoft-package-repository" id="add-the-microsoft-package-repository"></a>

Installing with APT can be done with a few commands. Before you install .NET, run the following commands to add the Microsoft package signing key to your list of trusted keys and add the package repository.

Open a terminal and run the following commands:

```bash
wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb
```

#### <mark style="color:green;">Install the SDK</mark> <a href="#install-the-sdk" id="install-the-sdk"></a>

The .NET SDK allows you to develop apps with .NET. If you install the .NET SDK, you don't need to install the corresponding runtime. To install the .NET SDK, run the following commands:

```bash
sudo apt-get update && \
  sudo apt-get install -y dotnet-sdk-8.0
```

#### <mark style="color:green;">Install the runtime</mark> <a href="#install-the-runtime" id="install-the-runtime"></a>

The ASP.NET Core Runtime allows you to run apps that were made with .NET that didn't provide the runtime. The following commands install the ASP.NET Core Runtime, which is the most compatible runtime for .NET. In your terminal, run the following commands:

```bash
sudo apt-get update && \
  sudo apt-get install -y aspnetcore-runtime-8.0
```

As an alternative to the ASP.NET Core Runtime, you can install the .NET Runtime, which doesn't include ASP.NET Core support: replace `aspnetcore-runtime-8.0` in the previous command with `dotnet-runtime-8.0`:

```bash
sudo apt-get install -y dotnet-runtime-8.0
```

</details>

<details>

<summary><mark style="color:yellow;">If you want to change the GCC Version in your environment</mark></summary>

You can change your GCC version in a Conda environment.  &#x20;

Here's how you can change the GCC version in a Conda environment:

**Create a New Conda Environment (Optional)**

If you don't already have a specific environment for your CUDA work, create one:

```bash
conda create -axolotl python=3.10
```

**Activate the Conda Environment**:

```bash
conda activate axolotl
```

**Install a Specific GCC Version**:

```bash
conda install gcc_linux-64=gcc_version
```

Replace `gcc_version` with the version of GCC you need, for example, `9.4.0`.

**Verify GCC Version**:

```bash
gcc --version
```

**Install CUDA Toolkit (if needed)**:

* If you haven't installed CUDA in your environment, you can do so using Conda (if available) or follow the CUDA Toolkit's installation guide:

```bash
conda install cudatoolkit=x.x
```

* Replace `x.x` with the version of the CUDA Toolkit you need.

</details>

<details>

<summary><mark style="color:yellow;">If you want to change the version of CUDA being used in your environment</mark></summary>

The Conda installation for CUDA is an efficient way to install and manage the CUDA Toolkit, especially when working with Python environments.&#x20;

<mark style="color:blue;">**Conda Overview**</mark>

* Conda can facilitate the installation of the CUDA Toolkit.

<mark style="color:blue;">**Installing CUDA Using Conda**</mark>

* Basic installation command: `conda install cuda -c nvidia`.
* This command installs all components of the CUDA Toolkit.

<mark style="color:blue;">**Uninstalling CUDA Using Conda**</mark>

* Uninstallation command: `conda remove cuda`.
* It removes the CUDA Toolkit installed via Conda.
* Special Tip: After uninstallation, check for any residual files or dependencies that might need manual removal.

<mark style="color:blue;">**Installing Previous CUDA Releases**</mark>

* Install specific versions using: `conda install cuda -c nvidia/label/cuda-<version>`.
* Replace `<version>` with the desired CUDA version (e.g., `11.3.0`).
* Special Tip: Installing previous versions can be crucial for compatibility with certain applications or libraries. Always check version compatibility with your project requirements.

<mark style="color:blue;">**Practical Example: Installing CUDA Toolkit**</mark><mark style="color:blue;">:</mark>

Create a virtual environment

* <mark style="color:yellow;">`conda install -c "nvidia/label/cuda-11.8.0" cuda-nvcc`</mark><mark style="color:yellow;">:</mark> Installs the NVIDIA CUDA Compiler (nvcc) from the specified NVIDIA channel on Conda. This is aligned with the CUDA version 11.8.0, ensuring compatibility with the specific version of PyTorch being used.
* <mark style="color:yellow;">`conda install -c anaconda cmake`</mark><mark style="color:yellow;">:</mark> Installs CMake, a cross-platform tool for managing the build process of software using a compiler-independent method.
* <mark style="color:yellow;">`conda install -c anaconda cmake`</mark><mark style="color:yellow;">:</mark> Installs 'lit', a tool for executing LLVM's integrated test suites.

**Additional Tools Installation (Optional)**:

* <mark style="color:yellow;">`conda install -c anaconda cmake`</mark><mark style="color:yellow;">:</mark> Installs CMake, a cross-platform tool for managing the build process of software using a compiler-independent method.
* <mark style="color:yellow;">`conda install -c conda-forge lit`</mark><mark style="color:yellow;">:</mark> Installs 'lit', a tool for executing LLVM's integrated test suites.

<mark style="color:blue;">**Installing PyTorch and Related Libraries**</mark>

* The <mark style="color:yellow;">`pip install`</mark> command is used to install specific versions of PyTorch (`torch`), along with its sister libraries `torchvision` and `torchaudio`. The `--index-url` specifies the PyTorch wheel for CUDA 11.8, ensuring that the installed PyTorch version is compatible with CUDA 11.8.
* These commands add a new PPA (Personal Package Archive) for Ubuntu toolchain tests and install GCC 11 and G++ 11. These are needed for building certain components that require C++ compilation, particularly for `deepspeed`, a deep learning optimization library.

### <mark style="color:green;">**Checking to see whether the revised version of CUDA is installed**</mark>

<mark style="color:blue;">**CUDA in Conda Environments**</mark>

* When you create a Conda environment and install a specific version of CUDA (like 11.8 in your case), you are installing CUDA toolkit libraries that are <mark style="color:purple;">**compatible with that version within that environment**</mark>**.**
* This installation <mark style="color:purple;">**does not change the system-wide CUDA version**</mark>, nor does it affect what `nvidia-smi` displays.
* The Conda environment's CUDA version is used by the programs and processes running within that environment. <mark style="color:purple;">It's independent of the system-wide CUDA installation.</mark>

<mark style="color:blue;">**Verifying CUDA Version in Conda Environment**</mark>

* To check the CUDA version in your Conda environment, you <mark style="color:purple;">should not rely on</mark> <mark style="color:yellow;">`nvidia-smi`</mark>. Instead, you can check the version of the CUDA toolkit you have installed in your environment. This can typically be done by checking the version of specific CUDA toolkit packages installed in the environment, like `cudatoolkit`.
* You can use a command like <mark style="color:yellow;">`conda list cudatoolkit`</mark> within your Conda environment to see the installed version of the CUDA toolkit in that environment.

<mark style="color:blue;">**Compatibility**</mark>

* It's important to ensure that the CUDA toolkit version within your Conda environment is compatible with the version supported by your NVIDIA driver (as indicated by `nvidia-smi`). <mark style="color:red;">If the toolkit version in your environment is</mark> <mark style="color:red;"></mark>*<mark style="color:red;">**higher**</mark>* <mark style="color:red;"></mark><mark style="color:red;">than the driver's supported version, you may encounter compatibility issues</mark>.

In summary, <mark style="color:yellow;">`nvidia-smi`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">shows the maximum CUDA version supported by your GPU's driver,</mark> not the version used in your current Conda environment. To check the CUDA version in a Conda environment, use Conda-specific commands to list the installed packages and their versions.

Another way of putting it:

1. **CUDA Driver Version:** <mark style="color:yellow;">The version reported by</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`nvidia-smi`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">is the CUDA driver version installed on your system</mark>, which is 12.3 in your case. This is the version of the driver software that allows your operating system to communicate with the NVIDIA GPU.
2. **CUDA Toolkit Version in PyTorch:** <mark style="color:yellow;">When you install PyTorch with a specific CUDA toolkit</mark> version (like `cu118` for CUDA 11.8), <mark style="color:yellow;">it refers to the version of the CUDA toolkit libraries that PyTorch uses for GPU acceleration</mark>. PyTorch packages these libraries with itself, so it does not rely on the system-wide CUDA toolkit installation.
3. **Compatibility:** The key point is compatibility. Your system's CUDA driver version (12.3) is newer and compatible with the CUDA toolkit version used by PyTorch (11.8). Generally, a newer driver version can support older toolkit versions without issues.
4. **Functionality Check:** <mark style="color:yellow;">As long as</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`torch.cuda.is_available()`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">returns</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`True`</mark>, it indicates that <mark style="color:yellow;">PyTorch is able to interact with your GPU using its bundled CUDA libraries</mark>, and you should be able to run CUDA-accelerated PyTorch operations on your GPUs.

In summary, your setup is fine for running PyTorch with GPU support. The difference in the CUDA driver and toolkit versions is normal and typically not a problem as long as the driver version is equal to or newer than the toolkit version required by PyTorch.

</details>

### <mark style="color:blue;">**Test Compatibility**</mark>

Below are some scripts to create to test for compatibility. &#x20;

These scripts will test that both your CPU and GPU are correctly processing the CUDA code.  It will also test to make sure there are no compatibility issues between the installed GCC version and the CUDA Toolkit version you are using.

<details>

<summary><mark style="color:blue;">Compatibility Test Scripts</mark></summary>

To test the compatibility of your GCC version with the CUDA Toolkit version installed, you can use a simple CUDA program. Below is a basic script for a CUDA program that performs a simple operation on the GPU. This script will help you verify that your setup is correctly configured for CUDA development.

First, create a simple CUDA program. Let's call it <mark style="color:yellow;">`test_cuda.cu`</mark>:

```cpp
#include <stdio.h>
#include <cuda_runtime.h>

// Kernel function to add two vectors
__global__ void add(int n, float *x, float *y) {
    int index = blockIdx.x * blockDim.x + threadIdx.x;
    int stride = blockDim.x * gridDim.x;

    for (int i = index; i < n; i += stride)
        y[i] = x[i] + y[i];
}

int main(void) {
    int N = 1<<25; // 33.6M elements

    float *x, *y;
    cudaEvent_t start, stop;

    cudaMallocManaged(&x, N*sizeof(float));
    cudaMallocManaged(&y, N*sizeof(float));

    for (int i = 0; i < N; i++) {
        x[i] = 1.0f;
        y[i] = 2.0f;
    }

    cudaEventCreate(&start);
    cudaEventCreate(&stop);
    cudaEventRecord(start);

    int blockSize = 256;
    int numBlocks = (N + blockSize - 1) / blockSize;
    add<<<numBlocks, blockSize>>>(N, x, y);

    cudaEventRecord(stop);
    cudaEventSynchronize(stop);

    float milliseconds = 0;
    cudaEventElapsedTime(&milliseconds, start, stop);
    printf("Time taken: %f ms\n", milliseconds);

    float maxError = 0.0f;
    for (int i = 0; i < N; i++)
        maxError = fmax(maxError, fabs(y[i]-3.0f));
    printf("Max error: %f\n", maxError);

    cudaEventDestroy(start);
    cudaEventDestroy(stop);
    cudaFree(x);
    cudaFree(y);
    
    return 0;
}

```

Next, create a shell script to compile and run this CUDA program. Name this script `test_cuda_compatibility.sh`:

```bash
#!/bin/bash

# Define the CUDA file
cuda_file="test_cuda.cu"

# Define the output executable
output_executable="test_cuda_executable"

# Compile the CUDA program
nvcc $cuda_file -o $output_executable

# Check if the compilation was successful
if [ $? -eq 0 ]; then
    echo "Compilation successful. Running the CUDA program..."
    ./$output_executable
else
    echo "Compilation failed."
fi
```

This script compiles the <mark style="color:yellow;">`test_cuda.cu`</mark> file using <mark style="color:yellow;">`nvcc`</mark>, the NVIDIA CUDA compiler, and then runs the compiled executable if the compilation is successful.

**How to Use the Script:**

1. Save the CUDA program code in a file named <mark style="color:yellow;">`test_cuda.cu`</mark><mark style="color:yellow;">.</mark>
2. Save the shell script in a file named <mark style="color:yellow;">`test_cuda_compatibility.sh`</mark><mark style="color:yellow;">.</mark>
3. Make the shell script executable:

```bash
chmod +x test_cuda_compatibility.sh
```

Run the script:

```bash
./test_cuda_compatibility.sh
```

If everything is set up correctly, the script will compile the CUDA program and run it, resulting in output from both the CPU and GPU.&#x20;

If there are compatibility issues between GCC and the CUDA Toolkit, the script will likely fail during compilation, and you'll see error messages indicating what went wrong.

</details>

<mark style="color:red;">**Remember:**</mark> Compatibility between the GCC version and the CUDA Toolkit is crucial. Make sure the GCC version you choose is compatible with your CUDA Toolkit version.

### <mark style="color:blue;">Where are you now?</mark>

We have now created a deep learning development environment optimised for NVIDIA GPUs, with compatibility across key components.

We have so far:

**-Installed CUDA Toolkit and Drivers**

**-Set up the NVIDIA Container Toolkit to allow access to NVIDIA Docker containers**

**-Ensured Host Compatibility by verifying** variables such as GCC (GNU Compiler Collection) and GLIBC (GNU C Library) are compatible with the CUDA version.

**-Created a Compatibility Check Script:**  Developing a script to check for compatibility issues&#x20;

With these components in place, your environment is tailored for deep learning development. It supports the development and execution of deep learning models, leveraging the computational power of GPUs for training and inference tasks.

### <mark style="color:blue;">==>  With the environment established for NVIDIA GPUS, the next step is creating the virtual environment for Axolotl and installing the code base</mark>

[^1]: This is the version of CUDA

[^2]: This is the version number

[^3]: This is the CUDA Version

[^4]: This is the virtual machine


# Platform Installation

The next step after ensuring NVIDIA Drivers and Enviornment are stable and compatible

With the virtual machine established and optimised, we will now establish the virtual environment for Axolotl

## <mark style="color:blue;">Download the axolotl library</mark>

```bash
git clone https://github.com/OpenAccess-AI-Collective/axolotl.git
```

Then navigate into the library:

```bash
cd axolotl
```

<details>

<summary>Reference: <mark style="color:green;">What does git clone do?</mark></summary>

The command <mark style="color:yellow;">`git clone`</mark> is a command-line instruction used with Git, a version control system, to create a local copy of a remote repository. Let's break down this command to understand each part:

1. <mark style="color:yellow;">**`git clone`**</mark>:
   * <mark style="color:yellow;">`git`</mark>: This is the command-line tool for using Git.
   * <mark style="color:yellow;">`clone`</mark>: A Git command used to copy a remote repository. It not only downloads the content of the repository but also creates a new local repository with all the history and branch information.
2. **Repository URL**
   * <mark style="color:yellow;">`https://github.com/OpenAccess-AI-Collective/axolotl.git`</mark>: This is the URL of the remote Git repository.
   * In this case, the repository is hosted on GitHub, a web-based platform for version control using Git.
   * The repository belongs to a user or organization named `OpenAccess-AI-Collective`, and the specific repository is named `axolotl`.
3. **Executing the Command**
   * When you execute this command in a terminal or command prompt, Git will:
     * Reach out to the URL provided.
     * Download the contents of the repository, including all the files, folders, branches, and history (commits).
     * Create a new directory in your current working directory with the same name as the repository (`axolotl` in this case).
     * Initialize a local Git repository in that directory and link it to the remote repository. This link is important for future commands like `git pull` (to fetch updates) and `git push` (to upload local changes).
4. **Post-Cloning**
   * After cloning, you can navigate into the newly created directory (`cd axolotl`) and start working with the files. You can also switch between branches, update the local repository with changes from the remote repository, and commit new changes.
5. **Purpose**
   * Cloning is often the first step when you want to work on a project from a remote repository, contribute to it, or just have a local copy for reference or backup.

In summary, `git clone https://github.com/OpenAccess-AI-Collective/axolotl.git` is a Git command to create a local copy of the `axolotl` repository from the `OpenAccess-AI-Collective` GitHub page. It sets up a new folder with all the repository data and establishes a connection between the local and remote repositories for ongoing version control.

</details>

### <mark style="color:blue;">Virtual Environment</mark>

<mark style="color:blue;">**Check for Anaconda**</mark><mark style="color:blue;">:</mark>

* Ensure Anaconda is installed and create and <mark style="color:yellow;">Anaconda environment</mark> called <mark style="color:yellow;">'axolotl'</mark>
* Type the following command into the terminal to check if Anaconda is installed and press Enter:

```bash
conda --version
```

If Anaconda is installed, this command will return the version number:

```bash
conda 23.10.0
```

* If you get an error or message saying that <mark style="color:yellow;">`conda`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">is not recognised,</mark> it means Anaconda is not installed.

If Anaconda is not installed - <mark style="color:blue;">follow the installation instructions below</mark> <mark style="color:blue;"></mark><mark style="color:blue;">**"Installing Miniconda":**</mark>

<details>

<summary><mark style="color:green;">Installing Miniconda</mark></summary>

<mark style="color:blue;">**Overview**</mark>

* **Miniconda** is a minimal installer for the Conda package manager. It provides a lightweight alternative to Anaconda, including only Conda, Python, and essential packages they depend on, plus a few other useful packages such as pip and zlib.
* Ideal for users who prefer a smaller footprint or want more control over installed packages.
* Additional packages can be installed using <mark style="color:yellow;">`conda install`</mark> from Anaconda’s public repository or other channels like conda-forge or bioconda.

<mark style="color:blue;">**Choosing Miniconda**</mark>

* The decision between Anaconda and Miniconda depends on user needs. The Anaconda or Miniconda page provides guidance for choosing the most suitable installation.

<mark style="color:blue;">**System Requirements**</mark>

* System requirements are available on the Miniconda documentation page, ensuring compatibility with user systems.

<mark style="color:blue;">**Installation Links and Release Notes**</mark>

* The latest Miniconda installer links are provided for Python 3.11.5 as of November 16, 2023.
* Installers are available for various platforms, including Windows, macOS (Intel and Apple M1), and Linux (with multiple architecture support).
* SHA256 hashes are provided for verifying the integrity of the downloaded files.
* For older Python versions or an archive of Miniconda versions, links are available on the documentation page.

<mark style="color:blue;">**Quick Installation (Command Line)**</mark>

* Quick installation instructions are provided for a streamlined setup process.
* For Linux, a typical installation involves moving to your home directory and then:

<mark style="color:green;">**Create a directory**</mark>

```bash
mkdir -p ~/miniconda3
```

<mark style="color:green;">**Downloading the installer:**</mark>

{% code overflow="wrap" %}

```bash
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
```

{% endcode %}

<mark style="color:green;">**Running the installer**</mark>

```bash
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
```

<mark style="color:green;">**Cleaning up**</mark>

```bash
rm -rf ~/miniconda3/miniconda.sh
```

* Post-installation, initializing Miniconda for bash and zsh shells is recommended:

<mark style="color:yellow;">`~/miniconda3/bin/conda init bash`</mark>

<mark style="color:green;">Additional Notes</mark>

* Graphical installer instructions and hash-checking guidelines are available in the Miniconda documentation.
* For specific system setups or advanced configurations, refer to the detailed Miniconda documentation.

This summary provides an overview of Miniconda’s key features, installation process, and system requirements, suitable for inclusion in GitBook documentation.

</details>

<mark style="color:blue;">**Create Environment**</mark>

* Axolotl <mark style="color:blue;">**requires**</mark> <mark style="color:blue;"></mark><mark style="color:blue;">Pytorch</mark> <mark style="color:yellow;">>=</mark><mark style="color:blue;">3.9</mark> <mark style="color:blue;">and Pytorch</mark> <mark style="color:yellow;">>=</mark><mark style="color:blue;">2.0</mark>
* We are going to use <mark style="color:yellow;">Python 3.10 - which is a compromise between the latest Python and the minimum Python environment required</mark>
* In this case, we want to create a conda environment with a specific version of Python, so <mark style="color:red;">**do not**</mark> use the `conda create --clone` command to <mark style="color:purple;">create a clone of the</mark> <mark style="color:purple;"></mark><mark style="color:purple;">**`base`**</mark><mark style="color:purple;">**&#x20;**</mark><mark style="color:purple;">**environment.**</mark>

<details>

<summary>When <mark style="color:yellow;">should</mark> you clone a base environment?</summary>

The `base` environment in Conda is the default environment that gets created when you install Conda. It is a special environment that <mark style="color:blue;">contains the Conda package manager itself, along with a collection of installed packages</mark>.&#x20;

#### <mark style="color:blue;">What is the Base Environment?</mark>

1. **Default Environment**:
   * The `base` environment is the default environment active upon the installation of Anaconda or Miniconda. It is where Conda itself, along with a set of pre-installed packages and Python, are located.
2. **Contains Conda**:
   * Crucially, the `base` environment includes the Conda package manager. This means you can use Conda to create and manage other environments from the `base` environment.
3. **Pre-installed Packages**:
   * When you install Anaconda, the `base` environment comes with a wide array of commonly used data science packages pre-installed. This makes it a ready-to-use environment for a variety of tasks.

#### <mark style="color:blue;">Why Do People Clone the Base Environment?</mark>

1. **Consistency**:
   * Cloning the `base` environment ensures a consistent starting point. You get a new environment with the same set of packages and configurations as the `base`, which is often a known, stable setup.
2. **Safety**:
   * Working directly in the `base` environment is generally discouraged because changes might affect the stability and functionality of the Conda system itself. Cloning it provides a safe playground where one can install, update, or remove packages without risking the integrity of the Conda installation.
3. **Ease of Setup**:
   * For users who want a quick setup that includes a broad range of pre-installed packages (like those found in Anaconda), cloning the `base` environment is a time-saver. It eliminates the need to manually install a long list of common packages.
4. **Replicability**:
   * Cloning can be used to replicate the `base` environment across different machines or for different users, ensuring that everyone is working with the same set of tools and libraries.
5. **Experimentation and Testing**:
   * Cloning the `base` environment to create a new one allows for experimentation and testing with different package versions or configurations. If something goes wrong, the `base` environment remains unaffected.

</details>

* Use the following command to create an environment named 'axolotl' the contains the base libraries

```bash
conda create -n axolotl python=3.10
```

Once the environment has been created, <mark style="color:yellow;">activate the environment:</mark>

```bash
conda activate axolotl
```

To ensure that there are <mark style="color:yellow;">no installed packages in the environment</mark>, enter the command:

```bash
pip3 freeze
```

To ensure we only download fresh copies of packages and do not use packages in the cache, enter this command. &#x20;

```bash
pip3 cache purge
```

Because the environment may be empty; we need to install lit and cmake to install from wheels.

The purpose of running `pip install lit cmake` is to <mark style="color:yellow;">set up a development environment with specific tools needed for compiling, building, testing, or packaging software</mark>, particularly in projects that might involve C/C++ code or require complex build configurations.

```bash
pip3 install lit cmake
```

The output should be similar to this:

```bash
Successfully installed cmake-3.29.2 lit-18.1.3
```

<details>

<summary><mark style="color:green;">What is 'wheels'?</mark>   <mark style="color:purple;">Python Package Format</mark></summary>

<mark style="color:blue;">**Definition**</mark><mark style="color:blue;">:</mark> A wheel is a built package format for Python, denoted with a `.whl` file extension. It's a packaging standard introduced by PEP 427, <mark style="color:yellow;">designed to replace the older</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`.egg`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">format.</mark>

<mark style="color:blue;">**Advantages**</mark><mark style="color:blue;">:</mark> Wheels are more efficient than the traditional <mark style="color:yellow;">**`setup.py`**</mark> script for installing Python packages.&#x20;

The key benefits include:

<mark style="color:blue;">**Faster Installation**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">Wheels are pre-built distributions</mark>, meaning <mark style="color:yellow;">the package does not need to be compiled on the end user's machine.</mark> This significantly speeds up the installation process, especially for packages that contain compiled extensions.

<mark style="color:blue;">**Consistency**</mark><mark style="color:blue;">:</mark> They provide a more consistent installation experience since they include pre-built binaries. This reduces issues stemming from variations in build environments.

<mark style="color:blue;">**Usage**</mark><mark style="color:blue;">:</mark> When you run <mark style="color:yellow;">`pip install package_name`</mark>, `pip` tries to find and download a wheel compatible with your platform and Python version. If it can't find a suitable wheel, it falls back to installing from the source distribution, which can be slower as it may require compiling code.

</details>

{% hint style="info" %}
If you <mark style="color:yellow;">make an error</mark> building your conda environment, you can <mark style="color:yellow;">delete it and start again.</mark>  Follow the instructions below to delete your conda environment and start again:
{% endhint %}

<details>

<summary><mark style="color:green;">How to delete a conda environment</mark></summary>

To delete a Conda environment, you can follow these steps:

Open your terminal

First deactivate any conda environment you may already be in:

```bash
conda deactivate
```

Activate the base environment (or any other environment you're not deleting) to ensure you don't accidentally delete the active environment. You can do this by running:

```csharp
conda activate base
```

To list all your Conda environments, use the following command:

```bash
conda env list
```

This will display a list of all the environments installed on your system along with their paths.  You should see the axolotl virtual environment.

To <mark style="color:purple;">delete the axolotl environment</mark>, use the following command:

```bash
conda env remove --name axolotl
```

If asked, confirm the deletion by <mark style="color:yellow;">typing</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`y`</mark> and pressing Enter when prompted.

The Conda environment will be deleted, and all its packages and dependencies will be removed from your system.

</details>

{% hint style="info" %}
Always make sure you select the Python interpreter in VSCode to the <mark style="color:yellow;">axolotl virtual environment</mark>
{% endhint %}

### <mark style="color:blue;">The Python Interpreter</mark>

Remember, when using Visual Studio Code with virtual environments, make sure that you ensure that you have <mark style="color:yellow;">set the Python Interpreter to your virtual environment</mark>

<mark style="color:green;">**Access the Command Palette**</mark>

Open VS Code, and from the menu, go to View > Command Palette, or use the keyboard shortcut Ctrl+Shift+P.

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FCxWedGhyirOuTMGRRcrM%2Fselect-interpreters-command.png?alt=media&amp;token=10d18c82-836e-4c80-b3fd-4c55301f1b5d" alt=""><figcaption></figcaption></figure>

<mark style="color:yellow;">The</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">**Python: Select Interpreter**</mark> command displays a list of available global environments, conda environments, and virtual environments.  &#x20;

The following image, for example, shows several Anaconda  installations along with a conda environment and a virtual environment (`env`) that's located within the workspace folder:

1. Open VS Code, and from the menu, go to <mark style="color:yellow;">View > Command Palette,</mark> or use the keyboard shortcut <mark style="color:yellow;">Ctrl+Shift+P.</mark>
2. In the Command Palette, type <mark style="color:yellow;">"Python: Select Interpreter"</mark> and select this command when it appears in the list.

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2Fs4pm1smyAZLaF7mw6dNQ%2Finterpreters-list.png?alt=media&amp;token=4e15b6b0-4098-49be-9c98-639e270060a6" alt=""><figcaption><p>Python: Select Interpreter</p></figcaption></figure>

{% hint style="warning" %}
Make sure you <mark style="color:yellow;">**select the axolotl virtual environment**</mark> we have created.
{% endhint %}

<details>

<summary>Reference: <mark style="color:green;">Python Interpreter</mark></summary>

To select an interpreter for Python and ensure your Conda virtual environment works properly in Visual Studio Code (VS Code):

**Access the Command Palette**

Open VS Code, and from the menu, go to View > Command Palette, or use the keyboard shortcut Ctrl+Shift+P.

**Use the Python: Select Interpreter Command**

In the Command Palette, type "Python: Select Interpreter" and select this command when it appears in the list. Alternatively, you can use the keyboard shortcut Ctrl+Shift+P to bring up the Command Palette and directly type "Python: Select Interpreter."

**Select an Interpreter**

Upon selecting the "Python: Select Interpreter" command, a list of available Python environments will be displayed. This list includes global environments, Conda environments, and virtual environments. You can choose from the listed environments. The list will look similar to the example shown in the documentation.

**Remember Your Selection**

**I**f you have a folder or workspace open in VS Code when you select an interpreter, the Python extension will remember your choice for that specific workspace. This means that the same interpreter will be used when you reopen that workspace in the future.

**Status Bar Confirmation**

After selecting an interpreter, the selected environment version will be <mark style="color:yellow;">displayed on the right side of the Status Bar in VS Code</mark>. This serves as a confirmation that the interpreter has been set.

**Optional**

**Prevent Automatic Activation:** By default, VS Code will <mark style="color:yellow;">automatically activate the selected environment when you open a terminal within VS Code</mark>. If you want to prevent this automatic activation, you can add `"python.terminal.activateEnvironment": false` to your settings.json file.

**Manually Specify an Interpreter (if needed)**

If <mark style="color:yellow;">VS Code doesn't automatically locate an interpreter you want to use, you can manually specify it.</mark> To do this, run the "Python: Select Interpreter" command and choose the "Enter interpreter path..." option from the top of the interpreters list. You can then either enter the full path to the Python interpreter directly or browse your file system to find it.

<mark style="color:yellow;">Selecting the Python interpreter in Visual Studio Code (VS Code) is necessary</mark> because there are <mark style="color:yellow;">various Python environments and installations that can coexist on a system.</mark>&#x20;

The reason why it can't always be done automatically is due to the complexity of managing and identifying Python environments, especially in cases where multiple versions or virtual environments are present. Here are some key reasons:

**User Preferences:** Developers may have their preferred Python interpreter for different types of projects. For example, they may want to use Python 3.8 for one project and Python 3.9 for another. Automatic selection might not align with their preferences.

**Path Variations:** Python interpreters can be located in various directories, and their paths may not always follow a consistent pattern. Manually selecting the interpreter allows users to specify the exact path.

**Conda Environments:** Conda is a package manager that creates isolated environments for projects. Detecting and managing Conda environments automatically can be complex due to the additional layer of isolation.

**Compatibility:** Python extensions and packages can have compatibility issues with certain Python versions. Manually selecting the interpreter allows users to ensure compatibility with their specific project requirements.

</details>

### <mark style="color:blue;">Pytorch Installation</mark>

### <mark style="color:yellow;">--> The environment should contain Pytorch 2.0</mark>

Go to the Pytorch website and select your variables - version, operating system, package (Conda), Language, Computer Platform.  It will provide you the command to run in your activated Conda environment.

<https://pytorch.org/get-started/locally/>

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FPdp9Ir2W3zlsnbvG13eB%2Fpytorch.png?alt=media&amp;token=c682bcb7-b81b-46c9-b771-650113f902a5" alt=""><figcaption><p>Enter in the required details</p></figcaption></figure>

Then take the output provided and go into your terminal and enter the command.  <mark style="color:yellow;">Ensure that you are within the conda environment 'axolotl'</mark> (conda activate axolotl) if you are not enter the command:

```bash
conda activate axolotl
```

The command below is for installation of <mark style="color:yellow;">Pytorch 2.1</mark> on a Linux terminal using Python and <mark style="color:yellow;">CUDA 12.1</mark>

{% code overflow="wrap" fullWidth="false" %}

```bash
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
```

{% endcode %}

<mark style="color:yellow;">Press (y)</mark> when asked to download the libraries suggested/

This is a breakdown of the command:

* <mark style="color:yellow;">**`conda install`**</mark><mark style="color:yellow;">:</mark> This indicates that you're using Conda
* <mark style="color:yellow;">**`pytorch`**</mark><mark style="color:yellow;">:</mark> This is the main PyTorch library
* <mark style="color:yellow;">**`torchvision`**</mark><mark style="color:yellow;">:</mark> A package for PyTorch that provides utilities for image and video processing
* <mark style="color:yellow;">**`torchaudio`**</mark><mark style="color:yellow;">:</mark> A package for PyTorch tailored for audio processing
* <mark style="color:yellow;">**`pytorch-cuda=12.1`**</mark><mark style="color:yellow;">:</mark> This specifies that you want the PyTorch build that is compatible with CUDA 12.1. &#x20;
* <mark style="color:yellow;">**`-c pytorch`**</mark><mark style="color:yellow;">:</mark> This tells Conda to install from the <mark style="color:yellow;">**`pytorch`**</mark> channel on Anaconda Cloud.
* <mark style="color:yellow;">**`-c nvidia`**</mark><mark style="color:yellow;">:</mark> This specifies to also use the <mark style="color:yellow;">**`nvidia`**</mark> channel, which is necessary for CUDA-related packages.&#x20;

### <mark style="color:green;">Instruction</mark>

When the pytorch installation command is executed, the terminal will ask if you want to install all of the Pytorch packages <mark style="color:yellow;">- reply yes.</mark>

By following these steps, you should be able to successfully install the Axolotl library on your Ubuntu 20.04 version.

<details>

<summary>Reference: <mark style="color:green;">How does Pytorch relate to CUDA?</mark></summary>

* <mark style="color:yellow;">PyTorch</mark> <mark style="color:blue;">**is an open-source machine learning library**</mark> for building and training neural networks.
* <mark style="color:yellow;">PyTorch</mark> <mark style="color:blue;">**uses CUDA to accelerate its operations on NVIDIA GPUs**</mark>. This acceleration is particularly crucial for training large and complex deep learning models, a process that can be computationally intensive and time-consuming.

<mark style="color:green;">**How They Work Together**</mark>

* <mark style="color:blue;">**Acceleration of Tensor Operations**</mark><mark style="color:blue;">:</mark> PyTorch performs a large number of operations on tensors, which are multi-dimensional arrays. *<mark style="color:yellow;">**CUDA accelerates these operations**</mark>* when executed on NVIDIA GPUs.
* <mark style="color:blue;">**Transparent Usage**</mark><mark style="color:blue;">:</mark> PyTorch *<mark style="color:yellow;">**abstracts the complexity of CUDA,**</mark>* making it easier for users. Developers can write standard PyTorch code, and if a CUDA-enabled GPU is available, PyTorch will automatically use CUDA to accelerate operations.
* <mark style="color:blue;">**CUDA Tensors**</mark><mark style="color:blue;">:</mark> PyTorch introduces 'CUDA tensors', similar to normal tensors but located in the GPU's memory. Operations on CUDA tensors are performed on the GPU, offering significant speed improvements.

<mark style="color:green;">**How to Think About Their Interaction**</mark>

* <mark style="color:blue;">**Complementary Roles**</mark><mark style="color:blue;">:</mark> Think of CUDA as an underlying engine that provides horsepower to PyTorch. While PyTorch handles the creation and manipulation of tensors and neural networks, CUDA provides the capability to execute these operations quickly on a GPU.
* <mark style="color:blue;">**Ease of Development**</mark><mark style="color:blue;">:</mark> From a developer’s perspective, CUDA’s complexities are mostly hidden. You write PyTorch code as usual, and PyTorch, coupled with CUDA, takes care of efficiently executing operations on the GPU.
* <mark style="color:blue;">**Scalability and Performance**</mark><mark style="color:blue;">:</mark> In environments where training large models or processing large datasets is required, the CUDA-PyTorch integration becomes critical. It allows leveraging GPU acceleration for better performance, reducing training time from days to hours or hours to minutes.
* <mark style="color:blue;">**Conditional Usage**</mark><mark style="color:blue;">:</mark> CUDA is utilized by PyTorch only if a compatible NVIDIA GPU is available. Otherwise, PyTorch defaults to using the CPU.

</details>

### <mark style="color:blue;">Install "Packaging" Package</mark>

Now install the <mark style="color:yellow;">**`packaging`**</mark> and <mark style="color:yellow;">**ninja**</mark> python packages.&#x20;

The <mark style="color:yellow;">**`packaging`**</mark> library provides utilities for version handling, specifiers, markers, requirements, tags, and more, which are often used in package management and installation.

<pre class="language-bash"><code class="lang-bash"><a data-footnote-ref href="#user-content-fn-1">pip3</a> install <a data-footnote-ref href="#user-content-fn-2">packaging</a> ninja
</code></pre>

### <mark style="color:blue;">**Run the Axolotl setup.py file**</mark>

* Ensure that you are in the <mark style="color:yellow;">base directory</mark> that contains the [`setup.py`](/creation-of-environment/setup.py-objectives) file.
* Run the installation command:

```bash
pip3 install -e '.[flash-attn,deepspeed]'
```

### <mark style="color:blue;">**`-e`**</mark><mark style="color:blue;">**&#x20;**</mark><mark style="color:blue;">**flag**</mark>

Stands for "editable" mode. When you install a package in editable mode, Python installs the package in a way that allows you to *<mark style="color:yellow;">**modify the package source code**</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">and see the changes directly without needing to reinstall the package</mark>*. This is particularly useful for development purposes.

<mark style="color:blue;">**`This is important`**</mark>

**`The`` `**<mark style="color:yellow;">**`dot "."`**</mark> <mark style="color:yellow;">represents the current directory</mark>. In this context, it indicates that <mark style="color:yellow;">`pip`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">should install the package located in the current directory.</mark>&#x20;

This directory contains the <mark style="color:yellow;">**`setup.py`**</mark> file, which is the build script for setuptools. It tells setuptools about your package (such as the name and version) and the files that belong to it.

<mark style="color:blue;">**`[flash-attn,deepspeed]`**</mark>

These are extras.&#x20;

Extras are *<mark style="color:yellow;">additional dependencies that are relevant for optional features of the package</mark>*.&#x20;

In this case, <mark style="color:yellow;">**`flash-attn`**</mark> and <mark style="color:yellow;">**`deepspeed`**</mark> are optional dependencies. When you specify these extras, <mark style="color:yellow;">**`pip`**</mark> will also install the dependencies associated with these features as defined in the package's <mark style="color:yellow;">**`setup.py`**</mark>.

This command will install the Axolotl package along with its dependencies as specified in <mark style="color:yellow;">**`setup.py`**</mark>`.`

{% hint style="info" %}
You can find a detailed explanation of the <mark style="color:blue;">**setup.py script**</mark> here:  [<mark style="color:yellow;">**setup.py decomposition**</mark>](/creation-of-environment/setup.py-objectives)
{% endhint %}

<details>

<summary><mark style="color:green;">setup.py installation process</mark></summary>

<mark style="color:green;">**Key Actions**</mark>

<mark style="color:blue;">**Installation Process**</mark><mark style="color:blue;">:</mark> The script is executing the installation of the Axolotl package. This involves several steps:

* <mark style="color:blue;">**Running Egg Info**</mark><mark style="color:blue;">:</mark> Gathering package information and writing it to the `axolotl.egg-info` directory.
* <mark style="color:blue;">**Building Distribution**</mark>: Creating a build directory and compiling the package into an 'egg' format, a type of Python distribution.
* <mark style="color:blue;">**Copying and Installing**</mark><mark style="color:blue;">:</mark> The compiled package is then copied to your Miniconda environment's `site-packages` directory.
* <mark style="color:blue;">**Processing Dependencies**</mark><mark style="color:blue;">:</mark> The script is also handling dependencies, downloading, and installing them as required. This includes packages like `gcsfs`, `s3fs`, `tensorboard`, `gradio`, `fschat`, `art`, `pynvml`, `scikit-learn`, `scipy`, `rouge-score`, and `evaluate`.

#### <mark style="color:green;">Warnings and Recommendations</mark>

1. <mark style="color:blue;">**Deprecation of**</mark><mark style="color:blue;">**&#x20;**</mark><mark style="color:blue;">**`setup.py install`**</mark><mark style="color:blue;">:</mark> The script warns that `setup.py install` is deprecated. This is a Setuptools deprecation warning, encouraging the use of more modern, standards-based tools like `pypa/build` and `pypa/installer`.&#x20;
   * **Implication**: While `setup.py install` still works, *<mark style="color:purple;">**it's recommended to use newer tools for package building and installation in future projects**</mark>* to stay aligned with Python packaging standards.
2. <mark style="color:blue;">**Deprecation of**</mark>**&#x20;**<mark style="color:yellow;">**`easy_install`**</mark>: Similar to the previous warning, there’s a notice about the deprecation of <mark style="color:yellow;">`easy_install`</mark>, a legacy package installation method.

#### <mark style="color:green;">Installation Summary</mark>

* The Axolotl library and its dependencies are being installed into your specified conda environment.
* The package is being built and installed using an older method (<mark style="color:yellow;">`setup.py install`</mark>), which, although operational, is not recommended due to deprecation warnings.
* The process involves the creation of a package distribution in the 'egg' format and the installation of various dependencies listed in the package's `requirements.txt` file.

</details>

### <mark style="color:blue;">Install Logging</mark>

We will be using Weights and Biases for training logging. &#x20;

Weights and Biases <mark style="color:yellow;">should already be installed,</mark> if it is not - to install Weights and Biases:

```bash
pip3 install wandb
```

<mark style="color:green;">**Login to Weights and Biases**</mark>

```
wandb login
```

When asked, complete the following fields for your Weights and Biases account:

<mark style="color:blue;">Username:</mark>&#x20;

<mark style="color:blue;">Password:</mark> &#x20;

<mark style="color:blue;">API Token:</mark>&#x20;

<details>

<summary>References: <mark style="color:green;">Weights and Biases</mark></summary>

\
To set up Weights and Biases (W\&B) for tracking experiments, follow these steps:

<mark style="color:blue;">**Create an Account and Install W\&B**</mark>

* Sign up for a free account at <https://wandb.ai/site> and log in to your W\&B account.
* Install the W\&B library on your machine in a Python 3 environment using pip:

```bash
pip install wandb
```

<mark style="color:blue;">**Log in to W\&B**</mark>

* In your Python script or notebook, import the W\&B Python SDK and log in using <mark style="color:yellow;">`wandb.login()`</mark>. You will be prompted to provide your API key.

```python
import wandb
wandb.login()
```

<mark style="color:blue;">**Start a Run and Track Hyperparameters**</mark>

* Initialise a W\&B Run object with <mark style="color:yellow;">`wandb.init()`</mark><mark style="color:yellow;">.</mark> You can specify project information and track hyperparameters in the configuration.

```python
run = wandb.init(
    project="my-awesome-project",
    config={
        "learning_rate": 0.01,
        "epochs": 10,
    },
)
```

A run is a fundamental unit in W\&B used to track metrics, logs, jobs, and more.

**Example Script**: Below is an example script that demonstrates how to use W\&B for tracking hyperparameters and metrics in a training script.

```python
import wandb
import random

wandb.login()

epochs = 10
lr = 0.01

run = wandb.init(
    project="my-awesome-project",
    config={
        "learning_rate": lr,
        "epochs": epochs,
    },
)

offset = random.random() / 5
print(f"lr: {lr}")

for epoch in range(2, epochs):
    acc = 1 - 2**-epoch - random.random() / epoch - offset
    loss = 2**-epoch + random.random() / epoch + offset
    print(f"epoch={epoch}, accuracy={acc}, loss={loss}")
    wandb.log({"accuracy": acc, "loss": loss})

# run.log_code()
```

<mark style="color:blue;">**View Results**</mark><mark style="color:blue;">:</mark> After running your script, you can view the tracked metrics, including accuracy and loss, in the W\&B App at <https://wandb.ai/home>.

<mark style="color:blue;">**What's Next?**</mark><mark style="color:blue;">:</mark> Explore other features of the W\&B ecosystem, such as integrations, reports, artifacts, sweeps, and more to enhance your machine learning experiments.

Additionally, the documentation provides answers to common questions, such as finding your API key, using W\&B in automated environments, setting up local installations, and temporarily disabling W\&B logging.

</details>

<details>

<summary>Reference: <mark style="color:green;">Weights and Biases python libraries</mark></summary>

When you installed the Weights and Biases (W\&B) library using the <mark style="color:yellow;">`pip install wandb`</mark> command, the following libraries were installed along with W\&B:

<mark style="color:blue;">**Click**</mark><mark style="color:blue;">:</mark> A library for creating command-line interfaces.

<mark style="color:blue;">**GitPython**</mark><mark style="color:blue;">:</mark> A Python library to interact with Git repositories.

<mark style="color:blue;">**psutil**</mark><mark style="color:blue;">:</mark> A cross-platform library for retrieving information on system utilization.

<mark style="color:blue;">**sentry\_sdk**</mark><mark style="color:blue;">:</mark> A Python SDK for Sentry, which helps with error tracking and monitoring.

<mark style="color:blue;">**docker-pycreds**</mark><mark style="color:blue;">:</mark> A library for working with Docker credentials.

<mark style="color:blue;">**setproctitle**</mark><mark style="color:blue;">:</mark> A library for setting the process title on Unix platforms.

<mark style="color:blue;">**appdirs**</mark><mark style="color:blue;">:</mark> A library for determining appropriate platform-specific directories.

<mark style="color:blue;">**protobuf**</mark><mark style="color:blue;">:</mark> A library for working with Protocol Buffers, used for efficient data serialization.

<mark style="color:blue;">**six**</mark><mark style="color:blue;">:</mark> A Python 2 and 3 compatibility library.

<mark style="color:blue;">**gitdb**</mark><mark style="color:blue;">:</mark> A library for interacting with Git object databases.

<mark style="color:blue;">**smmap**</mark><mark style="color:blue;">:</mark> A library for managing sparse memory-mapped files.

These libraries are dependencies that are required by W\&B to function properly.&#x20;

They help provide various functionalities, including version control, system monitoring, and efficient data serialisation, which are useful for tracking machine learning experiments effectively using W\&B.

</details>

<details>

<summary><mark style="color:green;">What are the main classes within the Transformer Library?</mark></summary>

### <mark style="color:purple;">The main classes within the library are:</mark>

1. <mark style="color:blue;">**Model Classes**</mark><mark style="color:blue;">:</mark> These classes correspond to specific transformer models. Each model class in the library is usually named after the model it represents, such as `BertModel` for BERT, `GPT2Model` for GPT-2, `T5Model` for T5, etc. These classes are used to <mark style="color:yellow;">i</mark>nstantiate model architectures<mark style="color:yellow;">,</mark> either from pretrained weights or from scratch.
2. <mark style="color:blue;">**Tokenizer Classes**</mark><mark style="color:blue;">:</mark> Tokenizers are responsible for preprocessing text input for transformer models. They <mark style="color:yellow;">convert text into a format that is understandable by the model.</mark> Classes like `BertTokenizer` and `GPT2Tokenizer` handle this task. There are also "fast" tokenizers (e.g., `BertTokenizerFast`), which are optimized for speed and provide additional functionalities.
3. <mark style="color:blue;">**Configuration Classes**</mark>: Configuration classes store the <mark style="color:yellow;">configurations of a model</mark> — for example, `BertConfig`. These classes contain all the settings necessary to build a model, such as the <mark style="color:yellow;">number of layers, hidden units, and attention heads</mark>.
4. <mark style="color:blue;">**Pipeline Classes**</mark><mark style="color:blue;">:</mark> The pipeline class abstracts away much of the <mark style="color:yellow;">preprocessing and postprocessing work involved in using models</mark>. For instance, the <mark style="color:yellow;">`pipeline`</mark> function allows users to perform tasks like text classification, question answering, and text generation using a simple API.
5. <mark style="color:blue;">**Trainer Class**</mark><mark style="color:blue;">:</mark> The `Trainer` class provides an easy-to-use <mark style="color:yellow;">interface for training, evaluating, and fine-tuning transformer models</mark>. It handles many training details like data collation, optimization, and logging.
6. <mark style="color:blue;">**Data Collator Classes**</mark><mark style="color:blue;">:</mark> These are utility classes used to collate batches of data. They are especially useful when dealing with variable-sized input, such as in language modeling tasks.
7. <mark style="color:blue;">**Optimizer and Scheduler Classes**</mark><mark style="color:blue;">:</mark> While not exclusive to Transformers, these classes are often used in conjunction with model training. They define the <mark style="color:yellow;">optimization algorithm and learning rate schedule</mark> used during training.

</details>

#### <mark style="color:purple;">If you are interested in understanding the libraries and processes installed via the setup.py script, please review the following pages.</mark>

[^1]: `pip3` is specifically designed for managing packages in Python 3 environments. It ensures that any packages you install or manage using it are for Python 3.

[^2]: * The `packaging` library in Python provides utilities for version handling, specifiers, markers, requirements, tags, and other functionalities related to package management.
    * It is often used in projects that need to parse, compare, or handle versions and requirements of Python packages.


# Axolotl Dependencies

The libraries contained within the axolotl virtual environment

<details>

<summary><mark style="color:green;"><strong>AutoGPTQ</strong></mark></summary>

AutoGPTQ is a software package designed to facilitate the quantization of large language models (LLMs), such as those based on the GPT architecture.&#x20;

It provides a user-friendly API that enables developers and researchers to efficiently apply weight-only quantization to these models using the GPTQ algorithm. Here's a detailed explanation of the key aspects and features of AutoGPTQ:

#### <mark style="color:green;">Key Features</mark>

<mark style="color:purple;">**Weight-Only Quantization**</mark><mark style="color:purple;">:</mark> AutoGPTQ focuses on quantizing the weights of a model, which can significantly reduce the memory footprint and improve the computational efficiency of the model without requiring extensive changes to the model architecture or training procedures.

<mark style="color:purple;">**User-Friendly APIs**</mark><mark style="color:purple;">:</mark> The package is designed to be easy to use, making the process of quantizing models accessible to a wider audience, including those who may not have a deep background in machine learning or optimization.

<mark style="color:purple;">**Integration with Popular Tools**</mark>: AutoGPTQ is integrated with the <mark style="color:yellow;">**`transformers`**</mark> library by Hugging Face, along with other optimization tools like <mark style="color:yellow;">**`optimum`**</mark> and <mark style="color:yellow;">**`peft`**</mark>. This integration allows for seamless quantization within existing workflows that use these popular libraries.

#### <mark style="color:green;">Performance Improvements</mark>

AutoGPTQ includes support for specialized hardware acceleration, such as Marlin int4\*fp16 matrix multiplication kernels, which can dramatically increase the speed of inference when using quantized models. The performance comparison provided shows that quantized models can achieve higher throughput in terms of tokens per second compared

</details>

<table><thead><tr><th width="216">Library/Package</th><th>Description</th></tr></thead><tbody><tr><td>auto-gptq</td><td>See expandable</td></tr><tr><td>packaging</td><td>A library for parsing, comparing, and manipulating version numbers and version specifiers</td></tr><tr><td>peft</td><td>A library for efficient fine-tuning of large language models using parameter-efficient techniques like adapter modules and prefix tuning.</td></tr><tr><td>transformers</td><td>Huggingface Transformer library</td></tr><tr><td>tokenizers</td><td>A fast and efficient tokenization library for NLP tasks</td></tr><tr><td>bitsandbytes</td><td>A library for quantization and compression techniques in deep learning models.</td></tr><tr><td>accelerate</td><td>A library for simplifying distributed training and mixed precision training in PyTorch</td></tr><tr><td>deepspeed</td><td>A deep learning optimization library for training large models efficiently, providing techniques like ZeRO (Zero Redundancy Optimizer).</td></tr><tr><td>pydantic</td><td>A library for data validation and settings management using Python type annotations</td></tr><tr><td>addict</td><td>A dictionary library that allows accessing nested dictionaries using dot notation</td></tr><tr><td>fire</td><td>A Python library by Google that automatically generates command-line interfaces (CLIs) from Python code.</td></tr><tr><td>PyYAML</td><td>A YAML parser and emitter for Python, used for serializing and deserialising YAML data.</td></tr><tr><td>datasets</td><td>A library providing a unified interface for loading and manipulating datasets in NLP and machine learning.</td></tr><tr><td>flash-attn</td><td>A library for fast and memory-efficient attention mechanisms in transformer models.</td></tr><tr><td>sentencepiece</td><td>A tokenization library for subword tokenization and byte-pair encoding (BPE)</td></tr><tr><td>wandb</td><td>A tool for visualising and tracking machine learning experiments, making it easier to share results and progress.</td></tr><tr><td>einops</td><td>Provides a more readable and flexible way to write tensor operations in deep learning models.</td></tr><tr><td>xformers</td><td>library for efficient transformer implementations and attention mechanisms</td></tr><tr><td>optimum</td><td>A library for optimizing and quantizing transformer models for deployment</td></tr><tr><td>hf_transfer</td><td>A library for transfer learning and fine-tuning models from the Hugging Face ecosystem</td></tr><tr><td>colorama</td><td>A Python module for producing colored terminal text and cursor positioning on multiple platforms.</td></tr><tr><td>numba</td><td>A high-performance Python compiler that accelerates computation by generating native machine code.</td></tr><tr><td>numpy</td><td>A package for scientific computing with Python, widely used for multi-dimensional array and matrix processing.</td></tr><tr><td><strong>bert</strong>-score==0.3.13</td><td>A specific version of a library that provides an evaluation metric for text generation.</td></tr><tr><td>evaluate==0.4.0</td><td>An unknown library, possibly related to model evaluation or metrics.</td></tr><tr><td>rouge-score==0.1.2</td><td>A specific version of a library used for evaluating summarization and other text generation tasks by comparing against reference texts.</td></tr><tr><td>scipy</td><td>A Python library used for scientific and technical computing, offering modules for optimization, integration, and statistics.</td></tr><tr><td>scikit-learn</td><td>A popular machine learning library in Python, known for its easy-to-use API and performance in classification, regression, and clustering algorithms.</td></tr><tr><td>pynvml</td><td>A Python wrapper for NVIDIA Management Library, used for monitoring and managing NVIDIA GPU devices.</td></tr><tr><td>art</td><td>An unknown library, potentially related to ASCII art rendering or similar artistic transformations in text.</td></tr><tr><td>fschat</td><td>A library for building chatbots and conversational AI systems</td></tr><tr><td>gradio</td><td>An easy-to-use library to rapidly create UIs for machine learning models, enabling quick prototyping and sharing of ML models.</td></tr><tr><td>tensorboard</td><td>A visualization toolkit for machine learning experimentation, providing metrics and visualizations for TensorFlow projects.</td></tr><tr><td>s3fs</td><td>A Pythonic file interface to S3, allowing easy interaction with Amazon S3 buckets as if they are local files.</td></tr><tr><td>gcsfs</td><td>A Pythonic file system interface to Google Cloud Storage, similar in functionality to <code>s3fs</code> but for Google's cloud storage solution.</td></tr><tr><td>mamba-ssm</td><td>A library for secure secret management in Python</td></tr><tr><td>trl</td><td>A library for reinforcement learning in transformers</td></tr><tr><td>zstandard</td><td>A compression library providing a fast and efficient compression algorithm</td></tr><tr><td>fastcore</td><td>A library of core utilities for fast.ai, a deep learning library</td></tr></tbody></table>


# setup.py objectives

Decomposition

Running the <mark style="color:yellow;">**`setup.py`**</mark> script after cloning a repository is a step in many Python projects, especially when the project is a package or library that you intend to use in your own code or applications.&#x20;

Here's what the <mark style="color:yellow;">`setup.py`</mark> script can  do:

### <mark style="color:blue;">Purpose of</mark> <mark style="color:yellow;">`setup.py`</mark>

#### <mark style="color:green;">**Installing the Package**</mark>

While cloning the repository brings the source code onto your local machine, it <mark style="color:blue;">**doesn't install the package into your Python environment**</mark>**.**&#x20;

The <mark style="color:yellow;">**`setup.py`**</mark> script is responsible for *<mark style="color:yellow;">**installing the package**</mark>* so that its modules can be easily imported and used in other Python scripts or projects.

<mark style="color:green;">**Setuptools and Distutils**</mark>

The <mark style="color:yellow;">**`setup.py`**</mark> script typically uses the <mark style="color:yellow;">**`setuptools`**</mark> or <mark style="color:yellow;">**`distutils`**</mark> libraries to handle the packaging and distribution of Python projects.&#x20;

These libraries provide a standardised way of defining package metadata, dependencies, and build instructions.

#### <mark style="color:green;">**Dependency Management**</mark>

The script also <mark style="color:blue;">**handles the installation of dependencies.**</mark>&#x20;

Many Python projects rely on external libraries, and <mark style="color:yellow;">**`setup.py`**</mark> typically reads a <mark style="color:yellow;">**`requirements.txt`**</mark> file or directly contains a list of these dependencies, ensuring that all necessary packages are installed.

#### <mark style="color:green;">**Compiling Extensions**</mark>

For packages that include native extensions (e.g., written in C or C++), the <mark style="color:yellow;">**`setup.py`**</mark> script <mark style="color:blue;">compiles these extensions into a format that Python can use</mark>.

#### <mark style="color:green;">**Distribution Formats**</mark>

**I**n addition to installing the package, `setup.py` can also be used to create distribution packages in various formats, such as source distributions (sdist) or wheel packages.&#x20;

These distribution packages can be uploaded to package repositories like PyPI (Python Package Index) for others to install using tools like <mark style="color:yellow;">**`pip`**</mark>.&#x20;

#### <mark style="color:green;">**Development Mode**</mark>

**I**n addition to the regular installation mode, <mark style="color:yellow;">**`setup.py`**</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">often supports a development mode installation</mark> using the command <mark style="color:yellow;">**`python setup.py develop`**</mark> or <mark style="color:yellow;">**`pip install -e`**</mark><mark style="color:yellow;">` `</mark><mark style="color:yellow;">`.`</mark>.&#x20;

This mode creates a <mark style="color:yellow;">symbolic link to the project's source code,</mark> allowing changes to the code to be immediately reflected without the need to reinstall the package.&#x20;

#### <mark style="color:green;">**Environment Configuration**</mark>

The script can set up environment variables or configuration files necessary for the package to run properly.

#### <mark style="color:green;">**Version Control**</mark>

It often manages the versioning of the package, making sure that the correct version is installed, and can handle upgrades to newer versions.

### <mark style="color:blue;">Why It's Necessary</mark>

<mark style="color:green;">**Cloning vs. Installing**</mark>

Cloning a repository only gets you the raw source code.&#x20;

<mark style="color:blue;">Without running</mark> <mark style="color:yellow;">**`setup.py`**</mark><mark style="color:blue;">, the Python interpreter won't know where to find the packages you're trying to use.</mark>&#x20;

The installation process <mark style="color:yellow;">registers the package with your Python interpreter,</mark> adding it to the <mark style="color:yellow;">**`sys.path`**</mark> variable used by Python to search for modules and packages.

<mark style="color:green;">**Ease of Use**</mark>

<mark style="color:blue;">Once installed, you can import and use the package from anywhere on your system</mark>, just like you would with any other installed Python library.

<mark style="color:green;">**Consistency and Reproducibility**</mark>

By installing the package in a standardised way, you ensure consistency, especially when sharing your code with others or deploying it to different environments. Everyone using your code will have the same setup and dependencies.

### <mark style="color:blue;">Running the Script</mark>

<mark style="color:green;">**Executing**</mark>

To run <mark style="color:yellow;">**`setup.py`**</mark>, you typically use a command like <mark style="color:yellow;">**`python setup.py install`**</mark>.&#x20;

This tells Python to execute the script and perform the installation steps defined within it.

<mark style="color:green;">**Best Practices**</mark>

It's recommended to install Python packages in a virtual environment to avoid conflicts with system-wide packages and manage dependencies more effectively.

In summary, running <mark style="color:yellow;">**`setup.py`**</mark> is an important step in installing a Python package from a source code repository.

It ensures that the package and its dependencies are correctly installed and configured in your Python environment, allowing you to use the package seamlessly in your own projects.


# script analysis

### <mark style="color:blue;">Dependencies</mark>

The script starts by importing necessary modules:

* <mark style="color:yellow;">**`platform`**</mark> module to retrieve information about the operating system.
* <mark style="color:yellow;">**`re`**</mark> module for regular expressions.
* <mark style="color:yellow;">**`PackageNotFoundError`**</mark> and `version` from <mark style="color:yellow;">**`importlib.metadata`**</mark> for handling package versions.
* <mark style="color:yellow;">**`find_packages`**</mark> and <mark style="color:yellow;">**`setup`**</mark> from <mark style="color:yellow;">**`setuptools`**</mark> for package discovery and setup configuration.

### <mark style="color:blue;">Parse Requirements</mark>

The <mark style="color:yellow;">**`parse_requirements()`**</mark> function is defined to parse the <mark style="color:yellow;">**`requirements.txt`**</mark> file:

* It initializes two lists: <mark style="color:yellow;">**`_install_requires`**</mark> for standard package requirements and `_dependency_links` for custom index URLs.
* It opens the <mark style="color:yellow;">**`requirements.txt`**</mark> file, reads each line, and strips any whitespace.
* It checks each line to determine if it is an extra requirement (e.g., "flash-attn", "deepspeed", etc.) or a standard package requirement.
* If the line starts with "--extra-index-url", it extracts the URL and appends it to <mark style="color:yellow;">**`_dependency_links`**</mark><mark style="color:yellow;">**.**</mark>
* If the line is not an extra requirement and is not empty or a comment, it appends the line to <mark style="color:yellow;">**`_install_requires`**</mark><mark style="color:yellow;">**.**</mark>

### <mark style="color:blue;">Platform-specific requirements</mark>

* If the operating system is "Darwin" (macOS), it removes the "xformers==0.0.22" requirement from <mark style="color:yellow;">**`_install_requires`**</mark><mark style="color:yellow;">**.**</mark>
* If the operating system is not "Darwin", it retrieves the version of the "torch" package using <mark style="color:yellow;">**`version("torch")`**</mark><mark style="color:yellow;">**.**</mark>
* It appends the "torch" package requirement with the specific version to <mark style="color:yellow;">**`_install_requires`**</mark><mark style="color:yellow;">**.**</mark>
* It parses the "torch" version using a regular expression to extract the major, minor, and patch version numbers.
* If the "torch" version is greater than or equal to "2.1", it removes the "xformers==0.0.22" requirement and appends "xformers>=0.0.23" to <mark style="color:yellow;">**`_install_requires`**</mark><mark style="color:yellow;">**.**</mark>
* If the "torch" package is not found, it skips this step.

### <mark style="color:blue;">The</mark> <mark style="color:yellow;">`setup()`</mark> <mark style="color:blue;">function is called to configure the package</mark>

* It sets the package name to "axolotl".
* It specifies the version as "0.4.0".
* It provides a short description and a long description for the package.
* It sets the package directory to "src".
* It uses `find_packages()` to automatically discover packages within the "src" directory.
* It sets the `install_requires` parameter to the `install_requires` list obtained from `parse_requirements()`.
* It sets the `dependency_links` parameter to the `dependency_links` list obtained from `parse_requirements()`.
* It defines extra requirements using the `extras_require` parameter, specifying additional dependencies for different features like "flash-attn", "deepspeed", "mamba-ssm", "auto-gptq", "mlflow", "lion-pytorch", and "galore".

Overall, this setup.py script is responsible for configuring the "axolotl" package, specifying its dependencies, handling platform-specific requirements, and defining extra requirements for optional features.&#x20;

It provides a flexible and customisable setup for the package, allowing users to install the necessary dependencies based on their specific needs.


# Huggingface Hub

Link to Huggingface Hub for acess to Models and Datasets

{% hint style="warning" %}

#### <mark style="color:orange;">We will be connecting to the Huggingface Hub to allow models and datasets to be downloaded to the Axolotl directory.</mark>

{% endhint %}

### <mark style="color:blue;">What is the Huggingface Hub?</mark>

The Hugging Face Hub is a platform with a collection of model and datasets.  Huggingface aim to be the github repository for artificial intelligence and machine learning.

{% hint style="info" %}
In the development phase, we will be using the Huggingface Hub to access models and datasets.   When working with enterprise clients, proprietary datasets and models can be stored on encrypted servers, not accessible to the public.
{% endhint %}

### <mark style="color:blue;">Signing Up for a Hugging Face Account</mark> <a href="#fb78" id="fb78"></a>

<mark style="color:green;">**Visit the Hugging Face Website:**</mark> Head to the Hugging Face website (<https://huggingface.co/>) to begin the account creation process.

<mark style="color:green;">**Click on “Sign Up”:**</mark> Locate the “Sign Up” button on the top right corner of the homepage and click on it.

<mark style="color:green;">**Choose a Sign-Up Method:**</mark> Hugging Face offers multiple sign-up methods, including Google, GitHub, and email. Select your preferred method and follow the prompts to complete the registration.

<mark style="color:green;">**Verify Your Email (if applicable):**</mark> If you choose to sign up via email, verify your email address by clicking on the confirmation link sent to your inbox.

<mark style="color:green;">**Complete Your Profile:**</mark> Enhance your Hugging Face experience by completing your profile. Add a profile picture, a short bio, and any other details you’d like to share with the community.

### <mark style="color:blue;">Creating an Access Token</mark>

To get a Huggingface access token follow these instructions:

{% embed url="<https://huggingface.co/docs/hub/en/security-tokens>" %}
Creating a Huggingface Access Token
{% endembed %}

### <mark style="color:blue;">Command Line Interface (CLI)</mark>

With a Huggingface account established and an access token you can now <mark style="color:yellow;">access the HuggingFace hub through a command line interface</mark>.

The <mark style="color:yellow;">**`huggingface_hub`**</mark> Python package comes with a built-in CLI called <mark style="color:yellow;">**`huggingface-cli`**</mark>. This tool allows you to interact with the Hugging Face Hub directly from a terminal.&#x20;

<details>

<summary>Reference: <mark style="color:green;">Details on the HuggingFace Command Line</mark></summary>

The huggingface\_hub Python package comes with a <mark style="color:orange;">**built-in CLI**</mark> called huggingface-cli. This tool allows you to *<mark style="color:blue;">**interact with the Hugging Face Hub directly from a terminal**</mark>*.&#x20;

For example, you can login to your account, create a repository, upload and download files, etc.&#x20;

It also comes with handy features to configure your machine or manage your cache. In this guide, we will have a look at the main features of the CLI and how to use them.

<mark style="color:green;">**Core Functionalities**</mark>

* <mark style="color:blue;">**User Authentication**</mark><mark style="color:blue;">:</mark> Allows login/logout and displays the current user information (<mark style="color:yellow;">**`login`**</mark>, <mark style="color:yellow;">**`logout`**</mark>, <mark style="color:yellow;">**`whoami`**</mark>).
* <mark style="color:blue;">**Repository Management**</mark><mark style="color:blue;">:</mark> Enables creation and interaction with repositories on Hugging Face (<mark style="color:yellow;">**`repo`**</mark>).
* <mark style="color:blue;">**File Operations**</mark>: Supports uploading, downloading files, and managing large files on the Hub (<mark style="color:yellow;">**`upload`**</mark>, <mark style="color:yellow;">**`download`**</mark>, <mark style="color:yellow;">**`lfs-enable-largefiles`**</mark>, <mark style="color:yellow;">**`lfs-multipart-upload`**</mark><mark style="color:yellow;">**)**</mark>.
* <mark style="color:blue;">**Cache Management**</mark><mark style="color:blue;">:</mark> Provides commands to scan and delete cached files (<mark style="color:yellow;">**`scan-cache`**</mark>, <mark style="color:yellow;">**`delete-cache`**</mark>).

<mark style="color:green;">**Usage**</mark>

* The CLI supports various commands and options, which can be explored using the `--help` flag.
* To interact with the Hub, such as downloading private repos or uploading files, users need to authenticate using a User Access Token.
* The CLI also supports downloading specific files or entire repositories, filtering files with patterns, and specifying revisions or local directories for downloads.

</details>

First, <mark style="color:yellow;">install the CLI</mark> and its extra dependencies, including the \[cli] extras, for an improved user experience:

```bash
pip3 install -U "huggingface_hub[cli]"
```

<details>

<summary>Reference: <mark style="color:green;">Python libraries installed with the Huggingface Hub</mark></summary>

The command you executed, <mark style="color:yellow;">`pip install -U "huggingface_hub[cli]`</mark>`"`, installs several Python libraries and their dependencies related to the Hugging Face Hub. Here's an explanation of the libraries downloaded:

<mark style="color:green;">**huggingface\_hub**</mark>

* This is the main library that provides access to the Hugging Face Hub, allowing you to interact with models, datasets, and repositories.
* It provides functionalities for uploading, downloading, and managing resources on the Hub.

<mark style="color:green;">**fsspec**</mark>

* fsspec is a Python library for managing filesystem-like abstractions. It is often used for handling remote and cloud-based file systems.
* In the context of the Hugging Face Hub, it likely helps in managing the storage and retrieval of model and dataset files.

<mark style="color:green;">**tqdm**</mark>

* tqdm is a popular library for adding progress bars to loops and other iterables.
* It's used in the CLI to display progress when uploading or downloading large files from the Hugging Face Hub.

<mark style="color:green;">**prompt-toolkit**</mark>

* prompt-toolkit is a library for building command-line interfaces (CLIs) with interactive features.
* It's used by the Hugging Face CLI for handling interactive prompts and user interactions.

<mark style="color:green;">**pfzy**</mark>

* pfzy is a Python library for fuzzy string matching and searching.
* It may be used in the CLI for fuzzy matching of commands or resources.

<mark style="color:green;">**InquirerPy**</mark>

* InquirerPy is a Python library for creating interactive command-line prompts with customisable menus and questions.
* It enhances the user experience when using the Hugging Face CLI by providing interactive prompts.

<mark style="color:green;">**wcwidth**</mark>

* wcwidth is a library for determining the printable width of characters when rendering text in a terminal.
* It helps ensure proper formatting and alignment of text in the CLI.

</details>

Verify that the <mark style="color:yellow;">CLI is correctly set up</mark> by running the following command:

```bash
huggingface-cli --help
```

You should see a list of available options and commands. If you encounter an error like "command not found: huggingface-cli," please refer to the installation guide for troubleshooting.

Now try this command:

```bash
huggingface-cli whoami 
```

When logged into the HuggingFace Hub this command prints your username and the organisations you are a part of on the Hub<mark style="color:yellow;">.</mark>

At this stage, <mark style="color:yellow;">we have not yet logged into the HuggingFace Hub</mark>, so the response will be:

```bash
Not logged in
```

<details>

<summary>Reference: <mark style="color:green;">Huggingface CLI Commands</mark></summary>

Use the `--help` option to get <mark style="color:yellow;">detailed information about a specific command.</mark> For example, to learn more about how to upload files using the CLI, run:

```bash
huggingface-cli upload --help
```

<mark style="color:green;">Examples</mark>

**Login to Your Hugging Face Account**

Use the following command to log in with your token obtained from huggingface.co/settings/tokens:

```bash
huggingface-cli login
```

<mark style="color:green;">**Upload Files to a Repository**</mark>

* To upload a file or folder to a repository on the Hub, use the `upload` command. Replace <mark style="color:yellow;">`<repository_name>`</mark> with the name of your repository and <mark style="color:yellow;">`<path_to_file_or_folder>`</mark> with the path to the file or folder you want to upload:

```bash
huggingface-cli upload <repository_name> <path_to_file_or_folder>
```

<mark style="color:green;">**Download Files from the Hub**</mark>

* Download files from the Hub using the `download` command. Specify the file or folder you want to download and the destination path:

```bash
huggingface-cli download <file_or_folder_to_download> <destination_path>
```

<mark style="color:green;">**Managing Repositories**</mark>

* You can interact with your repositories using the `repo` command. For example, to create a new repository, use:

```bash
huggingface-cli repo create <repository_name>
```

<mark style="color:green;">**Environment Information**</mark>

* To view information about your environment, use the `env` command:

```bash
huggingface-cli env
```

These tutorials will help you get started with the Hugging Face CLI for managing models, datasets, and repositories on the Hugging Face Hub.

</details>

{% hint style="info" %}
We will now be connecting to the Huggingface Hub to allow models and datasets to be downloaded to the Axolotl directory.
{% endhint %}

To connect to the Huggingface Hub <mark style="color:yellow;">prepare your machine to allow storing of the Huggingface token</mark>:

```bash
git config --global credential.helper store
```

This command tells Git to use a simple storage method to save your credentials, which can be a security risk as credentials are stored in plain text - but is fine during development.

<details>

<summary><mark style="color:green;">Explanation of Git Credentials</mark></summary>

A "git credential" refers to the <mark style="color:blue;">authentication details used by Git to access repositories,</mark> especially those requiring user verification, such as private repositories or when pushing changes to remote repositories.&#x20;

These credentials can be usernames and passwords, personal access tokens, SSH keys, or other forms of identity verification.

#### <mark style="color:green;">Types of Git Credentials</mark>

<mark style="color:blue;">**Username and Password**</mark><mark style="color:blue;">:</mark> The simplest form, but not recommended for security reasons, especially for repositories over HTTPS.

<mark style="color:blue;">**Personal Access Tokens (PATs)**</mark><mark style="color:blue;">:</mark> More secure than passwords, these tokens are used especially when two-factor authentication (2FA) is enabled. GitHub, for instance, requires PATs for authenticating over HTTPS.

<mark style="color:blue;">**SSH Keys**</mark><mark style="color:blue;">:</mark> Secure and commonly used, SSH keys pair a private key (kept on your machine) with a public key (added to your Git server). They are a popular choice for authentication.

#### <mark style="color:green;">Storage of Git Credentials</mark>

<mark style="color:blue;">**Credential Cache**</mark><mark style="color:blue;">:</mark> Temporarily stores credentials in memory for a short period. This is more secure than storing them on disk but requires re-entry after the cache timeout.

<mark style="color:blue;">**Credential Store**</mark><mark style="color:blue;">:</mark> Saves credentials in plain text in a file on your computer. It's convenient but less secure since the file can be read by anyone with access to your system.

<mark style="color:blue;">**SSH Agent**</mark><mark style="color:blue;">:</mark> For SSH keys, an SSH agent can store your private keys and handle authentication.

<mark style="color:blue;">**System Keychain**</mark><mark style="color:blue;">:</mark> Some Git clients can store credentials in the system's keychain or credential manager, offering a balance of convenience and security.

<mark style="color:blue;">**Environment Variables**</mark><mark style="color:blue;">:</mark> Sometimes used for automation, credentials can be set as environment variables, but this method has significant security downsides.

#### <mark style="color:green;">Best Practices</mark>

<mark style="color:blue;">**Prefer Token/Key-based Authentication**</mark><mark style="color:blue;">:</mark> Use personal access tokens or SSH keys over passwords for better security.

<mark style="color:blue;">**Keep Software Updated**</mark><mark style="color:blue;">:</mark> Ensure your Git client and any related credential management tools are up-to-date to benefit from security patches.

<mark style="color:blue;">**Use SSH Keys Wisely**</mark><mark style="color:blue;">:</mark> Protect your SSH private keys with strong passphrases and use ssh-agent for managing them.

<mark style="color:blue;">**Limit Token Scopes and Lifetimes**</mark><mark style="color:blue;">:</mark> When creating personal access tokens, grant only the necessary permissions and set a reasonable expiration.

<mark style="color:blue;">**Securely Store Sensitive Information**</mark><mark style="color:blue;">:</mark> Avoid storing credentials in plaintext files. Use system keychains or encrypted storage whenever possible.

<mark style="color:blue;">**Be Cautious with Environment Variables**</mark><mark style="color:blue;">:</mark> Be mindful of security risks when using environment variables for credentials, especially in shared or public environments.

<mark style="color:blue;">**Regularly Review and Rotate Credentials**</mark><mark style="color:blue;">:</mark> Regularly review your access tokens and SSH keys, revoking and replacing them as necessary.

<mark style="color:blue;">**Use Two-Factor Authentication (2FA)**</mark><mark style="color:blue;">:</mark> Wherever possible, enable 2FA for your Git hosting service accounts for an additional layer of security.

</details>

### <mark style="color:blue;">Enter the command to login to the Huggingface Hub</mark>

```bash
huggingface-cli login
```

You will be asked to enter a Huggingface token as per the code block below:

<pre class="language-bash" data-full-width="false"><code class="lang-bash"> > huggingface-cli login

    _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|
    _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|
    _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|
    _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|
    _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|

    A token is already saved on your machine. Run `huggingface-cli whoami` to get more information or `huggingface-cli logout` if you want to log out.
    Setting a new token will erase the existing one.
    To login, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .
Token: <a data-footnote-ref href="#user-content-fn-1">xxxxxxxxxxxxxxx</a>
</code></pre>

You will then be asked whether you want to add the Huggingface token to be added as a git credential.   <mark style="color:yellow;">Answer yes (y) to this question</mark>.  The output should be as below:

```
Add token as git credential? (Y/n) Y
Token is valid (permission: read).
Your token has been saved in your configured git credential helpers (store).
Your token has been saved to /home/jack/.cache/huggingface/token
Login successful
```

This command tells Git to use a simple storage method to save your credentials, which can be a security risk as credentials are stored in plain text - but is fine during development.

Now check to ensure you are logged in:

```bash
huggingface-cli whoami 
```

Now that we have logged into the HuggingFace hub the output should be:

```bash
<your name>
orgs:  <your organisation>
```

You can clearly see your name, and the name of the organization.

### <mark style="color:purple;">Congratulations!</mark>

#### <mark style="background-color:green;">You have established the Axolotl platform and connected to Huggingface.</mark>

[^1]: Enter your Hugginface token here


# Download the dataset

Once the model has been downloaded, the dataset is next

The next step is to <mark style="color:yellow;">download the dataset.</mark>&#x20;

Before doing this we will document how datasets work on Huggingface and then provide specific instructions on how to <mark style="color:blue;">**download dataset into the Axolotl platform.**</mark>

### <mark style="color:blue;">The HuggingFace Hub has numerous datasets</mark>

<details>

<summary><mark style="color:green;">HuggingFace Dataset Summary</mark></summary>

The Hugging Face Hub hosts a wide array of datasets for various tasks like translation, speech recognition, and image classification. These datasets are stored in Git repositories and contain scripts for data download and split generation.

1. <mark style="color:blue;">**Dataset Viewer**</mark><mark style="color:blue;">:</mark> Many datasets, like GLUE, feature a Dataset Viewer to preview data.
2. <mark style="color:blue;">**Repository Structure**</mark><mark style="color:blue;">:</mark> Each dataset repository has a specific structure for efficient data handling. Adhering to this structure ensures the dataset page on the Hub will have a Viewer.
3. <mark style="color:blue;">**Search and Filter**</mark><mark style="color:blue;">:</mark> Datasets can be searched and filtered by language, tasks, and licenses on the Hub.
4. <mark style="color:blue;">**Privacy Settings**</mark><mark style="color:blue;">:</mark> Dataset visibility can be toggled between private and public. For organization-owned datasets, these settings apply to all members.
5. <mark style="color:blue;">**Dataset Cards**</mark>: Each dataset is documented with a `README.md` file in the repository, acting as a dataset card. This card provides context and usage instructions and can include metadata like license, language, size, and tags for easy discovery.
6. <mark style="color:blue;">**Metadata in Dataset Cards**</mark><mark style="color:blue;">:</mark> Metadata added to the dataset card enables interactions on the Hub, like filtering and discovering datasets, and displaying licenses.
7. <mark style="color:blue;">**Linking to Papers**</mark><mark style="color:blue;">:</mark> Including a paper link on arXiv in the dataset card adds the dataset to relevant tags and facilitates finding models citing the same paper.
8. <mark style="color:blue;">**Gated Datasets**</mark><mark style="color:blue;">:</mark> Creators can control access to their datasets. Users must agree to terms and share contact information to access these datasets. Dataset owners can view user access reports.
9. <mark style="color:blue;">**Customizing User Access Prompts**</mark><mark style="color:blue;">:</mark> The access request dialog can be customized with additional text and checkbox fields for specific user agreements.
10. <mark style="color:blue;">**Manual Approval of Access**</mark><mark style="color:blue;">:</mark> Dataset authors can choose to manually review and approve access requests. An API is available for managing access requests.
11. <mark style="color:blue;">**Notifications**</mark><mark style="color:blue;">:</mark> Default notification settings send daily emails for new access requests. These can be customized for real-time updates or sent to a specific email.
12. <mark style="color:blue;">**Additional Customization**</mark><mark style="color:blue;">:</mark> Text in the gate's heading and button can be customized to suit specific requirements.

</details>

### <mark style="color:blue;">Uploading datasets for future use</mark>

<details>

<summary><mark style="color:green;">Uploading Huggingface Datasets</mark></summary>

Hugging Face's Hub allows for uploading and sharing of a wide range of datasets. Here's a summary of the key points for uploading datasets:

1. <mark style="color:blue;">**Account Creation**</mark><mark style="color:blue;">:</mark> Start by creating a Hugging Face Hub account.
2. <mark style="color:blue;">**Upload Using the Hub UI**</mark><mark style="color:blue;">:</mark> This user-friendly interface allows even non-developers to upload datasets.
3. <mark style="color:blue;">**Creating a Repository**</mark><mark style="color:blue;">:</mark> A repository is where your dataset files and their revision history are stored. It supports multiple dataset versions.
4. <mark style="color:blue;">**Uploading a Dataset**</mark><mark style="color:blue;">:</mark> After creating a repository, you can upload dataset files through the "Files and versions" tab. <mark style="color:green;">The Hub supports various file formats like .csv, .mp3, and .jpg</mark>.
5. <mark style="color:blue;">**Dataset Card Creation**</mark><mark style="color:blue;">:</mark> This is crucial for helping users discover and understand your dataset. It involves selecting important metadata tags and writing detailed documentation about your dataset.
6. <mark style="color:blue;">**Dataset Viewer**</mark><mark style="color:blue;">:</mark> For public datasets, the Dataset Viewer allows users to preview the data before downloading.
7. <mark style="color:blue;">**Using the huggingface\_hub Client Library**</mark><mark style="color:blue;">:</mark> This library offers advanced features for managing repositories and uploading datasets.
8. <mark style="color:blue;">**Using Other Libraries**</mark><mark style="color:blue;">:</mark> Libraries like 🤗 Datasets, Pandas, Dask, or DuckDB can also upload files to the Hub.
9. <mark style="color:blue;">**Using Git**</mark><mark style="color:blue;">:</mark> Dataset repositories, being Git repositories, allow the use of Git to push data files to the Hub.
10. <mark style="color:blue;">**Supported File Formats**</mark><mark style="color:blue;">:</mark> The Hub supports a variety of file formats, including CSV, JSON, Parquet, Text, Images, and Audio files. Compressed files in formats like ZIP, GZIP, and others are also supported.
11. <mark style="color:blue;">**Downloading Datasets**</mark><mark style="color:blue;">:</mark> The Hub also facilitates the downloading of datasets.

This summary provides a comprehensive overview of how to effectively utilize the Hugging Face Hub for uploading and managing datasets, emphasizing its accessibility and support for a wide range of file formats.

</details>

### <mark style="color:blue;">Download a Huggingface dataset</mark>

Downloading datasets from the Hugging Face Hub can be accomplished through several methods. &#x20;

We will be using the <mark style="color:yellow;">'git clone'</mark> method:

### <mark style="color:blue;">**Using Git**</mark>

* All datasets on the Hub are stored as Git repositories, allowing for cloning directly to the local machine.
* This method is particularly useful for large datasets or when you require the entire dataset repository.
* Before cloning, ensure Git Large File Storage (LFS) is installed with <mark style="color:yellow;">**`git lfs install`**</mark><mark style="color:yellow;">**.**</mark>
* Clone the dataset using the command:

```bash
git clone git@hf.co:datasets/<dataset ID>
```

* Replace <mark style="color:yellow;">**`<dataset ID>`**</mark> with the actual ID of the dataset you wish to clone (e.g., <mark style="color:yellow;">**`git clone git@hf.co:datasets/allenai/c4`**</mark>).
* If you have write-access to the dataset repository, you can also commit and push revisions.
* To push changes or access private repositories, add your SSH public key to your user settings on Hugging Face.

<details>

<summary>Reference: <mark style="color:green;">Using the Fast Download Library</mark></summary>

The <mark style="color:yellow;">**`HF_HUB_ENABLE_HF_TRANSFER`**</mark> environment variable, when set to <mark style="color:yellow;">**`True`**</mark>, enhances the speed of uploads and downloads from the Hugging Face Hub by utilizing a Rust-based package called <mark style="color:yellow;">**`hf_transfer`**</mark><mark style="color:yellow;">.</mark> Here's a summary of its functionality and considerations:

<mark style="color:blue;">**Enhanced Speed**</mark>

By default, Hugging Face Hub employs Python-based functions like <mark style="color:yellow;">**`requests.get`**</mark> and <mark style="color:yellow;">**`requests.post`**</mark> for uploads and downloads. While reliable, these methods may not be the most efficient for high-bandwidth machines. `hf_transfer` is a Rust-based package that optimizes bandwidth usage by <mark style="color:yellow;">splitting large files into smaller parts and transferring them concurrently with multiple threads</mark>. This approach has the potential to double the transfer speed.

<mark style="color:blue;">**Installation**</mark>

To use <mark style="color:yellow;">**`hf_transfer`**</mark>, you must install it separately from PyPI (Python Package Index) and set the environment variable <mark style="color:yellow;">**`HF_HUB_ENABLE_HF_TRANSFER`**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**to**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`1`**</mark>. This enables the Rust-based transfer logic for faster operations.

<mark style="color:blue;">**Limitations**</mark>

It is essential to be aware of certain limitations when using <mark style="color:yellow;">**`hf_transfer`**</mark><mark style="color:yellow;">:</mark>

* Debugging may be challenging since it is not purely Python-based.
* <mark style="color:yellow;">**`hf_transfer`**</mark> lacks some user-friendly features such as resumable downloads and proxy support. These omissions are intentional to maintain the simplicity and speed of the Rust logic.

<mark style="color:blue;">**Default State**</mark>

<mark style="color:yellow;">**`f_transfer`**</mark> is not enabled by default in Hugging Face Hub, meaning that you need to explicitly set <mark style="color:yellow;">**`HF_HUB_ENABLE_HF_TRANSFER`**</mark> to <mark style="color:yellow;">**`True`**</mark> if you wish to utilize its enhanced transfer capabilities.

In summary, <mark style="color:yellow;">**`HF_HUB_ENABLE_HF_TRANSFER`**</mark> is an environment variable that, when activated, leverages the Rust-based <mark style="color:yellow;">**`hf_transfer`**</mark> package to significantly improve upload and download speeds from the Hugging Face Hub.&#x20;

However, it's important to be aware of its limitations, including potential debugging challenges and the absence of certain features, as it is not the default transfer method in Hugging Face Hub.

</details>


# Types of Dataset Structures

Formats and Customizations

### <mark style="color:blue;">Axolotl Dataset Formats and Customisation</mark>

Axolotl is versatile in handling various dataset formats. &#x20;

Below are some of the formats you can use, with <mark style="color:blue;">**JSONL**</mark> being the recommended format:

### <mark style="color:green;">**Alpaca Format**</mark>

* Structure: <mark style="color:yellow;">`{`</mark><mark style="color:purple;">**`"instruction"`**</mark><mark style="color:yellow;">`: "your_instruction",`</mark>` `<mark style="color:purple;">**`"input"`**</mark><mark style="color:yellow;">`: "optional_input",`</mark>` `<mark style="color:purple;">**`"output"`**</mark><mark style="color:yellow;">`: "expected_output"}`</mark>
* Ideal for scenarios where you need to *<mark style="color:yellow;">p</mark><mark style="color:yellow;">**rovide specific instructions along with optional input data.**</mark>* The output field holds the expected result. This format is particularly useful for guided learning tasks.

### <mark style="color:green;">**ShareGPT Format**</mark>

* Structure: <mark style="color:yellow;">`{`</mark><mark style="color:purple;">**`"conversations"`**</mark><mark style="color:yellow;">`: [{"from": "human/gpt", "value":`</mark>` `<mark style="color:purple;">**`"dialogue_text"`**</mark><mark style="color:yellow;">`}]}`</mark>
* This format suits conversational models where *<mark style="color:yellow;">**interactions are between a human and a GPT-like model.**</mark>* It helps in training models to understand and respond in a dialogue setting, reflecting real-world conversational flows.

### <mark style="color:green;">**Completion Format**</mark>

* Structure: <mark style="color:yellow;">`{`</mark><mark style="color:purple;">**`"text"`**</mark><mark style="color:yellow;">`: "your_text_data"}`</mark>
* The completion format is straightforward and best for training models on raw text corpora.  It's ideal for scenarios where the model *<mark style="color:yellow;">**needs to learn from unstructured text**</mark>* without specific instructions or dialogue contexts.

### <mark style="color:blue;">**Adding Custom Prompts**</mark>

For datasets preprocessed with instruction-focused tasks:

* Structure: <mark style="color:yellow;">`{`</mark><mark style="color:purple;">**`"instruction"`**</mark><mark style="color:yellow;">`: "your_instruction",`</mark>` `<mark style="color:purple;">**`"output"`**</mark><mark style="color:yellow;">`: "expected_output"}`</mark>
* This format supports a direct instructional approach, where the model is trained to follow specific commands or requests. It's effective for task-oriented models.

#### <mark style="color:green;">**Incorporating this into your Axolotl YAML configuration**</mark>

```yaml
datasets:
  - path: repo
    type:
      system_prompt: ""
      field_system: system
      format: "[INST] {instruction} [/INST]"
      no_input_format: "[INST] {instruction} [/INST]"
```

This YAML config allows for a flexible setup, enabling the model to interpret and learn from the structured instructional format.

### <mark style="color:blue;">**Custom Pre-tokenized Dataset Usage**</mark>

To use a custom pre-tokenized dataset:

* <mark style="color:yellow;">**Do not**</mark> specify a <mark style="color:yellow;">**`type`**</mark> in your configuration.
* Ensure your dataset columns are precisely named as <mark style="color:yellow;">**`input_ids`**</mark><mark style="color:yellow;">,</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">**`attention_mask`**</mark>, and <mark style="color:yellow;">**`labels`**</mark><mark style="color:yellow;">.</mark>

This approach is beneficial <mark style="color:blue;">**when you have a dataset that is already tokenized**</mark> and ready for model consumption.&#x20;

It skips additional preprocessing steps, streamlining the training process for efficiency.

### <mark style="color:blue;">Interesting Points Regarding Datasets</mark>

* <mark style="color:purple;">**Format Flexibility**</mark><mark style="color:purple;">:</mark> Axolotl’s support for multiple formats allows for training models on diverse data types - from structured instructional data to informal conversational dialogues.
* <mark style="color:purple;">**Customisability**</mark><mark style="color:purple;">:</mark> The ability to customise datasets and their integration into the system via YAML configurations provides a high degree of control over the training process, allowing for fine-tuning specific to the desired output of the model.
* <mark style="color:purple;">**Efficiency in Pre-tokenized Data**</mark><mark style="color:purple;">:</mark> The support for pre-tokenized datasets is a significant time-saver, particularly in scenarios where datasets are vast and tokenization can become a computationally expensive step.

This variety and customisability make Axolotl a robust tool for training language models across different scenarios and requirements, enhancing its versatility in AI model development.


# Structuring Datasets for Fine-Tuning Large Language Models

### <mark style="color:blue;">Importance of Dataset Structure</mark>

The structure of a dataset plays a crucial role in how a large language model (LLM) learns during the fine-tuning process.

Properly structured datasets enable LLMs to understand the relationship between input and output data, as well as the specific task or behavior they are being trained for.

Different dataset formats are suitable for different scenarios, such as guided learning tasks, conversational models, or learning from unstructured text.

### <mark style="color:blue;">Impact of Labelling and Syntax</mark>

* Labelling and syntax play a significant role in guiding the LLM's learning process during fine-tuning.
* Curly braces <mark style="color:yellow;">**`{}`**</mark> are used to denote JSON objects, which provide a structured way to represent key-value pairs in the dataset.
* Quotation marks <mark style="color:yellow;">**`""`**</mark> are used to enclose string values, distinguishing them from keys or other data types.
* Labels like "instruction," "input," "output," "conversations," "from," and "value" serve as keys in the JSON object, providing context and meaning to the associated values.
* By using consistent labelling and syntax, the neural language model can learn to associate specific labels with their corresponding roles in the task or conversation.

### <mark style="color:blue;">Labels</mark>

In the input/output format, the "label" property is used to indicate whether a particular segment of text should be used for training the model or not.&#x20;

The labels are typically boolean values (true or false), *<mark style="color:yellow;">**but you can use other values as well,**</mark>* depending on your specific requirements.

The choice of label values depends on the task and the desired behavior of the model during training. Here are a few common scenarios:

#### <mark style="color:green;">Binary Labels (true/false)</mark>

* When using binary labels, a segment with "label": true is considered as part of the input or output that the model should learn from during training.
* A segment with "label": false is masked or ignored during training, meaning the model is not trained on that particular segment.
* Binary labels are commonly used when you want to selectively train the model on specific parts of the input or output while excluding others.

#### <mark style="color:green;">Multiple Labels</mark>

* In some cases, you might want to assign different labels to different segments to indicate their roles or importance in the training process.
* For example, you could use labels like "input", "output", "context", or "meta" to differentiate between different types of segments.
* By assigning different labels, you can control how the model processes and learns from each segment during training.

#### <mark style="color:green;">Numerical Labels</mark>

* Numerical labels can be used to assign weights or priorities to different segments.
* For instance, you could use labels like 1, 2, or 3 to indicate the importance or relevance of each segment in the training process.
* Higher numerical labels could be used for segments that are more critical for the model to learn from, while lower labels could be used for less important segments.

The choice of label values ultimately depends on how you want to control the model's learning process and what segments you consider important for training.

### <mark style="color:blue;">How LLMs Learn from Structured Datasets</mark>

* During fine-tuning, the model <mark style="color:yellow;">**processes the structured dataset**</mark> and <mark style="color:yellow;">**adjusts its internal parameters**</mark> to better <mark style="color:yellow;">**understand the relationships between input and output data.**</mark>
* The transformer architecture of the model, with its attention mechanism, allows the model to focus on relevant parts of the input and generate appropriate outputs based on the learned patterns.
* By encountering multiple examples in the dataset with consistent labelling and structure, the model gradually learns to generate outputs that match the expected format and content.
* The model learns to associate specific labels (e.g., "instruction," "input," "output") with their respective roles in the task, enabling it to generate coherent and relevant responses.

### <mark style="color:blue;">Benefits of Structured Datasets</mark>

* Structured datasets provide a <mark style="color:yellow;">**clear and consistent format**</mark> for the model to learn from, reducing ambiguity and improving the model's understanding of the task.
* <mark style="color:yellow;">**Consistent labelling and syntax**</mark> enable the model to generate outputs that adhere to the expected format, making it easier to integrate the model into downstream applications.
* Well-structured datasets facilitate the fine-tuning process by providing the model with <mark style="color:yellow;">**clear examples of input-output relationships**</mark>, leading to better performance and generalisation.

### <mark style="color:blue;">Considerations for Dataset Structure</mark>

* Choose a dataset format that aligns with the specific task or behavior you want the model to learn.
* Ensure consistency in labelling and syntax throughout the dataset to avoid confusion during the learning process.
* Use clear and descriptive labels that accurately represent the role of each data point in the task or conversation.
* Consider the balance between providing enough context and keeping the dataset concise to optimize the learning process.

By understanding the importance of dataset structure, common formats, labelling, and syntax, you can create well-structured datasets that enable LLMs to effectively learn and generate desired outputs during the fine-tuning process.&#x20;

The transformer architecture of LLMs, with its attention mechanism, leverages the structured nature of the dataset to learn patterns, associations, and relationships between input and output data, ultimately improving the model's performance on the target task.

### <mark style="color:blue;">Prompt Construction</mark>

The <mark style="color:yellow;">**`input_output`**</mark> format is described as an alternative to using predefined templates (like 'alpaca' or 'chatml') which can add unnecessary complexity or limit flexibility.&#x20;

With <mark style="color:yellow;">**`input_output`**</mark>, you have more control over the exact structure of your prompts.

The key feature of <mark style="color:yellow;">**`input_output`**</mark> is the ability to mask certain segments of your prompts so that the model doesn't train on them. This is done by setting <mark style="color:yellow;">**`train_on_inputs: false`**</mark> in your configuration.

To use <mark style="color:yellow;">**`input_output`**</mark>, you prepare your data in a <mark style="color:yellow;">specific JSON Lines (JSONL) format</mark>.&#x20;

Each line in the JSONL file represents a single prompt and consists of a series of "segments".

Each segment has two properties:

* "text": The actual text content of this segment.
* "label": A boolean indicating whether the model should train on this segment (true) or mask it (false).

Here are five different types of dataset structures using the input/output format for various domain use cases:

#### <mark style="color:green;">Sentiment Analysis</mark>

In this example, we'll create a dataset for sentiment analysis of movie reviews.&#x20;

The input will be a movie review, and the output will be the sentiment label (positive, negative, or neutral).

The segment with the movie review text has "label": true because it is the input that the model should learn from.

The segment with the sentiment label (e.g., "positive") has "label": false because it is the expected output that the model should generate, not learn from.

```json
{
  "segments": [
    {
      "label": true,
      "text": "<s>I absolutely loved this movie! The acting was superb, and the plot kept me engaged from start to finish. "
    },
    {
      "label": false,
      "text": "positive"
    },
    {
      "label": true,
      "text": "</s>"
    }
  ]
}
```

#### <mark style="color:green;">Named Entity Recognition (NER)</mark>

This dataset structure is designed for named entity recognition tasks, where the goal is to identify and classify named entities in a given text. The input will be a sentence, and the output will be the same sentence with named entities marked.

```json
{
  "segments": [
    {
      "label": true,
      "text": "<s>Apple Inc. is planning to launch the iPhone 15 in September 2023 in Cupertino, California. "
    },
    {
      "label": false,
      "text": "<ORG>Apple Inc.</ORG> is planning to launch the <PROD>iPhone 15</PROD> in <DATE>September 2023</DATE> in <LOC>Cupertino, California</LOC>."
    },
    {
      "label": true,
      "text": "</s>"
    }
  ]
}
```

#### <mark style="color:green;">Text Summarization</mark>

For text summarization tasks, the input will be a longer piece of text (e.g., a news article), and the output will be a concise summary.

```json
{
  "segments": [
    {
      "label": true,
      "text": "<s>In a groundbreaking discovery, scientists have found a new species of dinosaur in the Gobi Desert. The dinosaur, named Mongolraptor, is believed to be a close relative of the Velociraptor. The discovery sheds new light on the diversity of dinosaur species during the Late Cretaceous period. The findings were published in the journal Nature on Tuesday.\n"
    },
    {
      "label": false,
      "text": "Scientists discovered a new dinosaur species, Mongolraptor, in the Gobi Desert. The finding, published in Nature, reveals new information about dinosaur diversity in the Late Cretaceous period."
    },
    {
      "label": true,
      "text": "</s>"
    }
  ]
}
```

#### <mark style="color:green;">Machine Translation</mark>

This dataset structure is suitable for machine translation tasks, where the input is a sentence in one language, and the output is the translated sentence in another language.

```json
{
  "segments": [
    {
      "label": true,
      "text": "<s>Der schnelle braune Fuchs springt über den faulen Hund.\n"
    },
    {
      "label": false,
      "text": "The quick brown fox jumps over the lazy dog."
    },
    {
      "label": true,
      "text": "</s>"
    }
  ]
}
```

#### <mark style="color:green;">Dialogue Act Classification</mark>

In this example, we'll create a dataset for dialogue act classification, where the goal is to classify the intent of each utterance in a conversation. The input will be a conversational utterance, and the output will be the corresponding dialogue act label.

```json
{
  "segments": [
    {
      "label": true,
      "text": "<s>User: Hey, can you help me find a good Italian restaurant nearby?\n"
    },
    {
      "label": false,
      "text": "request_recommendation"
    },
    {
      "label": true,
      "text": "\nAssistant: Sure, I'd be happy to help! What's your price range and preferred location?\n"
    },
    {
      "label": false,
      "text": "request_information"
    },
    {
      "label": true,
      "text": "</s>"
    }
  ]
}
```

These examples demonstrate how the input/output format can be adapted for various tasks and domains.&#x20;

By carefully designing the structure of your dataset and deciding which segments to label for training, you can create custom datasets tailored to your specific use case.

### <mark style="color:blue;">How to configure the YAML file for input output</mark>

To use the <mark style="color:yellow;">**`input_output`**</mark> format, you specify it in your Axolotl configuration file:

```yaml
datasets:
  - path: output.jsonl
    type: input_output
```

When you run the preprocessing step with the <mark style="color:yellow;">**`--debug`**</mark> flag, Axolotl will print out the tokens along with their labels so you can verify that the correct segments are being masked.

* A <mark style="color:yellow;">**label of 1**</mark> means the token will be trained on.
* A <mark style="color:yellow;">**label of -100**</mark> means the token will be masked.

You can also inspect the materialised data after preprocessing to ensure your prompts are being assembled correctly.  This involves loading the tokenized data and decoding it back into text to see the final prompt structure.

The <mark style="color:yellow;">**`input_output`**</mark> format provides a flexible, template-free way to construct prompts for fine-tuning.&#x20;

By allowing you to mask specific segments, it gives you fine-grained control over what parts of your prompts the model actually learns from.&#x20;

The JSONL structure with "segments", "text", and "label" properties is a clear and machine-readable way to define these prompts.

### <mark style="color:blue;">A medical dataset</mark>

This dataset is a collection of medical question-answering data, intended for fine-tuning a language model to perform medical question answering tasks.&#x20;

Here's the table format of the dataset:

<table><thead><tr><th width="156">Column Name</th><th width="128">Data Type</th><th>Description</th></tr></thead><tbody><tr><td>id</td><td>int64</td><td>Unique identifier for each entry</td></tr><tr><td>ending0</td><td>string</td><td>Possible ending or option for the scenario</td></tr><tr><td>ending1</td><td>string</td><td>Possible ending or option for the scenario</td></tr><tr><td>ending2</td><td>string</td><td>Possible ending or option for the scenario</td></tr><tr><td>ending3</td><td>string</td><td>Possible ending or option for the scenario</td></tr><tr><td>ending4</td><td>string</td><td>Possible ending or option for the scenario</td></tr><tr><td>label</td><td>int64</td><td>Correct answer or label for the scenario</td></tr><tr><td>sent1</td><td>string</td><td>Sentence or statement part of the scenario</td></tr><tr><td>sent2</td><td>string</td><td>Continuation or second part of the scenario</td></tr><tr><td>startphrase</td><td>string</td><td>Initial phrase or context for the scenario</td></tr></tbody></table>

#### <mark style="color:green;">Let's break down the structure</mark>

The dataset is structured in a tabular format, with each row representing a single question-answer pair. The key columns are:

1. <mark style="color:yellow;">**`sent1`**</mark><mark style="color:yellow;">**:**</mark> This column contains the medical question or scenario. It provides the context for the question being asked.
2. <mark style="color:yellow;">**`sent2`**</mark><mark style="color:yellow;">**:**</mark> This column contains the specific question related to the medical scenario described in `sent1`.
3. <mark style="color:yellow;">**`ending0`**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**to**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`ending4`**</mark><mark style="color:yellow;">**:**</mark> These five columns <mark style="color:yellow;">**represent the possible answer choices**</mark> for the question. Each column contains a different answer option.
4. <mark style="color:yellow;">**`label`**</mark><mark style="color:yellow;">**:**</mark> This column <mark style="color:yellow;">**indicates the correct answer choice**</mark> for the given question. It is an integer value ranging from 0 to 4, corresponding to the `ending0` to `ending4` columns.

<mark style="color:green;">**Example Scenario**</mark>

* **Id:** 1,754
* **Startphrase:** "An 8-year-old boy is brought to the paediatrician because his mother is concerned about recent behavioural changes."
* **Sent2:** "Which of the following trinucleotide repeats is this child most likely to possess?"
* **Options:**
  * **Ending0:** CGG
  * **Ending1:** GAA
  * **Ending2:** CAG
  * **Ending3:** CTG
  * **Ending4:** GCC
* **Correct Answer:** Label 1 (Indicates the most likely trinucleotide repeat)

These entries illustrate how the dataset is structured to <mark style="color:yellow;">**provide multiple-choice questions**</mark> related to medical scenarios, which can be used for training or testing in medical education and artificial intelligence applications.

The structure of this dataset, with questions, answer choices, and labels, is well-suited for fine-tuning a language model for multiple-choice question answering tasks.&#x20;

By providing the model with a variety of medical scenarios and questions during fine-tuning, it learns to understand the context and select the most appropriate answer from the given choices.

The fine-tuning process leverages the pre-existing knowledge of the language model and adapts it specifically for the medical domain and the question answering task.&#x20;

The model learns to associate patterns in the input text with the correct answer choices, enabling it to make accurate predictions on new, unseen medical questions.


# Downloading Huggingface Datasets

Downloading methods

Hugging Face datasets can be <mark style="color:yellow;">downloaded and loaded</mark> using various methods.&#x20;

Here's a summary:

<mark style="color:blue;">**From Hugging Face Hub Without a Loading Script**</mark>

You can load datasets directly from any dataset repository on the Hub using the <mark style="color:yellow;">**`load_dataset()`**</mark> function. Provide the repository namespace and dataset name to load the dataset.

<mark style="color:blue;">**Local Loading Script**</mark>

If you have a local HuggingFace Datasets loading script, you can load the dataset by specifying the local path to the loading script file or the directory containing it.

<mark style="color:blue;">**Local and Remote Files**</mark>

Datasets stored as CSV, JSON, TXT, Parquet, or Arrow files on your computer or remotely can be loaded using the <mark style="color:yellow;">**`load_dataset()`**</mark> function. Specify the file type and the path or URL to the data files.

<mark style="color:blue;">**In-memory Data**</mark>

You can create a dataset directly from in-memory data structures like Python dictionaries and Pandas DataFrames using functions like <mark style="color:yellow;">**`from_dict()`**</mark> and <mark style="color:yellow;">**`from_pandas()`**</mark><mark style="color:yellow;">**.**</mark>

<mark style="color:blue;">**Offline**</mark>

Datasets can be loaded offline if they are stored locally or if you have previously downloaded them.

<mark style="color:blue;">**Specific Slice of a Split**</mark>

You can load specific slices of a dataset split by using the <mark style="color:yellow;">**`split`**</mark> parameter in the <mark style="color:yellow;">**`load_dataset()`**</mark> function.

<mark style="color:blue;">**Multiprocessing**</mark>

For datasets consisting of several files, you can speed up the downloading and preparation using the <mark style="color:yellow;">**`num_proc`**</mark> parameter to set the number of processes for parallel execution.

<mark style="color:blue;">**SQL**</mark>

Datasets can be read from SQL databases using <mark style="color:yellow;">**`from_sql()`**</mark> by specifying the URI to connect to your database.

<mark style="color:blue;">**Arrow Streaming Format**</mark>

The Huggingface Datasets library can load local Arrow files directly using <mark style="color:yellow;">**`Dataset.from_file()`**</mark><mark style="color:yellow;">**.**</mark> This method memory-maps the Arrow file without preparing the dataset in the cache.

<mark style="color:blue;">**Python Generator**</mark>

A dataset can be created from a Python generator with <mark style="color:yellow;">**`from_generator()`**</mark><mark style="color:yellow;">**.**</mark> This method supports loading data larger than available memory and can also define a sharded dataset.

### <mark style="color:blue;">Key Features and Functionalities</mark>

* **Flexibility**: The library <mark style="color:yellow;">**can handle datasets stored in various formats and locations, including local and remote repositories, and in-memory data.**</mark>
* **Dataset Splits**: Data can be <mark style="color:yellow;">**mapped to specific splits like 'train', 'test', and 'validation'**</mark> using the <mark style="color:yellow;">**`data_files`**</mark> parameter. This parameter accepts file paths mapped to split names.
* **Version Control**: You can load different versions of a dataset based on Git tags, branches, or commits using the <mark style="color:yellow;">**`revision`**</mark> parameter.
* **Subset Loading**: For large datasets, you have the option to load only a subset of files, which is useful for large datasets like C4 (around 13TB).
* **Pattern Matching**: Load files that match specific patterns or from specified directories within a dataset repository.
* **No Loading Script Required**: The library allows loading datasets without the need for a custom loading script, simplifying the process.

### <mark style="color:blue;">Custom Datasets</mark>

* **Custom Dataset Repositories**: Users can create their dataset repositories on the Hugging Face Hub, facilitating easy sharing and loading of datasets.


# Use Git to download dataset

Using git to download the datasets

You should be connected to the Huggingface Hub as well as having git installed and prepared on the virtual machine.  If you do not, for reference:

### <mark style="color:blue;">**Install Git LFS**</mark>

* First, ensure that Git LFS is installed on your machine. If it's not installed, you can download and install it from [Git LFS' website](https://git-lfs.github.com/).
* On most systems, you can install Git LFS using a package manager. For instance, on Ubuntu, you can use:

```bash
sudo apt-get install git-lfs
```

<mark style="color:green;">**Initialise Git LFS**</mark>

After installation, you need to set up Git LFS. In your terminal, run:

```bash
git lfs install
```

The output should be as follows:

```bash
Updated git hooks.
Git LFS initialized.
```

Navigate to the Huggingface datasets repository and search for the dataset you wish to download.  In this case we will download the 'alpaca-cleaned' dataset.

{% embed url="<https://huggingface.co/datasets>" %}

When you are in the datasets website, enter in the name of the required dataset below in the 'filtered by name' input box.  in this case, filter by <mark style="color:yellow;">'alpaca\_cleaned'</mark>

<div data-full-width="false"><figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FnGL54HbJgU8fbK851yHA%2FScreenshot%202023-12-24%20165803.png?alt=media&amp;token=05389075-5268-4b56-a75d-8bb72e78b1d8" alt=""><figcaption><p>The main landing page for Huggingface Datasets</p></figcaption></figure></div>

<details>

<summary><mark style="color:green;">Why are we using the Alpaca Cleaned dataset?</mark></summary>

The Alpaca-Cleaned dataset is a <mark style="color:yellow;">refined version of the original Alpaca Dataset</mark> from Stanford, addressing several identified issues to improve its quality and utility for instruction-tuning of language models. Key aspects of this dataset include:

<mark style="color:purple;">**Dataset Description and Corrections**</mark><mark style="color:purple;">:</mark>

* <mark style="color:green;">**Hallucinations**</mark><mark style="color:green;">:</mark> Fixed instances where the original dataset's instructions caused the model to <mark style="color:blue;">generate baseless answers,</mark> typically related to external web content.
* <mark style="color:green;">**Merged Instructions**</mark><mark style="color:green;">:</mark> Separated instructions that were improperly combined in the original dataset.
* <mark style="color:green;">**Empty Outputs**</mark><mark style="color:green;">:</mark> Addressed entries with missing outputs in the original dataset.
* <mark style="color:green;">**Missing Code Examples**</mark><mark style="color:green;">:</mark> Supplemented descriptions that lacked necessary code examples.
* <mark style="color:green;">**Image Generation Instructions**</mark><mark style="color:green;">:</mark> Removed unrealistic instructions for generating images.
* <mark style="color:green;">**N/A Outputs and Inconsistent Inputs**</mark>: Corrected code snippets with N/A outputs and standardized the formatting of empty inputs.
* <mark style="color:green;">**Incorrect Answers**</mark><mark style="color:green;">:</mark> Identified and fixed wrong answers, particularly in math problems.
* <mark style="color:green;">**Unclear Instructions**</mark><mark style="color:green;">:</mark> Clarified or re-wrote non-sensical or unclear instructions.
* <mark style="color:green;">**Control Characters**</mark><mark style="color:green;">:</mark> Removed extraneous escape and control characters present in the original dataset.

<mark style="color:purple;">**Original Alpaca Dataset Overview**</mark>

* Consisting of 52,000 instructions and demonstrations generated by OpenAI's `text-davinci-003` engine.
* Aimed at instruction-tuning to enhance language models' ability to follow instructions.
* Modifications from the original data generation pipeline include using `text-davinci-003`, a new prompt for instruction generation, and a more efficient data generation approach.
* The dataset is noted for its diversity and cost-effectiveness in generation.

<mark style="color:purple;">**Dataset Structure and Contents**</mark><mark style="color:purple;">:</mark>

* Fields include `instruction` (task description), `input` (context or additional information), and `output` (answer generated by `text-davinci-003`).
* The `text` field combines these elements using a specific prompt template.
* The dataset is primarily structured for training purposes, with 52,002 instances in the training split.

<mark style="color:purple;">**Intended Use and Considerations**</mark><mark style="color:purple;">:</mark>

* Primarily designed for training pretrained language models on instruction-following tasks.
* The dataset, primarily in English, poses potential risks like harmful content dissemination and requires careful use and further refinement to address errors or biases.

</details>

After filtering by dataset name, you will seen all the datasets attributable to that name.  We will be downloading <mark style="color:yellow;">yahma/alpaca\_cleaned</mark>:

<div data-full-width="false"><figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2Fk8g5WIOvSYBMI6QsBpuu%2Falpacale.png?alt=media&amp;token=03bf439f-362d-4189-b6b5-22bbd68b2e1c" alt=""><figcaption><p>The Alpaca_Cleaned dataset is highlighted in the yellow</p></figcaption></figure></div>

Once in the dataset repository, click on the button with <mark style="color:yellow;">three dots</mark> positioned horizontally.  This will provide the opportunity to use <mark style="color:yellow;">git clone</mark> to download the dataset to your directory.

<div data-full-width="false"><figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FUpehBkK8Y5nG63BcxEUR%2Fsss.png?alt=media&amp;token=8a6ffafa-739e-4de6-82af-7a15cd7dbdb7" alt=""><figcaption></figcaption></figure></div>

When you click on the three horizontal dolts, a <mark style="color:yellow;">dialog box appears providing the command line for a git clone download of the dataset</mark>.  Follow the instructions below to git clone the dataset into the axolotl environment.

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FCQWUAQLmmJtIZ6SSYDPY%2Fss.png?alt=media&amp;token=5251bdb8-b113-4c38-8f84-ec99d176a3ff" alt=""><figcaption><p>git clone dataset</p></figcaption></figure>

Go into the primary axolotl directory and then enter the following command:

```bash
git clone https://huggingface.co/datasets/yahma/alpaca-cleaned
```

This command will create a folder called datasets and download the specified Huggingface dataset into it.

<details>

<summary><mark style="color:green;">Git Clone deconstruction</mark></summary>

In the context of the command `git clone`` `<mark style="color:yellow;">`https://huggingface.co/datasets/yahma/alpaca-cleaned`</mark>, "yahma" refers to the username or organisation name within the Hugging Face Datasets repository. Here's a breakdown of the components of this command:

1. <mark style="color:yellow;">**`git clone`**</mark><mark style="color:yellow;">:</mark> This is a Git command used to clone a repository. It makes a copy of the specified repository and downloads it to your local machine.
2. <mark style="color:yellow;">**`https://huggingface.co/datasets`**</mark><mark style="color:yellow;">:</mark> This URL points to the Hugging Face Datasets repository. Hugging Face hosts machine learning models, datasets, and related tools. The <mark style="color:yellow;">`/datasets`</mark> part indicates that the repository being cloned is a dataset repository.
3. <mark style="color:yellow;">**`yahma`**</mark><mark style="color:yellow;">:</mark> This is the username or the name of the organisation on the Hugging Face platform that owns the repository you are cloning. In this case, 'yahma' is the entity that has uploaded or maintained the dataset named 'alpaca-cleaned'.
4. <mark style="color:yellow;">**`alpaca-cleaned`**</mark><mark style="color:yellow;">:</mark> This is the name of the specific dataset repository under the user or organization 'yahma' on Hugging Face. The name suggests it might be a cleaned or processed version of a dataset related to "alpaca".

When you run this command, you are cloning the 'alpaca-cleaned' dataset from the 'yahma' user or organisation's space on Hugging Face to your local machine. This allows you to use or analyze the dataset directly on your computer.

</details>

{% hint style="warning" %}

## <mark style="color:orange;">**Remember where you stored your dataset - it is required for when you prepare your Axolotl configuration file**</mark>

{% endhint %}


# Popular Datasets

is is an analysis of the most popular datasets.  You can skip this section if you want to move forward to downloading the dataset for your fine tuning run.

### <mark style="color:blue;">Alpaca Cleaned</mark>

The cleaned Alpaca Dataset is a *<mark style="color:yellow;">**curated and cleaned version of the original dataset**</mark>* used to train the Alpaca Large Language Model (LLM).&#x20;

The original dataset, generated using GPT-3, had several issues that impacted its quality and usefulness for training machine learning models. The cleaned version addresses these problems to improve the performance of models trained on this data.

{% embed url="<https://huggingface.co/datasets/yahma/alpaca-cleaned>" %}
Alpaca Cleaned Dataset
{% endembed %}

#### <mark style="color:green;">Key points</mark>

1. The cleaned dataset fixes issues such as hallucinations, merged instructions, empty outputs, missing code examples, incorrect answers, and unclear instructions.
2. On April 8, 2023, the remaining uncurated instructions (around 50,000) were replaced with data from the GPT-4-LLM dataset. Curation of the new data is ongoing.
3. The average prompt length in the cleaned dataset is longer than the original, with many prompts exceeding 256 tokens. It is recommended to set the maximum prompt length to at least 512 or higher during fine-tuning.

This cleaned dataset aims to provide a higher-quality resource for training and fine-tuning LLMs, ultimately leading to better-performing models with reduced hallucinations.

### <mark style="color:blue;">Open Orca</mark>

The OpenOrca dataset is a collection of augmented [FLAN Collection data](https://arxiv.org/abs/2301.13688). Currently \~1M GPT-4 completions, and \~3.2M GPT-3.5 completions.&#x20;

It is tabularised in alignment with the distributions presented in the ORCA paper and currently represents a partial completion of the full intended dataset, with ongoing generation to expand its scope.&#x20;

{% embed url="<https://huggingface.co/datasets/Open-Orca/OpenOrca>" %}
Open Orca Dataset
{% endembed %}

{% embed url="<https://arxiv.org/abs/2306.02707>" %}
Orca Paper
{% endembed %}

{% embed url="<https://arxiv.org/abs/2301.13688>" %}
Flan Collection
{% endembed %}

### <mark style="color:blue;">Financial Phrase Databank</mark>

The Financial Phrase Bank dataset is a collection of approximately 5,000 English sentences from financial news articles, annotated for sentiment analysis. The sentences are labeled as positive, negative, or neutral based on their potential impact on stock prices, from an investor's perspective.

{% embed url="<https://huggingface.co/datasets/financial_phrasebank>" %}
Financial Phrase Bank dataset&#x20;
{% endembed %}

Key features of the dataset:

* Contains 4,840 sentences from English financial news
* Sentences are labeled as 'positive', 'negative', or 'neutral'
* Annotations were performed by 16 people with financial background
* Available in four configurations based on annotator agreement percentages: 50%, 66%, 75%, and 100%
* No predefined train/validation/test split
* Sourced from a subset of 10,000 articles covering various companies, industries, and news sources
* Annotators were from the same institution (Aalto University School of Business)

This dataset is useful for training and benchmarking sentiment analysis models in the financial domain.&#x20;

### <mark style="color:blue;">Evol-Instruct</mark>

The training dataset for this fine tuned model is initialised with Code Alpaca, a 20K instruction-following dataset.&#x20;

The Evol-Instruct technique is then applied to evolve the dataset, with the evolved data being merged with the original dataset after each round.&#x20;

{% embed url="<https://arxiv.org/abs/2306.08568>" %}
WizardCoder
{% endembed %}

#### <mark style="color:green;">Dataset Focus and Views</mark>

The dataset used for training WizardCoder plays a crucial role in its superior performance.

The authors start with Code Alpaca, a 20K instruction-following dataset, and *<mark style="color:yellow;">**iteratively evolve it using the Code Evol-Instruct method.**</mark>*&#x20;

{% embed url="<https://huggingface.co/datasets/nickrosh/Evol-Instruct-Code-80k-v1>" %}
The Evol-Instruct dataset
{% endembed %}

This process *<mark style="color:yellow;">**generates a more complex and diverse dataset**</mark>* tailored specifically for code-related tasks.

The adaptation of Evol-Instruct to the code domain involves several key modifications:

1. Refining evolutionary instructions by removing deepening, complicating input, and In-Breadth Evolving.
2. Simplifying the form of evolutionary prompts by unifying the prompt template.
3. Incorporating code-specific evolutionary instructions, such as code debugging and time-space complexity constraints.

These modifications demonstrate the importance of domain-specific adaptations when creating datasets for training LLMs.&#x20;

By considering the unique characteristics and requirements of the code domain, the authors were able to generate a dataset that effectively enhances the performance of the Code LLM.

### <mark style="color:blue;">Creating Datasets for LLM training</mark>

When creating similar datasets for training LLMs, I believe the following points should be considered:

1. Start with a high-quality initial dataset relevant to the target domain.
2. Identify domain-specific characteristics and requirements that can be incorporated into the dataset evolution process.
3. Develop a set of evolutionary instructions and prompts that align with the domain-specific goals and constraints.
4. Iteratively evolve the dataset using the adapted evolutionary instructions, evaluating the performance of the model after each round of evolution.
5. Continuously refine the evolutionary process based on the observed performance and domain-specific insights gained during the training process.

By following these guidelines and adapting the Evol-Instruct method to the specific domain of interest, researchers can create high-quality datasets that enable the training of LLMs with superior performance in their respective domains.

{% embed url="<https://arxiv.org/abs/2304.12244>" %}

### <mark style="color:blue;">The Persuasion Dataset</mark>

The Persuasion Dataset is structured to provide a comprehensive understanding of the effectiveness of various arguments, whether human-written or model-generated, in altering a person's stance on specific claims. Here is a structured table summarizing the dataset's components:

<table data-header-hidden><thead><tr><th width="268"></th><th></th></tr></thead><tbody><tr><td><strong>Column Name</strong></td><td><strong>Description</strong></td></tr><tr><td><strong>worker_id</strong></td><td>Identifier for the participant who annotated their stance.</td></tr><tr><td><strong>claim</strong></td><td>The statement or assertion presented for argument.</td></tr><tr><td><strong>argument</strong></td><td>The argument provided, crafted by either a human or a language model.</td></tr><tr><td><strong>source</strong></td><td>Indicates whether the argument was generated by a human or a specific model.</td></tr><tr><td><strong>prompt_type</strong></td><td>The type of prompt used to generate the argument.</td></tr><tr><td><strong>rating_initial</strong></td><td>Participant's initial rating of the claim before reading the argument.</td></tr><tr><td><strong>rating_final</strong></td><td>Participant's final rating of the claim after being exposed to the argument.</td></tr><tr><td><strong>persuasiveness_metric</strong></td><td>Numerical score indicating the persuasiveness of the argument (not shown in the initial explanation but deducible from context).</td></tr></tbody></table>

#### <mark style="color:green;">Explanation of the Dataset Structure</mark>

<mark style="color:purple;">**Worker ID**</mark><mark style="color:purple;">:</mark> This is crucial for tracking the responses from individual participants across multiple claims or arguments, ensuring the data is consistent and attributed correctly.

<mark style="color:purple;">**Claim**</mark><mark style="color:purple;">:</mark> The focal point of the dataset; these are statements or topics on which arguments are based. Understanding the diversity and nature of claims is essential for analyzing how different types of arguments perform.

<mark style="color:purple;">**Argument:**</mark> Central to the dataset, these entries show the persuasive text presented to participants. The content and quality of these arguments are key to the research being conducted.

<mark style="color:purple;">**Source:**</mark> Distinguishing between human and model-generated arguments allows researchers to compare the effectiveness of natural versus artificial persuasive techniques.

<mark style="color:purple;">**Prompt Type:**</mark> Knowing the prompt type helps in understanding the context or angle from which the argument was developed, which can influence its persuasiveness.

<mark style="color:purple;">**Ratings (Initial and Final):**</mark> These metrics are critical as they provide a before-and-after snapshot of the participant's stance on a claim, showing the direct impact of the argument. The change between these ratings can be used as a quantitative measure of argument effectiveness.

<mark style="color:purple;">**Persuasiveness Metric:**</mark> Although not included in the initial column listing, it's plausible that such a metric could be calculated from the difference between initial and final ratings, quantifying how persuasive an argument was.

This dataset can be immensely useful for developing and refining language models aimed at persuasive writing. It also provides insights into human cognitive biases and response patterns to different forms of rhetoric, aiding in fields like marketing, political science, and psychology.

Below is some more background on this interesting dataset:

<details>

<summary><mark style="color:green;"><strong>Anthropic Persuasion</strong></mark></summary>

The paper "Measuring the Persuasiveness of Language Models" by Anthropic researchers explores the relationship between the scale of AI language models and their ability to generate persuasive arguments.&#x20;

The study compares the persuasiveness of arguments generated by various Anthropic models (Claude 1, 2, and 3) and two classes of models (compact and frontier) with human-written arguments.

<mark style="color:green;">**Methodology**</mark>

1. The researchers curated 28 topics on complex and emerging issues where people are less likely to have hardened views, such as online content moderation, ethical guidelines for space exploration, and the appropriate use of AI-generated content. They created 56 opinionated claims, with supporting and opposing claims for each topic.
2. For human-written arguments, three participants were randomly assigned to each claim and asked to write persuasive messages of approximately 250 words. Participants were informed that their submissions would be evaluated, with the most persuasive author receiving additional compensation.
3. For AI-generated arguments, the researchers used four distinct prompts to generate arguments: Compelling Case, Role-playing Expert, Logical Reasoning, and Deceptive. The ratings of changed opinions were averaged across these prompts to calculate the persuasiveness of the AI-generated arguments.
4. To assess persuasiveness, participants were shown a claim without an accompanying argument and asked to report their initial level of support on a 1-7 Likert scale. They were then shown an argument supporting the claim, written by either a human or an AI model, and asked to re-rate their stance. The persuasiveness metric was defined as the difference between the final and initial support scores.
5. As a control condition, the researchers presented participants with Claude 2 generated arguments that attempt to refute indisputable factual claims to quantify opinion changes due to extraneous factors.

<mark style="color:green;">Findings</mark>

1. Claude 3 Opus was found to be roughly as persuasive as humans, with no statistically significant difference.
2. A general scaling trend was observed: as models get larger and more capable, they become more persuasive.
3. The control condition worked as expected, with the persuasiveness score close to zero for indisputable factual claims.

<mark style="color:green;">Lessons Learned: The researchers discussed the limitations of their study, including:</mark>

1. Persuasion is difficult to study in a lab setting, and the results may not transfer to the real world.
2. Evaluating the persuasiveness of arguments is inherently subjective.
3. The experimental design has limitations, such as studying only single-turn arguments, using human writers who may not be experts in persuasion, and not exploring human + AI collaboration.
4. Cultural and linguistic context may limit the generalizability of the findings.
5. The study might suffer from an anchoring effect, limiting the magnitude of the persuasiveness effect observed.
6. Different prompting methods work differently across models, with the Deceptive strategy being the most persuasive overall.
7. The researchers did not measure the longer-term effects of being exposed to persuasive arguments.

</details>


# Download cleaned Alpaca dataset

Test the engines

The <mark style="color:yellow;">last instruction</mark> entered was to git clone the alpaca-cleaned dataset to the local directory:&#x20;

```bash
git clone https://huggingface.co/datasets/yahma/alpaca-cleaned
```

This command downloaded this Huggingface <mark style="color:yellow;">42MB json dataset</mark> into the directory you created called **datasets.**

**Within datasets, this directory is located at alpaca-cleaned.  The full path is:**

*<mark style="color:blue;">your primary directory</mark>*/<mark style="color:yellow;">axolotl</mark>/<mark style="color:purple;">datasets</mark>/<mark style="color:green;">alpaca-cleaned</mark>

The screenshot below shows the contents of the alpaca-cleaned dataset.  Note that it is in JSON format and that the training set is in Alpaca format:

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2Fri7aHYnC0GYr0GN1eBON%2FScreenshot%202023-12-05%20114449.png?alt=media&amp;token=78ea6bb0-cf92-42ac-90b9-de0a1a7f6c9f" alt=""><figcaption><p>A screenshot from VS Code demonstrating the contents of the alpaca-cleaned dataste</p></figcaption></figure>

### <mark style="color:blue;">What is Alpaca format?</mark>

When using <mark style="color:blue;">instruction fine tuning.</mark> there are various formats for the training set.  The Alpaca format has become one of the 'standards' for the structure of a dataset

#### <mark style="color:green;">**Data Structure in**</mark><mark style="color:green;">**&#x20;**</mark><mark style="color:green;">**`alpaca_data.json`**</mark>

This dataset is formatted as a <mark style="color:blue;">JSON file</mark>, where each entry is represented as a dictionary with the following key-value pairs:

**Instruction&#x20;**<mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`instruction`**</mark><mark style="color:yellow;">**)**</mark><mark style="color:yellow;">:</mark>

* Type: String <mark style="color:yellow;">(</mark><mark style="color:yellow;">`str`</mark><mark style="color:yellow;">)</mark>
* Description: Specifies the task to be performed by the model.

**Input (`input`)**:

* Type: String <mark style="color:yellow;">(</mark><mark style="color:yellow;">`str`</mark><mark style="color:yellow;">)</mark> optional.
* Description: Provides additional context or information needed to perform the task described in the <mark style="color:yellow;">`instruction`</mark><mark style="color:yellow;">.</mark>
* Example: If the instruction is "Summarize the following article", the input would be the text of the article.

Prevalence: In the original 52k Alpaca dataset, approximately <mark style="color:yellow;">40% of the entries</mark> in the dataset include an `input` field.

**Output&#x20;**<mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`output`**</mark><mark style="color:yellow;">**)**</mark><mark style="color:yellow;">:</mark>

* Type: String <mark style="color:yellow;">(</mark><mark style="color:yellow;">`str`</mark><mark style="color:yellow;">)</mark>
* Description: The response generated by the text-davinci-003 model, which represents the answer or completion of the task defined in the <mark style="color:yellow;">`instruction`</mark>.

### <mark style="color:blue;">**Fine-Tuning Prompts for Alpaca Model**</mark>

Two distinct prompt structures were used in the fine-tuning process, depending on whether the <mark style="color:yellow;">`input field`</mark> is present or not.

**For Entries with&#x20;**<mark style="color:yellow;">**Non-Empty Input Field**</mark>:

{% code fullWidth="false" %}

```json
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input}

### Response:
```

{% endcode %}

**For Entries with&#x20;**<mark style="color:yellow;">**Empty Input Field**</mark>:

```json
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:
```

**For a full review of the different types of dataset techniques and structures used in Axolotl please visit datasets.**


# Template-free prompt construction

Sourced from the Axolotl documentation with permission

Template-free prompt construction with the <mark style="color:yellow;">**`input_output`**</mark> format

### <mark style="color:blue;">Background</mark>

#### <mark style="color:green;">Masking Inputs</mark>

One of the most popular features of [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) is setting the following configuration value:

```bash
train_on_inputs: false
```

If you declare a [dataset formats](https://github.com/OpenAccess-AI-Collective/axolotl?tab=readme-ov-file#dataset) such as <mark style="color:yellow;">**`alpaca`**</mark> or <mark style="color:yellow;">**`chatml`**</mark>, axolotl knows what is an input (i.e. human) vs. an output (i.e. the assistant) and <mark style="color:yellow;">masks the input labels</mark> so that your model can focus on predicting the outputs only.

### <mark style="color:blue;">You may not want prompt templates</mark>

However, there are many situations where you don’t want to use one of these formats or templates. This is because they can:

* Add unnecessary boilerplate to your prompts.
* Create artifacts like special delimiters <mark style="color:yellow;">`<|im_start|>`</mark> that can quickly become footguns if you don’t include them correctly at inference time.
* Enforce a *chat* interface when you do not want one. Sometimes you just want to fine-tune a model to a very specific task and do NOT want multi-turn conversations, roles, etc.
* Limit you to only certain roles that the template allows.

#### <mark style="color:green;">The</mark> <mark style="color:green;"></mark><mark style="color:green;">`input_output`</mark> <mark style="color:green;"></mark><mark style="color:green;">format</mark>

You can construct your prompts without a template by using the `input_output` format, by setting `type: input_output` in your configuration file like this:

#### <mark style="color:green;">**config.yml**</mark>

```bash
train_on_inputs: false # Mask segments of your data
datasets:
  - path: output.jsonl
    type: input_output  # use template free prompt construction
```

Unlike <mark style="color:yellow;">**`type: completion`**</mark>, which is also template-free, <mark style="color:yellow;">**`type: input_output`**</mark> allows you to mask segments of your text. More details on how this works are described below.

## <mark style="color:blue;">Usage</mark>

This is how you can use the <mark style="color:yellow;">**`input_output`**</mark> format:

#### <mark style="color:green;">Prepare Data</mark>

To use the `input_output` format, collect your data in the following format into a <mark style="color:blue;">**jsonl file**</mark> (below is the first row from the file <mark style="color:yellow;">`output`</mark><mark style="color:yellow;">.jsonl\`</mark> pretty printed):

```bash
$ head -n1 output.jsonl | python -m json.tool

{.cell-output .cell-output-stdout}
    {
        "segments": [
            {
                "label": true,
                "text": "<s>Hello\n"
            },
            {
                "label": true,
                "text": "hi there!. "
            },
            {
                "label": false,
                "text": "goodbye "
            },
            {
                "label": true,
                "text": "farewell</s>"
            }
        ]
    }
```

Set <mark style="color:yellow;">**`label:false`**</mark> when you want to mask a segment of text so that the model isn’t trained on it. Some things to keep in mind:

{% hint style="warning" %}
&#x20;<mark style="color:blue;">**EOS, BOS, spaces, newlines etc. are entirely up to you.**</mark>&#x20;

Axolotl concatenates all the segments <mark style="color:yellow;">as-is.</mark>&#x20;

The tokenizer doesn’t add anything additional. Notice how I added spaces, newlines, `<s>` (BOS), and `</s>` (EOS) myself.&#x20;

Make sure you check the materialised output to validate that the prompt is getting assembled how you like.
{% endhint %}

#### <mark style="color:green;">Use</mark> <mark style="color:green;"></mark><mark style="color:green;">`type:`</mark>` `<mark style="color:yellow;">`input_output`</mark>

Let’s materialise data with our <mark style="color:yellow;">`output.jsonl`</mark> file by setting <mark style="color:yellow;">`type: input_output`</mark> in our axolotl config:

```bash
# training_config.yaml
base_model: mistralai/Mistral-7B-v0.1
data_seed: 49
seed: 49

datasets:
  - path: output.jsonl
    type: input_output
val_set_size: 0.1

sequence_len: 896
sample_packing: false

micro_batch_size: 2
gradient_accumulation_steps: 3
eval_batch_size: 2
num_epochs: 1
learning_rate: 0.0002

train_on_inputs: false
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"
```

You can use the following command to materialise your data. The <mark style="color:yellow;">**`--debug`**</mark> flag will print the tokens, along with the labels so you can verify that the correct items are being ignored:

{% code overflow="wrap" %}

```bash
$ python -m axolotl.cli.preprocess training_config.yaml --debug

[2024-03-05 23:36:46,969] [INFO] [axolotl.check_example_labels:35] [PID:607731] [RANK:0] <s>(1, 1) Hello(22557, 22557)
(13, 13) hi(12014, 12014) there(736, 736) !(28808, 28808) .(28723, 28723) (28705, 28705) good(-100, 1179) bye(-100, 17664) (-100, 28705) fare(19111, 19111) well(5458, 5458) </s>(2, 2)
```

{% endcode %}

The format is <mark style="color:yellow;">**`decoded_token`**</mark><mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`label`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`token_id`**</mark><mark style="color:yellow;">**)**</mark>, for example, <mark style="color:yellow;">**`<s>(1, 1)`**</mark> means that the token is <mark style="color:yellow;">**`<s>`**</mark>, the label is <mark style="color:yellow;">**`1`**</mark> and the token\_id is <mark style="color:yellow;">**`1`**</mark>. When the label is <mark style="color:yellow;">**`-100`**</mark> then that token is ignored for training.

#### <mark style="color:green;">Check the prompts</mark>

Here is another way to check the materialised output:

```bash
from transformers import AutoTokenizer
from datasets import load_from_disk
import yaml

directory = !ls last_run_prepared/
with open('training_config.yaml', 'r') as f:
    cfg = yaml.safe_load(f)
model_id = cfg['base_model']
tok = AutoTokenizer.from_pretrained(model_id)
ds = load_from_disk(f'last_run_prepared/{directory[0]}/')
```

```
>>> row = ds[0]
>>> print(tok.decode(row['input_ids']))
<s> Hello
    hi there!.  goodbye  farewell</s>
```

We can check that the right tokens are ignored by comparing the labels to each token:

```bash
import pandas as pd
pd.DataFrame([{'token': tok.decode(i), 'label': l, 'id':i} for i,l in
              zip(row['input_ids'], row['labels'])])
```

<table><thead><tr><th width="166" align="center">token</th><th width="283" align="center">label</th><th align="center">id</th></tr></thead><tbody><tr><td align="center">0</td><td align="center">&#x3C;s></td><td align="center">1</td></tr><tr><td align="center">1</td><td align="center">Hello</td><td align="center">22557</td></tr><tr><td align="center">2</td><td align="center"></td><td align="center">13</td></tr><tr><td align="center">3</td><td align="center">hi</td><td align="center">12014</td></tr><tr><td align="center">4</td><td align="center">there</td><td align="center">736</td></tr><tr><td align="center">5</td><td align="center">!</td><td align="center">28808</td></tr><tr><td align="center">6</td><td align="center">.</td><td align="center">28723</td></tr><tr><td align="center">7</td><td align="center"></td><td align="center">28705</td></tr><tr><td align="center">8</td><td align="center">good</td><td align="center">-100</td></tr><tr><td align="center">9</td><td align="center">bye</td><td align="center">-100</td></tr><tr><td align="center">10</td><td align="center"></td><td align="center">-100</td></tr><tr><td align="center">11</td><td align="center">fare</td><td align="center">19111</td></tr><tr><td align="center">12</td><td align="center">well</td><td align="center">5458</td></tr><tr><td align="center">13</td><td align="center">&#x3C;/s></td><td align="center">2</td></tr></tbody></table>

If we look at the input data, the above table seems correct!&#x20;

The jsonl version is repeated below for reference:

```bash
$ head -n1 output.jsonl | python -m json.tool

{.cell-output .cell-output-stdout}
    {
        "segments": [
            {
                "label": true,
                "text": "<s>Hello\n"
            },
            {
                "label": true,
                "text": "hi there!. "
            },
            {
                "label": false,
                "text": "goodbye "
            },
            {
                "label": true,
                "text": "farewell</s>"
            }
        ]
    }
```


# Downloading models

General Introduction

We can download the model to our local host from the Huggingface Model Hub.

If you are unfamiliar with the Huggingface Model Hub, the Youtube video below provides a rundown.

{% embed url="<https://www.youtube.com/watch?t=2s&v=XvSGPZFEjDY>" %}
Worth watching despite the moustache
{% endembed %}

<details>

<summary><mark style="color:green;">Summary of Huggingface Model Hub</mark></summary>

Navigating the Hugging Face Model Hub:

1. Access the model hub by clicking on the <mark style="color:yellow;">"Models" tab</mark> in the <mark style="color:yellow;">upper right corner</mark> of the Hugging Face landing page.
2. The model hub interface is divided into several parts:
   * <mark style="color:blue;">Left side:</mark> Categories for tailoring model search
     * <mark style="color:green;">Tasks:</mark> Various tasks such as NLP, computer vision, and speech recognition
     * <mark style="color:green;">Libraries:</mark> Model backbones (PyTorch, TensorFlow, JAX) and high-level frameworks (transformers, etc.)
     * <mark style="color:green;">Datasets:</mark> Filter models trained on specific datasets
     * <mark style="color:green;">Languages:</mark> Filter models that handle selected languages
     * <mark style="color:green;">License:</mark> Choose the license under which the model is shared
   * <mark style="color:blue;">Right side:</mark> Available models on the hub, ordered by downloads by default
3. Clicking on a model <mark style="color:yellow;">opens its model card</mark>, which contains crucial information:

   * Description, intended use, limitations, and biases
   * Code snippets for model usage
   * Training procedure, data processing, evaluation results, and copyrights
   * Inference API on the right for testing the model with user inputs

   The "Files and versions" tab displays the model repositories' architecture, branches, commit history.

</details>

### <mark style="color:blue;">The Model Card</mark>

Model cards are files that accompany the models and provide handy information.&#x20;

Under the hood, model cards are simple Markdown files with additional metadata.&#x20;

Model cards are essential for discoverability, reproducibility, and sharing.  You can find a model card as the <mark style="color:yellow;">**`README.md`**</mark> file in any model repository.

The model card should describe:

* the model
* its intended uses and potential limitations, including biases and ethical considerations
* the training parameters and experimental info&#x20;
* which datasets were used to train your model
* the model’s evaluation results

### <mark style="color:blue;">Downloading Models from Hugging Face Hub</mark>

The Hugging Face Hub provides several ways to download and use models, depending on your requirements and the tools you are using. This documentation will guide you through the different methods available.

### <mark style="color:blue;">Using Git</mark>

All models on the Model Hub are stored as Git repositories, *<mark style="color:yellow;">**which means you can clone them locally using Git commands.**</mark>* To clone a model, follow these steps:

If you have not already, install Git LFS (Large File Storage) by running:

```bash
git lfs install
```

Clone the model repository using the following command:

```bash
git clone git@hf.co:<MODEL ID>
```

Replace <mark style="color:yellow;">`<MODEL ID>`</mark> with the actual model ID.  For example:

```bash
git clone git@hf.co:bigscience/bloom
```

If you have write access to a particular model repository, you can also commit and push revisions to the model.

By following these methods, you can easily download and use models from the Hugging Face Hub in your projects, whether you're using integrated libraries, the Hugging Face Client Library, or Git directly.


# Phi 2.0 details

Introductory fine tuning exercise

To introduce you to the process of fine tuning a language model, we will begin with Phi 2.0. &#x20;

Phi 2.0 is a <mark style="color:yellow;">small but powerful model</mark>, and a great way to begin learning how to fine tune a language model.

### <mark style="color:blue;">Phi 2.0 Review</mark>

Phi 2 is a <mark style="color:yellow;">relatively small model with 2.7 billion parameters,</mark> yet it outperforms models of comparable size like Mamba and Google's Gemini Nano, as well as models 20-25 times its size, according to the benchmarks.

Phi 2 was <mark style="color:yellow;">trained on high-quality synthetic data,</mark> including textbook-quality code, common sense reasoning, logic, science, and theory of mind exercises generated by GPT-3.5 and filtered by GPT-4. This synthetic data approach allowed for more training epochs.

Training on <mark style="color:yellow;">synthetic data tends to result in less toxic models,</mark> as evidenced by Phi 2's lower toxicity scores even before reinforcement learning.

The Phi 2 researchers believe that enormous amounts of compute have been wasted on ineffective training data, and that carefully curated synthetic data can lead to more efficient and higher-quality models.

Phi 2's performance suggests that achieving ChatGPT-level capabilities with a 1 billion parameter model may be possible. Extrapolating further, a 1.5 trillion parameter model trained this way could potentially imitate a 1.5 quadrillion parameter model.

However, Phi models are sensitive to prompt variations, and longer prompts may cause the model to forget, ignore, or misinterpret parts of the prompt.

The Phi 2 model itself is open-sourced, although the full training dataset has not been released yet.

The key takeaways are that Phi 2 demonstrates the potential of using high-quality synthetic data to train smaller, more efficient models that can rival the performance of much larger models, and that this approach could lead to significant advancements in AI capabilities in the near future. However, the model's sensitivity to prompts is a limitation to keep in mind.&#x20;

#### <mark style="color:blue;">Click on link below to review Phi 2.0 at Huggingface model repository</mark>

{% embed url="<https://huggingface.co/microsoft/phi-2>" %}
Huggingface model repository - Phi 2.0
{% endembed %}

The expandables below give you some insight as to the files that come with the Phi 2.0 model, and what they mean.

<details>

<summary><mark style="color:green;">Files and Versions - Explanation of Contents</mark></summary>

Here's an explanation of each file in the Hugging Face model card:

<mark style="color:yellow;">`.gitattributes`</mark><mark style="color:yellow;">:</mark> This file is used to define attributes for different file types in the Git repository. It can specify how certain files should be treated, such as whether they should be normalized or how line endings should be handled.

<mark style="color:yellow;">`LICENSE`</mark><mark style="color:yellow;">:</mark> This file contains the license under which the model is distributed. It specifies the terms and conditions for using, modifying, and distributing the model.

<mark style="color:yellow;">`README.md`</mark><mark style="color:yellow;">:</mark> This file provides an overview of the model, including its purpose, usage instructions, and any other relevant information. It serves as the main documentation for the model.

<mark style="color:yellow;">`added_tokens.json`</mark>: This file contains information about any additional tokens that have been added to the model's vocabulary beyond the standard tokens.

<mark style="color:yellow;">`config.json`</mark><mark style="color:yellow;">:</mark> This file holds the configuration settings for the model, such as the model architecture, hyperparameters, and other model-specific details.

<mark style="color:yellow;">`configuration_phi.py`</mark><mark style="color:yellow;">:</mark> This Python file likely contains the implementation of the model's configuration class, which is used to load and manage the model's configuration.

<mark style="color:yellow;">`generation_config.json`</mark><mark style="color:yellow;">:</mark> This file specifies the configuration settings for text generation using the model, such as the maximum sequence length, temperature, and other generation-related parameters.

<mark style="color:yellow;">`merges.txt`</mark><mark style="color:yellow;">:</mark> This file is part of the tokenizer and contains the byte pair encoding (BPE) merges used for tokenization.

<mark style="color:yellow;">`model-00001-of-00002.safetensors`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">and</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`model-00002-of-00002.safetensors`</mark><mark style="color:yellow;">:</mark> These files contain the serialized model weights, split into two parts.&#x20;

The `.safetensors` format is a <mark style="color:blue;">memory-mapped format for storing tensors safely.</mark>

<mark style="color:yellow;">`model.safetensors.index.json`</mark><mark style="color:yellow;">:</mark> This file likely contains metadata or an index for the serialized model weights stored in the <mark style="color:yellow;">`.safetensors`</mark> files.

<mark style="color:yellow;">`modeling_phi.py`</mark><mark style="color:yellow;">:</mark> This Python file contains the implementation of the model architecture and its forward pass.

<mark style="color:yellow;">`special_tokens_map.json`</mark><mark style="color:yellow;">:</mark> This file maps special token names to their corresponding token IDs in the model's vocabulary.

<mark style="color:yellow;">`tokenizer.json`</mark><mark style="color:yellow;">:</mark> This file contains the serialized tokenizer object, which is used to tokenize input text into a format suitable for the model.

<mark style="color:yellow;">`tokenizer_config.json`</mark><mark style="color:yellow;">:</mark> This file holds the configuration settings for the tokenizer, such as the vocabulary size and any special tokens.

<mark style="color:yellow;">`vocab.json`</mark><mark style="color:yellow;">:</mark> This file contains the vocabulary of the model, mapping tokens to their corresponding IDs.

These files collectively define the model architecture, weights, configuration, tokenizer, and other necessary components for using the model in downstream tasks or applications.

</details>

<details>

<summary><mark style="color:green;">config.json contents and explanation</mark></summary>

The <mark style="color:yellow;">`config.json`</mark> file contains the configuration settings for the Transformer-based language model called "Phi".

1. <mark style="color:yellow;">`"_name_or_path": "microsoft/phi-2"`</mark><mark style="color:yellow;">:</mark> Specifies the name or path of the pre-trained model.
2. <mark style="color:yellow;">`"architectures": ["PhiForCausalLM"]`</mark><mark style="color:yellow;">:</mark> Indicates the architecture class used for the model, which is `PhiForCausalLM` (Phi model for causal language modeling).
3. <mark style="color:yellow;">`"auto_map": { ... }`</mark><mark style="color:yellow;">:</mark> Defines the mapping between the auto classes <mark style="color:yellow;">(</mark><mark style="color:yellow;">`AutoConfig`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">and</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`AutoModelForCausalLM`</mark><mark style="color:yellow;">)</mark> and their corresponding implementation classes (`PhiConfig` and `PhiForCausalLM`).
4. <mark style="color:yellow;">`"attention_dropout": 0.0`</mark><mark style="color:yellow;">:</mark> Sets the dropout probability for the attention layers. A value of 0.0 means no dropout is applied.
5. <mark style="color:yellow;">`"bos_token_id": 50256`</mark> and <mark style="color:yellow;">`"eos_token_id": 50256`</mark><mark style="color:yellow;">:</mark> Specifies the token IDs for the beginning-of-sequence (BOS) and end-of-sequence (EOS) tokens.
6. <mark style="color:yellow;">`"embd_pdrop": 0.0`</mark><mark style="color:yellow;">:</mark> Sets the dropout probability for the embedding layers.
7. <mark style="color:yellow;">`"hidden_act": "gelu_new"`</mark><mark style="color:yellow;">:</mark> Specifies the activation function used in the hidden layers, which is the "gelu\_new" variant of the Gaussian Error Linear Unit (GELU) activation.
8. <mark style="color:yellow;">`"hidden_size": 2560`</mark><mark style="color:yellow;">:</mark> Defines the dimensionality of the model's hidden states.
9. <mark style="color:yellow;">`"initializer_range": 0.02`</mark><mark style="color:yellow;">:</mark> Sets the range for initializing the model's weights.
10. <mark style="color:yellow;">`"intermediate_size": 10240`</mark><mark style="color:yellow;">:</mark> Specifies the dimensionality of the intermediate (feed-forward) layers.
11. <mark style="color:yellow;">`"layer_norm_eps": 1e-05`</mark><mark style="color:yellow;">:</mark> Sets the epsilon value for layer normalization to provide numerical stability.
12. <mark style="color:yellow;">`"max_position_embeddings": 2048`</mark><mark style="color:yellow;">:</mark> Defines the maximum sequence length that the model can handle.
13. <mark style="color:yellow;">`"model_type": "phi"`</mark><mark style="color:yellow;">:</mark> Indicates the type of the model, which is "phi".
14. <mark style="color:yellow;">`"num_attention_heads": 32`</mark><mark style="color:yellow;">:</mark> Specifies the number of attention heads in each attention layer.
15. <mark style="color:yellow;">`"num_hidden_layers": 32`</mark><mark style="color:yellow;">:</mark> Defines the number of hidden layers (Transformer blocks) in the model.
16. <mark style="color:yellow;">`"num_key_value_heads": 32`</mark><mark style="color:yellow;">:</mark> Specifies the number of key-value pairs in each attention head.
17. <mark style="color:yellow;">`"partial_rotary_factor": 0.4`</mark><mark style="color:yellow;">:</mark> Defines the partial rotary factor used in rotary position embedding.
18. <mark style="color:yellow;">`"qk_layernorm": false`</mark><mark style="color:yellow;">:</mark> Indicates whether layer normalization is applied to the query and key vectors in the attention mechanism.
19. <mark style="color:yellow;">`"resid_pdrop": 0.1`</mark><mark style="color:yellow;">:</mark> Sets the dropout probability for the residual connections.
20. <mark style="color:yellow;">`"rope_scaling": null`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">and</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`"rope_theta": 10000.0`</mark><mark style="color:yellow;">:</mark> Specify the scaling and theta values for RoPE (Rotary Position Embedding).
21. <mark style="color:yellow;">`"tie_word_embeddings": false`</mark><mark style="color:yellow;">:</mark> Indicates whether the input and output word embeddings are tied (shared).
22. <mark style="color:yellow;">`"torch_dtype": "float16"`</mark><mark style="color:yellow;">:</mark> Specifies the data type used for the model's parameters (float16 for half-precision).
23. <mark style="color:yellow;">`"transformers_version": "4.37.0"`</mark><mark style="color:yellow;">:</mark> Indicates the version of the Transformers library used.
24. <mark style="color:yellow;">`"use_cache": true`</mark><mark style="color:yellow;">:</mark> Enables caching of the model's key-value pairs during inference for faster generation.
25. <mark style="color:yellow;">`"vocab_size": 51200`</mark><mark style="color:yellow;">:</mark> Defines the size of the model's vocabulary.

These configuration settings determine the architecture, hyperparameters, and behavior of the Phi model. They are used to initialise and configure the model during training and inference.

</details>

<details>

<summary><mark style="color:green;">model.safetensors.index.json</mark></summary>

The <mark style="color:yellow;">`model.safetensors.index.json`</mark> file is an index file that *<mark style="color:yellow;">**maps the names of the model's parameters to their corresponding locations within the**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`.safetensors`**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**files.**</mark>*

In this case, the model's parameters are stored in two separate `.safetensors` files:&#x20;

`model-00001-of-00002.safetensors` and `model-00002-of-00002.safetensors`.&#x20;

The index file helps the model loading process identify where each parameter is located.

Let's break it down:

1. The <mark style="color:yellow;">`metadata`</mark> field contains information about the total size of the model parameters in bytes (5,559,367,680 bytes, which is <mark style="color:yellow;">approximately 5.6 GB</mark>).
2. The <mark style="color:yellow;">`weight_map`</mark> field is a dictionary where each key represents the name of a model parameter, and the corresponding value indicates the file in which that parameter is stored.

For example, the entry <mark style="color:yellow;">`"model.embed_tokens.weight": "model-00001-of-00002.safetensors"`</mark> means that the <mark style="color:yellow;">`model.embed_tokens.weight`</mark> parameter is stored in the <mark style="color:yellow;">`model-00001-of-00002.safetensors`</mark> file.

The parameter names provide information about the model architecture:

* <mark style="color:yellow;">`model.embed_tokens.weight`</mark> represents the embedding layer weights.
* <mark style="color:yellow;">`model.layers.0.input_layernorm.bias`</mark> and <mark style="color:yellow;">`model.layers.0.input_layernorm.weight`</mark> represent the layer normalization parameters for the input of the first layer.
* <mark style="color:yellow;">`model.layers.0.self_attn.dense.bias`</mark> and <mark style="color:yellow;">`model.layers.0.self_attn.dense.weight`</mark> represent the parameters of the dense layer in the self-attention mechanism of the first layer.
* ... and so on for each layer of the model.

The <mark style="color:yellow;">`.safetensors`</mark> format is a way to store the model parameters efficiently and safely. It allows for faster loading times and helps prevent issues like model corruption.

In summary, the <mark style="color:yellow;">`model.safetensors.index.json`</mark> file acts as a map that tells the model loading process where to find each parameter within the <mark style="color:yellow;">`.safetensors`</mark> files. This enables the model to be loaded correctly and efficiently.

</details>

With an understanding of the model characteristics, we will now download it to our local directory.


# Downloading Phi 2.0

#### <mark style="color:blue;">Click on this link below to review Phi 2.0 at the Huggingface model repository:</mark>

{% embed url="<https://huggingface.co/microsoft/phi-2>" %}

The screen cut below is from the Huggingface Models Database - specifically Phi 2.0.

To download the model, <mark style="color:yellow;">left click on the vertical three dots</mark> next to the "TRAIN" button. &#x20;

You can click on the diagram below to see the three vertical dots:

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FgroSKzoDbfKuKutWhc9h%2Fimage.png?alt=media&amp;token=47f2e2a8-d167-402a-b5f8-ffb111e1f707" alt=""><figcaption></figcaption></figure>

When you <mark style="color:yellow;">left click</mark> the three vertical docks you can see the instructions to download the model to your local host.

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2F1Si3SfGCHU5Z4UYTxBRH%2Fimage.png?alt=media&amp;token=b13ab603-4335-47c0-8254-71bbb1f04db7" alt=""><figcaption><p>Instructions on how to download a Huggingface model</p></figcaption></figure>

Enter the command below at your prompt:

```bash
git clone https://huggingface.co/microsoft/phi-2
```

If you don't want to download the entire model files due to bandwidth issues,  you can set the environmental variable as per below prior to downloading the model using the git clone command,.

```bash
GIT_LFS_SKIP_SMUDGE=1
```

Setting `GIT_LFS_SKIP_SMUDGE=1` tells Git LFS to <mark style="color:yellow;">skip the automatic downloading of the actual content of large files</mark> tracked by LFS when cloning a repository.&#x20;

Instead, <mark style="color:yellow;">only the pointer files are cloned.</mark>&#x20;

The "smudge" process is what usually replaces these pointers with the actual file content during cloning. &#x20;

By skipping this, you speed up the cloning process and save bandwidth, especially useful if you don't need the large files immediately or are only interested in the repository's code or smaller files.

By running this command, you clone the <mark style="color:yellow;">`microsoft/phi-2`</mark> repository without immediately downloading the large files tracked by LFS.  If you later decide that you need these large files, you can use Git LFS commands to fetch them.

If you download all the files using the git clone command you should see the following folder and associated files in your VS Code IDE:

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FKSmRy2Uh1DQOtogk7HAZ%2Fimage.png?alt=media&amp;token=1e21c64c-8b43-4a50-af4a-e8f440569598" alt=""><figcaption><p>Phi 2.0 downloaded files</p></figcaption></figure>

With the model downloaded to your local directory, the next step is to <mark style="color:yellow;">download the dataset</mark> you wish to use for fine tuning the model.


# Available Models

Base Model + Model Type + Tokenizer Type

Below are a selection of the different models <mark style="color:yellow;">available for training on the axolotl platform</mark> for your reference:

<details>

<summary><mark style="color:purple;">Cerebras Model</mark></summary>

Cerebras has released Cerebras-GPT, a suite of seven GPT models, open-sourced to foster accessible advanced AI models. These models, ranging from 111 million to 13 billion parameters, were trained using the Chinchilla formula for optimal compute efficiency. Notably, Cerebras-GPT models boast faster training times, lower costs, and reduced energy consumption compared to other publicly available models.

**Training and Architecture**

The models were trained on the CS-2 systems of the Andromeda AI supercomputer, leveraging a simple, data-parallel weight streaming architecture. This approach allowed for rapid training, avoiding complexities of model partitioning. A new scaling law, derived from this training, predicts model performance based on the compute budget, a significant contribution to AI research.

**Accessibility and Usage**

Cerebras-GPT models are fully open-source, available on Hugging Face and GitHub under the Apache 2.0 license. This release includes all models, weights, and checkpoints, with detailed training methods and performance results documented in a paper. The Cerebras CS-2 systems are also accessible on-demand via Cerebras Model Studio.

**Large Language Models (LLMs) Landscape**

Cerebras-GPT addresses the growing trend of large, closed-source LLMs like OpenAI's GPT-4, which often lack transparency in model architecture and training details. Cerebras-GPT, conversely, is open and reproducible, aiming to make LLMs more accessible for both research and commercial applications.

**Compute-Optimal Training**

The Chinchilla recipe, which dictates using 20 data tokens per model parameter, guided the training of Cerebras-GPT models. This method contrasts with models trained on fixed data amounts regardless of size, leading to Cerebras-GPT achieving lower loss per compute unit across all model sizes.

**Model Performance and Scaling Law**

Cerebras-GPT models show state-of-the-art efficiency on common downstream language tasks. The scaling law established by Cerebras offers a predictive measure of model loss before training, optimizing the training process for various model sizes using the open Pile dataset.

**Training Infrastructure**

Training large models like Cerebras-GPT typically requires complex and technical infrastructure, especially on GPUs. However, Cerebras used standard data parallelism on its CS-2 systems, avoiding the need for intricate model parallelism. This simplicity allows researchers to focus more on model design than on distributed systems engineering.

**Conclusion**

Cerebras' release of Cerebras-GPT and its training infrastructure aims to democratize large model training and research. By providing efficient training methods, open-source models, and accessible infrastructure, Cerebras contributes to the advancement of the generative AI industry.

</details>

<details>

<summary><mark style="color:purple;">Code-Llama</mark></summary>

Code Llama, released on August 24, 2023, is a large language model (LLM) developed for coding tasks. It's built upon Llama 2 and fine-tuned specifically for generating and discussing code. The model is made freely available for both research and commercial purposes.

**Capabilities and Features**

* **Enhanced Coding Abilities**: Code Llama can generate and interpret code from both code and natural language prompts. It's proficient in code completion and debugging, supporting popular programming languages like Python, C++, Java, PHP, Typescript, C#, and Bash.
* **Model Variants**: Three sizes of Code Llama are available, with 7B, 13B, and 34B parameters. These models have been trained on 500 billion tokens of code and code-related data.
* **Fill-in-the-Middle (FIM) Capability**: The 7B and 13B models can insert code into existing snippets, aiding tasks like code completion.
* **Serving and Latency**: The models are designed to cater to varying needs. The 7B model works on a single GPU, while the 34B model offers superior results with higher latency. The smaller models are faster, suitable for real-time tasks.

**Specialised Versions**

* **Code Llama – Python**: A Python-specific variation, further fine-tuned on 100 billion tokens of Python code, catering to the significant role of Python in AI and code generation.
* **Code Llama – Instruct**: This variant is instruction fine-tuned to better comprehend and respond to natural language instructions, making it ideal for generating helpful and safe code responses.

**Use Cases and Impact**

Code Llama aims to streamline developer workflows, focusing on efficiency and reducing repetitive tasks. It's intended for a broad range of sectors, including research, industry, open-source projects, NGOs, and businesses. The release is part of an effort to foster innovation and safety in AI, with the community encouraged to explore, evaluate, and enhance the model's capabilities.

</details>

<details>

<summary><mark style="color:purple;">Falcon</mark></summary>

Falcon LLM is a suite of generative large language models (LLMs) designed to advance various applications, featuring models with different parameter sizes and a high-quality dataset.

**Falcon 40B**

* **Specifications**: Falcon 40B, with 40 billion parameters, was trained on one trillion tokens.
* **Ranking**: It was the top-ranked open-source AI model on Hugging Face’s leaderboard for two months post-launch.
* **Accessibility**: Offered royalty-free, Falcon 40B democratizes AI technology.
* **Multilingual Support**: Supports multiple languages including English, German, Spanish, French, Italian, and others.
* **Usage**: Serves as a base model for fine-tuning according to specific needs.
* **Call for Proposals**: Invited proposals for innovative use cases, with selected ones receiving investment in training compute.
* **Efficiency**: Uses less training compute compared to GPT-3, Chinchilla AI, and PaLM-62B.
* **Data Quality**: Trained on around five trillion tokens from diverse sources, with a focus on high-quality data extraction.

**Falcon 180B**

* **Specifications**: Falcon 180B is a more powerful model with 180 billion parameters, trained on 3.5 trillion tokens.
* **Performance**: Tops the Hugging Face Leaderboard and excels in reasoning, coding, proficiency, and knowledge tests.
* **Comparison**: Ranks closely behind OpenAI's GPT 4 and on par with Google's PaLM 2 Large, despite being smaller in size.
* **Availability**: Available for both research and commercial use, subject to Terms & Conditions and Acceptable Use Policy.

</details>

<details>

<summary><mark style="color:purple;">Mistral</mark></summary>

Mistral AI has released Mistral 7B, a highly efficient and powerful language model with 7.3 billion parameters. It stands out for its performance in various benchmarks, surpassing larger models in the Llama series.

**Key Features and Performance**

* **Benchmarking**: Mistral 7B outperforms the Llama 2 13B in all benchmarks and competes closely with Llama 1 34B. It approaches CodeLlama 7B's performance in coding tasks while maintaining proficiency in English tasks.
* **Efficiency**: Uses Grouped-query attention (GQA) and Sliding Window Attention (SWA), enhancing inference speed and handling longer sequences more efficiently.
* **Licensing**: Released under the Apache 2.0 license, allowing unrestricted use, including local deployment and cloud-based implementation.

**Accessibility and Deployment**

* Mistral 7B can be downloaded and used anywhere with a reference implementation.
* It is deployable on major cloud platforms like AWS, GCP, and Azure, using vLLM inference server and skypilot.
* Also available on HuggingFace.

**Fine-Tuning and Customization**

* Mistral 7B is easy to fine-tune for specific tasks. A fine-tuned model for chat, demonstrating superior performance compared to Llama 2 13B chat, is provided.

**Technical Advancements**

* **Sliding Window Attention**: Each layer attends to the previous 4,096 hidden states, resulting in a linear compute cost. This mechanism allows higher layers to access information further in the past, enhancing model performance.
* **Local Attention**: Employs fixed attention span with rotating buffers in the cache, reducing memory usage during inference.

**Benchmark Details**

* Mistral 7B was evaluated across various themes like commonsense reasoning, world knowledge, reading comprehension, math, and code.
* On reasoning, comprehension, and STEM reasoning benchmarks, Mistral 7B performs comparably to a hypothetical Llama 2 model three times its size.
* Notably, it performs well on all evaluations except knowledge benchmarks, where it matches the performance due to its parameter limit.

**Fine-tuning for Chat Applications**

* Mistral 7B Instruct, fine-tuned on public datasets from HuggingFace, shows excellent generalization capabilities. It outperforms all 7B models on MT-Bench and compares favorably to 13B chat models.

</details>

<details>

<summary><mark style="color:purple;">XJen 7b</mark></summary>

Salesforce has introduced XGen-7B, a significant addition to the realm of open-source generative AI models. This large language model (LLM) stands out for its extended context support, allowing for more comprehensive interactions.

**Key Features of XGen-7B**

* **Model Size**: XGen-7B has 7 billion parameters, indicative of its extensive capabilities.
* **Context Window**: A notable feature is its 8K context window, which enables it to handle larger prompts and generate longer responses. This context window represents the combined size of input and output text.
* **Tokenization**: It uses the OpenAI tokenizing system, similar to models like GPT-3 and GPT-4, for converting text into tokens.
* **Performance**: Salesforce claims that XGen-7B achieves results comparable to or better than other state-of-the-art models of similar size.

**Variants of XGen-7B**

Salesforce has released three variants of the XGen-7B model:

1. **XGen-7B-4K-base**: Supports a 4K context window.
2. **XGen-7B-8K-base**: Trained with additional data for an 8K context length.
3. **XGen-7B-{4K,8K}-inst**: Trained on instructional data for applications like chatbots, using reinforcement learning from human feedback (RLHF) techniques.

**Licensing and Usage**

* The first two variants, XGen-7B-4K-base and XGen-7B-8K-base, are released under the Apache 2.0 open-source license, allowing for commercial use.
* The instructional variant is available only for research purposes.

**Training and Capabilities**

* Salesforce used a mix of datasets, including RedPajama, Wikipedia, and their own Starcoder, for training XGen-7B.
* The model supports 22 languages, making it multilingual.
* XGen-7B excels in Massive Multitask Language Understanding, answering multiple-choice questions across various knowledge domains.
* It also performs well in conversations, long-form Q\&A, and summarization.

**Limitations**

Salesforce acknowledges that XGen-7B shares common limitations of LLMs, such as potential biases, toxicity, and hallucinations.

</details>


# Configuration for Training

Base Model + Model Type + Tokenizer Type

Before we begin establishing the training run for a model, we provide below the range of configuration options for the Axolotl training platform.

### <mark style="color:blue;">Below are the primary categories for the</mark> <mark style="color:yellow;">YAML Training Configuration File</mark>

| <mark style="color:green;">Category</mark> |
| :----------------------------------------: |
|             Model Configuration            |
|         Data Loading and Processing        |
|           Sequence Configuration           |
|            Adapter Configuration           |
|           Logging and Monitoring           |
|           Training Configuration           |
|              Training Options              |
|            Attention Mechanisms            |
|      Training Schedules and Evaluation     |
|         Debugging and Optimization         |
|               Special Tokens               |

The expandable below provides an example YAML configuration scripts for using LoRA parameter efficient fine tuning for Meta's Llama2 model.

<details>

<summary>Reference: <mark style="color:green;">The full training script for Llama2</mark></summary>

```yaml
base_model: NousResearch/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true

load_in_8bit: true
load_in_4bit: false
strict: false

datasets:
  - path: mhenrichsen/alpaca_2k_test
    type: alpaca
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./lora-out

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_table_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"
```

</details>

We will walk through establishing the configuration file for Phi 2.0.


# Datasets

### <mark style="color:blue;">Dataset preprocessing in Axolotl involves several steps</mark>

1. Parsing the dataset based on the specified dataset format.
2. Transforming the dataset according to the chosen prompt strategy.
3. Tokenizing the dataset using the configured model and tokenizer.
4. Shuffling and merging multiple datasets together if more than one is used.

There are two ways to perform dataset preprocessing in Axolotl:

Before starting the training process by running the command:

```bash
python -m axolotl.cli.preprocess /path/to/your.yaml --debug
```

This approach allows you to preprocess the datasets separately from the training process.

During the training process itself. In this case, the preprocessing happens automatically when you start the training.

### <mark style="color:blue;">The benefits of preprocessing datasets include</mark>

* <mark style="color:green;">**Faster training iterations:**</mark> When training interactively or performing sweeps (restarting the trainer frequently), preprocessing the datasets beforehand can save time and avoid the frustration of waiting for preprocessing to complete each time.
* <mark style="color:green;">**Caching:**</mark> Axolotl caches the tokenized/formatted datasets based on a hash of dependent training parameters. This means that if the same preprocessing configuration is used, Axolotl can intelligently retrieve the preprocessed data from the cache, saving time and resources.

The path of the cache is controlled by the <mark style="color:yellow;">**`dataset_prepared_path:`**</mark> parameter in the configuration YAML file.&#x20;

If left empty, the processed dataset will be cached in the default path <mark style="color:yellow;">**`./last_run_prepared/`**</mark> during training, but any existing cached data there will be ignored.&#x20;

By explicitly setting <mark style="color:yellow;">**`dataset_prepared_path: ./last_run_prepared`**</mark><mark style="color:yellow;">**,**</mark> the trainer will use the pre-processed data from the cache.

### <mark style="color:blue;">Edge cases to consider</mark>

* Custom prompt strategies or user-defined prompt templates:&#x20;
* If you are writing a custom prompt strategy or using a user-defined prompt template, the trainer may not be able to detect changes in the prompt templating logic automatically. In such cases, if you have <mark style="color:yellow;">**`dataset_prepared_path: ...`**</mark> set, the trainer may not pick up the changes you made and will continue using the old prompt.

Here's an example of how you can implement dataset preprocessing in Axolotl:

Define your dataset configuration in the YAML file:

```yaml
codedatasets:
  - path: /path/to/dataset1.json
    type: json
  - path: /path/to/dataset2.csv
    type: csv

dataset_prepared_path: ./preprocessed_data
```

<mark style="color:green;">Run the preprocessing command</mark>

```bash
python -m axolotl.cli.preprocess /path/to/your.yaml --debug
```

This command will preprocess the datasets specified in the YAML file and cache the preprocessed data in the `./preprocessed_data` directory.

<mark style="color:green;">Start the training process</mark>

```bash
python -m axolotl.cli.train /path/to/your.yaml
```

The trainer will automatically use the preprocessed data from the cache, saving time and resources during training.

Remember to be cautious when making changes to custom prompt strategies or user-defined prompt templates. If you have `dataset_prepared_path: ...` set, make sure to either update the cache by running the preprocessing command again or remove the `dataset_prepared_path:` parameter to force the trainer to preprocess the data with the updated prompt logic.

By understanding and properly implementing dataset preprocessing in Axolotl, you can optimize your training workflow, save time, and ensure that your datasets are properly prepared for training your machine learning models.


# Model Selection - General

### <mark style="color:blue;">Model Configuration</mark>

The first configuration block of the <mark style="color:yellow;">Axolotl YAML configuration file</mark> is 'model type'.  It comprises three main configurations.

1. <mark style="color:blue;">**base\_model**</mark> -  This is the Huggingface model that contains \*.pt, \*.safetensors, or \*.bin files
2. <mark style="color:blue;">**model\_type**</mark> - If you want to specify the type of model to load
3. <mark style="color:blue;">**tokenizer\_type**</mark> - Corresponding tokenizer for the model&#x20;

<mark style="color:green;">Model Configuration example for Phi 2.0</mark>

```yaml
base_model: microsoft/phi-2
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
```

<mark style="color:green;">Model Configuration example for Llama2</mark>

```yaml
base_model: NousResearch/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true
```

<mark style="color:green;">Model Configuration example for TinyLLama</mark>

```yaml
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
```

<mark style="color:green;">Model Configuration example for Code-LLama</mark>

```yaml
base_model: codellama/CodeLlama-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: CodeLlamaTokenizer
```

We will now move to setting up the Axolotl training configuration file for Phi 2.0.


# Phi 2.0

### <mark style="color:blue;">Phi 2.0 Model Details</mark>

<mark style="color:blue;">**Phi-2 Description**</mark><mark style="color:blue;">:</mark> It's a 2.7 billion parameter Transformer model trained with various NLP synthetic texts and filtered websites.&#x20;

It's designed to address common sense, language understanding, and logical reasoning.

<mark style="color:blue;">**Intention**</mark><mark style="color:blue;">:</mark> The model is not fine-tuned with reinforcement learning from human feedback. It is intended for research, especially in safety challenges like reducing toxicity, understanding biases, etc.

<mark style="color:blue;">**Formats**</mark><mark style="color:blue;">:</mark> Phi-2 is optimised for QA format, chat format, and code format.&#x20;

#### <mark style="color:green;">Sample Code</mark>

* <mark style="color:blue;">**Execution Modes**</mark><mark style="color:blue;">:</mark> Four modes are detailed, depending on whether you use CUDA or CPU and the floating-point precision (FP16 or FP32).
* <mark style="color:blue;">**Recommended Usage**</mark><mark style="color:blue;">:</mark> A Python example is provided to demonstrate how to load and use the model, including tokenizing inputs and generating outputs.

### <mark style="color:blue;">The generic configuration file for training Phi 2.0 from axolotl</mark>

#### Source: <mark style="color:blue;">axolotl</mark>/<mark style="color:green;">examples</mark>/<mark style="color:yellow;">phi</mark>/<mark style="color:purple;">phi2-ft.yml</mark>

```yaml
base_model: microsoft/phi-2
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: garage-bAInd/Open-Platypus
    type: alpaca

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./phi-sft-out

sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true

adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_torch
adam_beta2: 0.95
adam_epsilon: 0.00001
max_grad_norm: 1.0
lr_scheduler: cosine
learning_rate: 0.000003

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: True
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
resize_token_embeddings_to_32x: true
special_tokens:
  pad_token: "<|endoftext|>"
```

We will now continue to <mark style="color:yellow;">populate the training configuration file.</mark>


# Phi 2.0 - Model Configuration

### <mark style="color:blue;">Model Configuration</mark>

The first configuration block of the Axolotl configuration file is <mark style="color:blue;">'model type'.</mark>  It comprises three main configurations.

1. base\_model
2. model\_type
3. tokenizer\_type

```yaml
base_model: microsoft/phi-2
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
```

Below are explanations of the Huggingface Transformer classes that are used within the Axolotl training script specific to the model and tokenizer type.

<details>

<summary>Reference: <mark style="color:yellow;">AutoModelForCausalLM</mark> <mark style="color:green;">- class within the Huggingface Transformer library</mark></summary>

The <mark style="color:yellow;">**`AutoModelForCausalLM`**</mark> class is part of the Hugging Face Transformers library and is designed to provide a convenient way to instantiate and work with pre-trained models for causal language modelling tasks.

Causal language modelling, also known as autoregressive language modeling, is a type of language modelling task where the model predicts the next token in a sequence based on the previous tokens.&#x20;

In other words, given a sequence of tokens, the model learns to predict the probability distribution of the next token. This is commonly used for tasks like text generation, where the model generates text by predicting one token at a time based on the previously generated tokens.

The <mark style="color:yellow;">**`AutoModelForCausalLM`**</mark> class is a subclass of <mark style="color:yellow;">**`_BaseAutoModelClass`**</mark>, which is a <mark style="color:blue;">base class for all the auto model classes in the Transformers library</mark>.&#x20;

The purpose of the auto model classes is to provide a unified interface for loading and using pre-trained models for various tasks.

Here's how the <mark style="color:yellow;">**`AutoModelForCausalLM`**</mark> class works:

1. It has a class attribute <mark style="color:yellow;">**`_model_mapping`**</mark> that maps the model names to their corresponding classes for causal language modelling. This mapping allows the class to automatically determine the appropriate model class based on the provided model name or path.
2. When you call the <mark style="color:yellow;">**`from_pretrained()`**</mark> method of <mark style="color:yellow;">**`AutoModelForCausalLM`**</mark> and provide a pre-trained model name or path, it automatically retrieves the corresponding model class from the <mark style="color:yellow;">**`_model_mapping`**</mark> based on the model name.
3. It then initializes and returns an instance of the retrieved model class, which can be used for causal language modelling tasks.

The advantage of using <mark style="color:yellow;">**`AutoModelForCausalLM`**</mark> is that you don't need to know the specific model class for a given pre-trained model. You can simply provide the model name or path, and the class will handle the instantiation of the appropriate model class for you.

For example, if you have a pre-trained GPT-2 model and want to use it for causal language modelling, you can do the following:

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")
```

This code will <mark style="color:blue;">automatically load the GPT-2 model</mark> and return an instance of the appropriate model class for causal language modelling.

Overall, the <mark style="color:yellow;">**`AutoModelForCausalLM`**</mark> class provides a convenient and flexible way to work with pre-trained models for causal language modelling tasks, abstracting away the need to know the specific model classes and allowing you to focus on using the models for your desired task.

</details>

<details>

<summary>Reference: <mark style="color:yellow;"><strong>AutoTokenizer</strong></mark> <mark style="color:green;">- a class within the Huggingface Transformer library</mark></summary>

The <mark style="color:yellow;">**`AutoTokenizer`**</mark> class is a powerful and versatile tool in the Hugging Face Transformers library that simplifies the process of instantiating the appropriate tokenizer for a given pretrained model.

It serves as a high-level interface that automatically selects and initializes the correct tokenizer class based on the provided pretrained model name or path.

Let's dive into the key aspects of the <mark style="color:yellow;">**`AutoTokenizer`**</mark> class and understand how it enhances the usability and flexibility of tokenization in natural language processing tasks:

Automatic Tokenizer Selection

* The primary purpose of the <mark style="color:yellow;">**`AutoTokenizer`**</mark> class is to *<mark style="color:yellow;">**automatically determine the appropriate tokenizer class to use based on the pretrained model.**</mark>*
* It eliminates the need for users to manually specify the tokenizer class, saving time and reducing the chances of errors.
* The class leverages various methods to infer the tokenizer class, such as examining the model's configuration, using pattern matching on the model name or path, or utilizing a tokenizer configuration file.

<mark style="color:green;">**Pretrained Model Support**</mark>

* The `AutoTokenizer` class seamlessly integrates with pretrained models available on the Hugging Face Model Hub or locally saved models.
* It accepts a <mark style="color:yellow;">**`pretrained_model_name_or_path`**</mark> parameter, which can be a model identifier, a path to a directory containing the necessary files, or a URL to a specific file.
* This flexibility allows users to easily load tokenizers associated with a wide range of pretrained models, enabling quick experimentation and transfer learning.

<mark style="color:green;">**Tokenizer Instantiation**</mark>

* The <mark style="color:yellow;">**`from_pretrained()`**</mark> class method is the primary entry point for instantiating tokenizers using the <mark style="color:yellow;">**`AutoTokenizer`**</mark><mark style="color:yellow;">**.**</mark>
* It takes care of downloading and caching the required files, such as vocabulary files, if they are not already present locally.
* The method accepts various parameters to customize the tokenizer's behavior, such as specifying the tokenizer type, using a fast tokenizer variant, or providing additional keyword arguments.

<mark style="color:green;">**Fast Tokenizers**</mark>

* The <mark style="color:yellow;">**`AutoTokenizer`**</mark> class supports the use of fast tokenizers, which are implemented in Rust and offer improved performance compared to their Python counterparts.
* By setting the <mark style="color:yellow;">**`use_fast`**</mark> parameter to <mark style="color:yellow;">**`True`**</mark> (default), the class automatically selects the fast tokenizer variant if available for the given model.
* If a fast tokenizer is not available, it gracefully falls back to the standard Python-based tokenizer.

<mark style="color:green;">**Trust Remote Code**</mark>

* The <mark style="color:yellow;">**`trust_remote_code`**</mark> parameter allows users to control whether the <mark style="color:yellow;">**`AutoTokenizer`**</mark> should trust and execute custom tokenization code defined in the model's repository.
* This feature is useful for models that require specific tokenization logic but should be used with caution and only for trusted repositories.

<mark style="color:green;">**Tokenizer Configuration**</mark>

* The <mark style="color:yellow;">**`AutoTokenizer`**</mark> class utilizes configuration objects <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`PretrainedConfig`**</mark><mark style="color:yellow;">**)**</mark> to determine the appropriate tokenizer class to instantiate.
* It first attempts to load the tokenizer configuration from a dedicated file <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`tokenizer_config`**</mark><mark style="color:yellow;">**)**</mark> associated with the pretrained model.
* If the tokenizer configuration is not available, it falls back to using the model's configuration <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`AutoConfig`**</mark><mark style="color:yellow;">**)**</mark> to infer the tokenizer class.

<mark style="color:green;">**Tokenizer Registration**</mark>

* The <mark style="color:yellow;">**`AutoTokenizer`**</mark> class provides a <mark style="color:yellow;">**`register()`**</mark> method that allows users to register new tokenizer classes.
* This feature is particularly useful for extending the <mark style="color:yellow;">**`AutoTokenizer`**</mark> to support custom or newly developed tokenizers.
* By registering a configuration class along with the corresponding slow and fast tokenizer classes, users can seamlessly integrate their own tokenizers into the <mark style="color:yellow;">**`AutoTokenizer`**</mark> ecosystem.

In summary, the <mark style="color:yellow;">**`AutoTokenizer`**</mark> class is a powerful tool that simplifies the process of initializing tokenizers for pretrained models.&#x20;

It abstracts away the complexities of manually selecting and instantiating tokenizer classes, allowing users to focus on their natural language processing tasks.&#x20;

With its automatic tokenizer selection, support for pretrained models, fast tokenizer variants, and extensibility through registration, the <mark style="color:yellow;">**`AutoTokenizer`**</mark> class greatly enhances the usability and flexibility of tokenization in the Hugging Face Transformers library.

</details>

The next step after determining the model type configurations is to configure the <mark style="color:blue;">data loading and processing parameters</mark>


# Phi 2.0 - Model Quantization

With the model configured, we next have to determine whether whether we will be using quantization in the training process.

Given Phi 2.0 is such a small model, in this case we will not be using quantization during the training process.

### <mark style="color:blue;">Model Quantization</mark>

<pre class="language-yaml"><code class="lang-yaml"><strong>#These are the default values
</strong><strong>
</strong><strong>llm_int8_has_fp16_weight: false
</strong>bnb_4bit_quant_type: nf4  #4 bit normal float data type
bnb_4bit_use_double_quant: true

#You can override the default values as per below

load_in_8bit: true
load_in_4bit: false
strict: false
</code></pre>

### <mark style="color:blue;">load\_in\_8bit: true or false</mark>

This is configuration flag that determines <mark style="color:yellow;">whether the model should be loaded in</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">**8-bit precision.**</mark>  If it is set to "true," the model will be loaded in 8-bit precision.

#### <mark style="color:green;">**Memory Efficiency**</mark>

8-bit precision reduces the memory footprint of the model compared to higher precision formats (like the default 16-bit). This is because it requires less memory to store each weight in the model.

Loading a model in 8-bit precision can accelerate model loading and inference times. This is due to the reduced computational load compared to higher precision formats.

While 8-bit precision is more efficient, it can slightly reduce the accuracy of the model compared to full precision (32-bit). This happens because of the reduced resolution in representing the weights and activations.

<details>

<summary>Reference: <mark style="color:yellow;">BitsAndBytesConfig Class</mark> <mark style="color:green;">from the Transformers Library</mark></summary>

This class is a wrapper for configuring and managing the quantization settings when loading a model using the <mark style="color:yellow;">`bitsandbytes`</mark> library.

Quantization is a technique used to reduce the memory footprint and computational cost of deep learning models by representing weights and activations with lower-precision data types, such as int8 or 4-bit floating-point numbers.&#x20;

The `bitsandbytes` library provides methods for quantizing models, and the <mark style="color:yellow;">`BitsAndBytesConfig`</mark> class *<mark style="color:yellow;">**acts as a configuration object to control the quantization settings.**</mark>*

Let's go through the main aspects of the `BitsAndBytesConfig` class:

<mark style="color:green;">Initialization</mark>

* The class takes several arguments in its constructor to configure the quantization settings.
* The main arguments are <mark style="color:yellow;">`load_in_8bit`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">and</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`load_in_4bit`</mark>, which are mutually exclusive and determine whether to use 8-bit or 4-bit quantization.
* Other arguments include threshold values, module exclusion lists, and settings specific to the <mark style="color:yellow;">`bitsandbytes`</mark> library.

<mark style="color:green;">Properties and Setters</mark>

* The class provides properties and setters for the <mark style="color:yellow;">`load_in_4bit`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">and</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`load_in_8bit`</mark> attributes.
* The setters enforce the mutual exclusivity of these attributes and validate the input values.

<mark style="color:green;">Post-initialization</mark>

* The <mark style="color:yellow;">`post_init()`</mark> method is called after initialization to perform safety checks on the provided arguments.
* It ensures that the arguments have the correct data types and raises a <mark style="color:yellow;">`ValueError`</mark> if any inconsistencies are found.
* It also checks the version of the <mark style="color:yellow;">`bitsandbytes`</mark> library to ensure compatibility with 4-bit quantization.

<mark style="color:green;">Quantization Methods</mark>

* The `is_quantizable()` method returns `True` if the model is quantizable based on the `load_in_8bit` or `load_in_4bit` flags.
* The `quantization_method()` method returns the specific quantization method used, such as "llm\_int8", "fp4", or "nf4", based on the configuration.

<mark style="color:green;">Serialization</mark>

* The `to_dict()` method serializes the configuration instance to a Python dictionary, converting the PyTorch data types to strings for serialization.
* The `to_diff_dict()` method serializes only the attributes that differ from the default configuration, providing a more concise representation.

<mark style="color:green;">Representation</mark>

* The `__repr__()` method provides a string representation of the configuration instance, displaying the class name and the serialized dictionary.

The <mark style="color:yellow;">`BitsAndBytesConfig`</mark> class is designed to work seamlessly with the Transformers library and the `bitsandbytes` library for quantizing models. It provides a convenient way to configure and manage the quantization settings when loading a model.

Here are a few key points to note:

* The class supports both 8-bit and 4-bit quantization, controlled by the `load_in_8bit` and `load_in_4bit` flags.
* It allows specifying threshold values for outlier detection in 8-bit quantization, which can help maintain performance for large models.
* It provides options to exclude certain modules from quantization and to enable offloading of non-quantized parts to CPU.
* The class performs validation checks to ensure the consistency and compatibility of the provided arguments.

Overall, the `BitsAndBytesConfig` class is an important component in the Transformers library for enabling quantization of models using the `bitsandbytes` library. It provides a flexible and configurable interface to control the quantization settings and optimize the performance and memory usage of deep learning models.

</details>

### <mark style="color:blue;">The supporting academic paper</mark>

This <mark style="color:blue;">November 2022</mark> paper paper presents a novel quantization method for large language models (LLMs) that enables efficient inference without performance degradation. The key points of the paper are:

The authors develop a two-part quantization procedure called LLM.int8() that allows for the use of 8-bit matrix multiplication in feed-forward and attention projection layers of transformers, reducing memory requirements by half while maintaining full precision performance.

The authors demonstrate that LLM.int8() enables inference in LLMs with up to 175B parameters without any performance degradation, making such models more accessible to researchers and practitioners.

{% embed url="<https://arxiv.org/abs/2208.07339>" %}
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
{% endembed %}

### <mark style="color:blue;">load\_in\_4bit: true or false</mark>

This is configuration flag that determines whether the model should be loaded in **4-bit precision**.  If it is set to "true," the model will be loaded in 4-bit precision.

4-bit precision takes the concept of memory efficiency further, halving the memory requirements compared to 8-bit. This can be crucial for deploying large models on limited hardware.

Similar to 8-bit, 4-bit precision can lead to even faster loading and inference times due to the further reduced computational requirements.

The trade-off in accuracy might be more pronounced in 4-bit precision. The reduced bit-depth means that the model's ability to represent nuanced information in weights and activations is more limited. This might affect tasks that require high precision or are sensitive to small changes in weights.

### <mark style="color:blue;">strict: true or false</mark>

If set to false, default weights will be chosen where missing in adapters.  This is a component of 'bits and bytes' library

<details>

<summary>Reference: <mark style="color:green;">What is 'bits and bytes'?</mark></summary>

<mark style="color:green;">**Overview**</mark>

The <mark style="color:yellow;">**`bitsandbytes`**</mark> repository provides a lightweight wrapper around CUDA custom functions, primarily focusing on 8-bit optimizers, matrix multiplication <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`LLM.int8()`**</mark><mark style="color:yellow;">**),**</mark> and quantization functions.&#x20;

This tool is designed to enhance the performance and efficiency of machine learning models, particularly in the context of CUDA-enabled computing environments.

<mark style="color:green;">**Key Features**</mark>

* <mark style="color:blue;">**8-bit Optimizers**</mark><mark style="color:blue;">:</mark> Specialised for reducing memory usage and improving computational efficiency.
* <mark style="color:blue;">**Matrix Multiplication (LLM.int8())**</mark><mark style="color:blue;">:</mark> Offers optimized matrix multiplication capabilities.
* <mark style="color:blue;">**Quantization Functions**</mark><mark style="color:blue;">:</mark> Includes various methods for quantizing models, contributing to reduced model sizes and potentially faster inference times.

<mark style="color:green;">**Requirements**</mark>

* Python version 3.8 or higher.
* Linux distribution (Ubuntu, MacOS, etc.) with CUDA version greater than 10.0.
* Note: CUDA 10.0 is deprecated, and future support is focused on CUDA >= 11.0 with release 0.39.0.

<mark style="color:green;">**Installation**</mark>

* Installable via pip (<mark style="color:yellow;">**`pip install bitsandbytes`**</mark>).
* In cases where compilation from source is necessary, users are encouraged to submit a bug report and follow the provided compilation instructions.

<mark style="color:green;">**Usage Highlights**</mark>

* <mark style="color:blue;">**Int8 Inference with HuggingFace Transformers**</mark><mark style="color:blue;">:</mark> Allows models to load in 8-bit for reduced memory usage.
* <mark style="color:blue;">**8-bit Optimizer Usage**</mark><mark style="color:blue;">:</mark> Users can easily switch to 8-bit optimizers by replacing their existing optimizers with the corresponding 8-bit version from `bitsandbytes`.
* <mark style="color:blue;">**Mixed 8-bit Training and Int8 Inference**</mark><mark style="color:blue;">:</mark> The library supports both mixed 8-bit training with 16-bit main weights and full 8-bit inference.

<mark style="color:green;">**Features**</mark>

* Advanced techniques for 8-bit matrix multiplication and LLM.int8() inference.
* A range of 8-bit optimizers including Adam, AdamW, RMSProp, LARS, LAMB, and Lion.
* A stable embedding layer feature for improved stability in NLP models.
* Fast and efficient algorithms for quantile estimation.

<mark style="color:green;">**Requirements & Hardware Compatibility**</mark>

* Requires Anaconda, cudatoolkit, and PyTorch.
* Compatible with NVIDIA GPUs, specifically Turing or newer for LLM.int8(), and Kepler or newer for 8-bit optimizers and quantization.
* Supports CUDA versions from 10.2 to 12.0.
* Note: The library is currently supported only on Linux distributions.

</details>

<details>

<summary><mark style="color:green;">Summary of Tim Dettmers' Presentation on 8-Bit Methods for Efficient Deep Learning</mark></summary>

His main thesis is that computationally efficient methods will accelerate progress in understanding deep learning.

<mark style="color:green;">**Key Points from Tim's Presentation**</mark>

**8-Bit Methods for Large Models**: Tim highlights the importance of making large models more accessible through quantization, which <mark style="color:yellow;">reduces the memory footprint.</mark>

**Quantization Explained**: He explains quantization as a process of <mark style="color:yellow;">converting floating-point or real representations into discrete buckets</mark>, akin to histogram binning.

**Linear vs. Nonlinear Quantization**: <mark style="color:yellow;">Linear (integer) quantization involves equally wide bins, while nonlinear quantization allows varying bin widths</mark>.

**Error Reduction in Quantization**: Tim illustrates how the <mark style="color:yellow;">choice of bins impacts precision and error distribution in quantized values.</mark>

**4-Bit Inference**: His recent work shows that <mark style="color:yellow;">4-bit inference is highly effective for large transformers</mark>.

**Floating Point Data Types**: The presentation delves into the structure of floating point data types, explaining the roles of exponent bits and fraction bits.

**Dynamic Exponent Data Type**: Tim introduces a <mark style="color:yellow;">unique data type he developed with a dynamic exponent</mark>, which offers flexibility in approximating large and small values with varying precision.

**8-Bit Optimizers**: The focus shifts to 8-bit optimizers, crucial for memory efficiency in training large models, particularly in language modeling.

Tim discusses <mark style="color:yellow;">reducing memory usage by approximately 40% by converting 32-bit Adam optimizer buffers to 8-bit.</mark>

This reduction is significant as it helps make large models more memory-efficient.

<mark style="color:yellow;">Outliers in Adam optimizer buffers cause issues in quantization</mark>, leading to increased error and ineffective 8-bit quantization.

Tim presents an example showing how <mark style="color:yellow;">outliers can skew the data</mark>, leading to a waste of bits and loss of effective representation.

To address the problem of outliers, Tim <mark style="color:yellow;">proposes chunking Adam states into blocks and quantizing each block independently</mark>.

This method isolates the impact of outliers to specific blocks, enhancing the stability of 8-bit optimizers.

The process involves <mark style="color:yellow;">chunking state into blocks</mark>, finding the maximum value for normalization, and storing the index for 8-bit representation.

This method ensures compact yet effective optimization, <mark style="color:yellow;">comparable to 32-bit optimizers</mark>.

This achievement indicates significant memory savings without compromising performance.

8-bit optimizers are efficient in mapping onto hardware, with the <mark style="color:yellow;">main overhead being the dequantization process.</mark>

<mark style="color:yellow;">Outliers become a significant problem in models larger than 6.7 billion parameters</mark>, causing performance drops.

Tim's research identifies <mark style="color:yellow;">systematic outliers that emerge with scale</mark> and become problematic at specific model sizes.

Outliers in large models exhibit systematic and emergent properties, affecting the same dimensions across layers.

These outliers impact all layers in a transformer model once a certain scale is reached.

The emergence of outliers follows an exponential trend, leading to a phase shift-like effect at a certain scale.

Understanding and addressing this exponential trend is key to managing outliers in large models.

A novel approach was developed to identify and process these outliers in 16-bit while handling the rest in 8-bit, effectively maintaining efficiency while addressing the problem.

<mark style="color:green;">**Efficiency of 8-Bit Matrix Multiplication**</mark>

* By applying this method, 99.9% of weights are computed in 8-bit, with a small portion in 16-bit for outliers. This approach achieves performance equivalent to 16-bit computations while halving memory size.
* This makes large models like Llama 65B accessible on consumer hardware, significantly lowering the barrier to entry for working with such models.

<mark style="color:green;">**Few-Shot and Zero-Shot Performance**</mark>

* The few-shot performance of models using 8-bit methods is comparable to 16-bit models. Tim highlighted a strong correlation between zero-shot performance and perplexity in language models, indicating that complexity evaluations can reliably predict zero-shot performance.

<mark style="color:green;">**Understanding Outliers in Transformer Models**</mark>

* Outliers tend to be concentrated in specific columns of the input batch and are more prevalent in larger models. These outliers are crucial for attention mechanisms in transformers.
* They are context-independent, aiding the attention mechanism in focusing on specific values by providing predictable patterns for the model to cancel out unnecessary information.

<mark style="color:green;">**Trade-Offs in Activation Functions**</mark>

* Replacing traditional activation functions like softmax with more stable alternatives can increase stability but may lead to a drop in performance.
* This presents a research challenge in balancing stability with maintaining or enhancing model performance.

<mark style="color:green;">**Impact of Precision and Parameter Count on Model Efficiency**</mark>

* An interesting finding is that models with the same number of bits but different distributions of precision and parameter count (e.g., 8-bit with more parameters vs. 4-bit with fewer parameters) exhibit the same inference latency.
* This equivalence in performance is due to the nature of GPU computations, where memory loading is significantly more costly than computation. Therefore, the memory used during inference, rather than the computational complexity, often dictates the performance.

</details>


# Phi 2.0 - Data Loading and Paths

With the model defined and fully configured, we  will now configure the datasets section.

First off, this configuration file is all about <mark style="color:yellow;">specifying the datasets you want to use for fine-tuning your model and where you would like the fine tuned model stored once training is complete.</mark>

You can provide one or more datasets under the <mark style="color:yellow;">**`datasets`**</mark> section. &#x20;

Each dataset can come from different sources, like a HuggingFace dataset repo, an S3 or GS path, or even a local JSON file.

For each dataset, you can choose the <mark style="color:yellow;">type of prompt</mark> you want to use during training.&#x20;

Axolotl supports a bunch of prompt types, like "alpaca", "sharegpt", "gpteacher", "oasst", and "reflection". You can specify the prompt type using the `type` parameter.

If you're using a file-based dataset, you can specify the data type using the <mark style="color:yellow;">**`ds_type`**</mark> parameter. It supports formats like JSON, Arrow, Parquet, text, and CSV. You can also provide the path to the source data files using the <mark style="color:yellow;">**`data_files`**</mark> parameter.

### <mark style="color:blue;">Basic datasets configuration</mark>

<pre class="language-yaml"><code class="lang-yaml"><strong>datasets:
</strong>  - path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
    type: alpaca
    ds_type: json
    data_files:
  - alpaca_data_cleaned.json
dataset_prepared_path:
val_set_size: 0.20

output_dir: ./phi-out
</code></pre>

This component of the configuration file tells the platform where the dataset it located, and how the dataset should be split up into <mark style="color:yellow;">training versus validation.</mark>

We will only be using a number of these parameters for our fine tuning.  A full analysis of configurations is analysed below, and a summary can be found here:  [<mark style="color:orange;">**data loading and processing configurations**</mark>](/axolotl-configuration-files/data-loading-and-processing)<mark style="color:orange;">**.**</mark>

***

### <mark style="color:green;">Full assessment of all of the datasets configuration options</mark>

### <mark style="color:purple;">`path:`</mark>

#### This parameter indicates the <mark style="color:yellow;">path to a dataset that has been prepared for training</mark>.&#x20;

As highlighted, this can be a HuggingFace dataset repository, an AWS S3 storage path (s3://), or Google Cloud Path (gs\://) or "json" for a local dataset.

Here's an example of a Google Cloud Storage path:

```yaml
gs://my-bucket/datasets/my_dataset.json
```

In this example, "my-bucket" is the name of the Google Cloud Storage bucket, and "datasets/my\_dataset.json" is the path to the specific file within that bucket.

To determine the source of your data files, right click on the folder containing the data files in VS Code and then select 'copy URL'.&#x20;

```yaml
path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
```

### <mark style="color:purple;">`type:`</mark>

Indicates the type of prompt to use for training, such as <mark style="color:yellow;">"alpaca", "sharegpt", "gpteacher", "oasst", or "reflection".</mark>

We will be using <mark style="color:yellow;">Alpaca in this example</mark>.  For a full explanation of each data 'type' please refer to this documentation: data type

```yaml
type: alpaca
```

### <mark style="color:purple;">`ds_type:`</mark>

(Optional) Specifies the data type when the <mark style="color:yellow;">`path`</mark> is a file, such as "json", "arrow", "parquet", "text", or "csv".

Huggingface datasets can come in a number of formats.  You can configure the data loading process to take into account the data type - csv, json, parquet.   <mark style="color:yellow;">We will be using JSON format.</mark>

```yaml
ds_type: json
```

### <mark style="color:purple;">`data_files:`</mark>

(Optional) Specifies the path to the source data files.

### <mark style="color:purple;">`dataset_prepared_path`</mark><mark style="color:purple;">:</mark>

Specifies the relative path where Axolotl attempts to save the prepared dataset as an arrow file after packing the data together. This allows subsequent training attempts to load faster.

```yaml
path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
```

### <mark style="color:purple;">`output_dir:`</mark>&#x20;

This is the path where the full fine tuned model will be saved to:

```bash
output_dir: ./completed-model
```

### <mark style="color:blue;">Advanced: Explanation of all other data related configuration options</mark>

### <mark style="color:purple;">`shards`</mark><mark style="color:purple;">:</mark>&#x20;

Axolotl allows you to split your data into shards, which can be handy for large datasets. You can specify the number of shards using the <mark style="color:yellow;">**`shards`**</mark> parameter.

### <mark style="color:purple;">`name`</mark><mark style="color:purple;">:</mark>&#x20;

If you're using a dataset from a repository, you can give it a name using the <mark style="color:yellow;">**`name`**</mark> parameter. This can help you keep track of which dataset configuration you're using.

### <mark style="color:purple;">`train_on_split`</mark><mark style="color:purple;">:</mark>&#x20;

By default, Axolotl uses the "train" split for training, but you can specify a different split using the `train_on_split` parameter.

### <mark style="color:purple;">`conversation`</mark><mark style="color:purple;">:</mark>&#x20;

If you're using the "sharegpt" prompt type, you can specify the conversation type using the <mark style="color:yellow;">**`conversation`**</mark> parameter. Axolotl uses the FastChat library for conversations, and you can find the available conversation types in the FastChat documentation.

{% embed url="<https://github.com/lm-sys/FastChat/tree/main>" %}

### <mark style="color:purple;">`field_human`</mark> <mark style="color:purple;"></mark><mark style="color:purple;">and</mark> <mark style="color:purple;"></mark><mark style="color:purple;">`field_model`</mark><mark style="color:purple;">: (optional)</mark>

You can also specify the keys to use for the human and assistant roles in the conversation using the <mark style="color:yellow;">`field_human`</mark> and `f`<mark style="color:yellow;">`ield_model`</mark> parameters.

### <mark style="color:purple;">`roles`</mark><mark style="color:purple;">:</mark>&#x20;

If your dataset has additional keys that you want to use as input or output roles, you can specify them under the `roles` section. The `input` role is used for masking, and the `output` role is used for generation.

### <mark style="color:purple;">`input`</mark><mark style="color:purple;">:</mark>

&#x20;(Optional) A list of keys to be masked based on <mark style="color:yellow;">**`train_on_input`**</mark><mark style="color:yellow;">**.**</mark>

### <mark style="color:purple;">`output`</mark><mark style="color:purple;">:</mark>&#x20;

(Optional) A list of keys to be used as output.

### <mark style="color:blue;">Custom user instruction prompt</mark>

Allows defining a custom prompt for each dataset.

### <mark style="color:purple;">`system_prompt`</mark><mark style="color:purple;">:</mark>&#x20;

(Optional) Specifies the system prompt.

### <mark style="color:purple;">`system_format`</mark><mark style="color:blue;">:</mark>

&#x20;(Optional) Specifies the format of the system prompt, with "{system}" as a placeholder.

### <mark style="color:purple;">`field_system`</mark><mark style="color:purple;">,</mark> <mark style="color:purple;"></mark><mark style="color:purple;">`field_instruction`</mark><mark style="color:purple;">,</mark> <mark style="color:purple;"></mark><mark style="color:purple;">`field_input`</mark><mark style="color:purple;">,</mark> <mark style="color:purple;"></mark><mark style="color:purple;">`field_output`</mark><mark style="color:purple;">:</mark>&#x20;

(Optional) Specify the column names for the respective fields in the dataset.

### <mark style="color:purple;">`format`</mark><mark style="color:purple;">:</mark>&#x20;

Specifies the format of the prompt, which can include placeholders for instruction and input.

### <mark style="color:purple;">`no_input_format`</mark><mark style="color:blue;">:</mark>

&#x20;Specifies the format of the prompt when there is no input, excluding the "{input}" placeholder.

1. <mark style="color:blue;">**`field`**</mark><mark style="color:blue;">**:**</mark> (Optional) For "completion" datasets only, specifies the field to use instead of the "text" column.
2. <mark style="color:blue;">**`shuffle_merged_datasets`**</mark><mark style="color:blue;">**:**</mark> (Optional) By default, Axolotl shuffles the merged datasets, but you can disable this behavior by setting <mark style="color:yellow;">**`shuffle_merged_datasets`**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**to**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`false`**</mark>.

### <mark style="color:blue;">Datasets for Evaluation</mark>

### <mark style="color:purple;">`test_datasets`</mark>

A list of one or more datasets used for evaluating the model.

* <mark style="color:blue;">**`path`**</mark><mark style="color:blue;">**:**</mark> Specifies the path to the test dataset file.
* <mark style="color:blue;">**`ds_type`**</mark><mark style="color:blue;">**:**</mark> Specifies the data type of the test dataset, such as "json".
* <mark style="color:blue;">**`split`**</mark><mark style="color:blue;">**:**</mark> Specifies the split to use for the test dataset. For "json" datasets, the default split is called "train".
* <mark style="color:blue;">**`type`**</mark><mark style="color:blue;">**:**</mark> Specifies the type of the test dataset, such as "completion".
* <mark style="color:blue;">**`data_files`**</mark><mark style="color:blue;">**:**</mark> Specifies the list of data files for the test dataset.

Note: You can use either <mark style="color:blue;">**`test_datasets`**</mark> or <mark style="color:blue;">**`val_set_size`**</mark>, but not both.

### <mark style="color:purple;">`dataset_prepared_path`</mark><mark style="color:purple;">:</mark>

Specifies the relative path where Axolotl attempts to save the prepared dataset as an arrow file after packing the data together. This allows subsequent training attempts to load faster.

### <mark style="color:purple;">`push_dataset_to_hub`</mark>

If you want to push the prepared dataset to the Hugging Face Hub, you can specify the repository path using the <mark style="color:yellow;">**`push_dataset_to_hub`**</mark> parameter.

### <mark style="color:purple;">`hub_model_id`</mark>

You can also push the fine-tuned model checkpoints to the Hugging Face Hub by specifying the private repository path using the <mark style="color:yellow;">**`hub_model_id`**</mark> parameter.

### <mark style="color:purple;">`hub_strategy`</mark>

The <mark style="color:yellow;">**`hub_strategy`**</mark> parameter allows you to control how the checkpoints are pushed to the hub.   Refer to the Hugging Face Trainer documentation for more information.


# Phi 2.0 - Sequence Configuration

Before training or fine-tuning begins, the <mark style="color:yellow;">input data must be correctly formatted and prepared</mark>.

We will be configuring the:

1. Sequence Length
2. Sample Packing
3. Padding to Sequence

```yaml
sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true
```

### <mark style="color:blue;">sequence\_len</mark>

This parameter sets the <mark style="color:yellow;">maximum allowable length for input sequences</mark>. Sequences longer than this length may be truncated or split during training.

&#x20;This limit is essential since transformers process input data in fixed-size blocks.

Sequences longer than this length are either truncated or split.&#x20;

Truncation means cutting off the part of the sequence that exceeds the limit, while splitting involves dividing a long sequence into smaller segments, each within the maximum length.

The choice of sequence length affects memory usage and computational requirements. Longer sequences can capture more context but require more computational resources.

Axolotl recommend the maximum length should typically be less than 2048 as most models have a token/context limit of 2048

### <mark style="color:blue;">sample\_packing</mark>

A flag that determines whether sample packing should be used.&#x20;

This is a method to optimise the training process by packing multiple shorter sequences into a single training example (batch).  It can increases training efficiency by reducing padding needs and better utilizing GPU memory. This technique is particularly useful when dealing with variable-length sequences.

**Implementation**: If set to <mark style="color:yellow;">**`true`**</mark>, sequences that are shorter than <mark style="color:yellow;">**`sequence_len`**</mark> are concatenated with others to form a packed batch. This process continues until the maximum sequence length is reached or no more sequences are available for packing.

### <mark style="color:blue;">pad\_to\_sequence\_len</mark>

This is a flag that controls whether sequences should be padded to match the specified sequence length.

This ensures that <mark style="color:yellow;">**all sequences in a batch are of the same length**</mark>, which is necessary for parallel processing by the model.   Shorter sequences are extended (padded) with special tokens (usually <mark style="color:yellow;">**`[PAD]`**</mark>) to reach the defined maximum sequence length.

Padding is a standard practice in training neural networks on sequences of varying lengths, but it can introduce additional computational overhead, especially with longer sequence lengths.

If set to "true," input sequences will be padded with special tokens to reach the maximum sequence length defined earlier.


# Phi 2.0 - Lora Configuration

Given the small size of the Phi 2.0 model, we will not be fine tuning using Lora.

We will stick to the generic template provided by Axolotl, which as you can see is not configured.

For your reference. Axolotl point towards this explanation from Anyscale on fine tuning using Lora:

{% embed url="<https://www.anyscale.com/blog/fine-tuning-llms-lora-or-full-parameter-an-in-depth-analysis-with-llama-2>" %}
Fine-Tuning LLMs: LoRA or Full-Parameter? An in-depth Analysis with Llama 2
{% endembed %}

```yaml
adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:
```


# Phi 2.0 - Logging

We will be using Weights and Biases to log and monitor our fine tuning runs.

If you are new to Weights and Biases, please take the time to review their quickstart guide:

{% embed url="<https://docs.wandb.ai/quickstart>" %}
Weights and Biases Quickstart
{% endembed %}

&#x20;A [run](https://docs.wandb.ai/guides/runs) is the basic building block of W\&B. You will use them often to [track metrics](https://docs.wandb.ai/guides/track), [create logs](https://docs.wandb.ai/guides/artifacts), [create jobs](https://docs.wandb.ai/guides/launch).

Once you have created a Weights and Biases account, login to  your Weights and Biases account by entering the following command at the command prompt:

```bash
wandb login
```

You will be asked to enter your Weights and Biases account details as per below:

<mark style="color:blue;">Username:</mark>&#x20;

<mark style="color:blue;">Password:</mark> &#x20;

<mark style="color:blue;">API Token:</mark>&#x20;

### <mark style="color:blue;">Configuration of Logging Component</mark>

<mark style="color:green;">Project</mark>

The project naming convention can be simplistic but informational.   Name your project whatever you like.

<mark style="color:green;">Entity</mark>

The <mark style="color:yellow;">entity</mark> is simply \<your-organisation>.

The configuration file should look like the below script:

```yaml
wandb_project: <your project name>
wandb_entity: <your-organisation> 
wandb_name: Set the name of your wandb run
wandb_run_id: Set the ID of your wandb run
```

<details>

<summary>Reference: <mark style="color:green;">The Weights and Biases Script in the Axolotl Library</mark></summary>

<mark style="color:yellow;">src/axolotl/utils/wandb\_.py</mark>

The script defines a function `setup_wandb_env_vars` that configures environment variables for wandb based on a given configuration. Here's a detailed explanation of what this function does:

<mark style="color:green;">**Module Description**</mark>

* The comment `"""Module for wandb utilities"""` indicates that this Python file is intended to provide utility functions for working with wandb.

<mark style="color:green;">**Import Statements**</mark>

* The script imports the `os` module, which provides functions for interacting with the operating system, including managing environment variables.
* It imports `DictDefault` from `axolotl.utils.dict`, which is a custom dictionary utility, likely providing some default behavior for dictionary operations.

<mark style="color:green;">**Function Definition**</mark>

`setup_wandb_env_vars(cfg: DictDefault)`:

* The function takes a single parameter `cfg`, which is expected to be an instance of `DictDefault`. This parameter likely contains configuration settings.

<mark style="color:green;">**Setting Environment Variables for wandb**</mark>

* The function iterates through all keys in the `cfg` dictionary.
* If a key starts with `"wandb_"`, the function retrieves its value.
* If the value is a non-empty string, the function sets an environment variable with the name of the key in uppercase and assigns it the value from `cfg`. For example, if `cfg` contains `{"wandb_api_key": "your_key"}`, it sets an environment variable `WANDB_API_KEY` with the value `"your_key"`.

<mark style="color:green;">**Enabling wandb Integration**</mark>

* The function checks if `cfg.wandb_project` exists and is a non-empty string.
* If `cfg.wandb_project` is set, `cfg.use_wandb` is set to `True`, and any existing `WANDB_DISABLED` environment variable is removed. This implies that wandb should be enabled and operational for the current session.
* If `cfg.wandb_project` is not set, the function sets the environment variable `WANDB_DISABLED` to `"true"`, effectively disabling wandb integration.

<mark style="color:green;">**Usage Context**</mark>

* This script is useful for dynamically configuring wandb based on a set of configuration parameters, especially in scenarios where different wandb settings are needed for different runs or experiments.
* It automates the process of setting up wandb environment variables, enabling or disabling wandb tracking based on the provided configuration.

In summary, the script is a utility for configuring wandb environment variables in a Python environment based on a provided configuration, enabling or disabling wandb tracking as needed. This is particularly useful in machine learning workflows where experiment tracking needs to be managed programmatically.

</details>


# Phi 2.0 - Training Configuration

This is the default training configuration.  We will leave it as is for the time being.

```yaml
gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_torch
adam_beta2: 0.95
adam_epsilon: 0.00001
max_grad_norm: 1.0
lr_scheduler: cosine
learning_rate: 0.000003
```

### <mark style="color:blue;">EExplanation of each configurations</mark>

<mark style="color:yellow;">**`gradient_accumulation_steps: 1`**</mark>

* Gradient accumulation is a technique used to simulate larger batch sizes without increasing memory usage.
* It allows the model to accumulate gradients over multiple forward passes before performing a single backward pass and updating the weights.
* Setting <mark style="color:yellow;">**`gradient_accumulation_steps`**</mark> to 1 means that the model will update its weights after every batch, effectively not using gradient accumulation.
* This is suitable when the available memory is sufficient to process the desired batch size without the need for accumulation.
* However, if memory is limited and you want to simulate a larger batch size, increasing this value can be beneficial.

<mark style="color:yellow;">**`micro_batch_size: 2`**</mark>

* The micro-batch size determines the number of samples processed in each forward pass of the model.
* In this case, setting <mark style="color:yellow;">**`micro_batch_size`**</mark> to 2 means that the model will process 2 samples at a time.
* Smaller micro-batch sizes can be useful when dealing with limited memory resources, as they require less memory per forward pass.
* However, smaller micro-batch sizes may result in slower training times and potentially noisier gradients.
* The optimal micro-batch size depends on the available hardware and the specific requirements of the training task.

<mark style="color:yellow;">**`num_epochs: 4`**</mark>

* The number of epochs represents the number of times the model will iterate over the entire training dataset.
* Setting <mark style="color:yellow;">**`num_epochs`**</mark> to 4 means that the model will go through the training data 4 times.
* Increasing the number of epochs can allow the model to learn more from the data and potentially improve its performance.
* However, training for too many epochs may lead to overfitting, where the model becomes too specialized to the training data and fails to generalize well to unseen data.
* The optimal number of epochs depends on factors such as the size and complexity of the dataset, the model architecture, and the learning rate.

<mark style="color:yellow;">**`optimizer: adamw_torch`**</mark>

* The optimizer is responsible for updating the model's weights based on the computed gradients.
* AdamW (Adam with Weight Decay) is an optimization algorithm that extends the original Adam optimizer by adding L2 regularization.
* AdamW is well-suited for training deep learning models and has been shown to perform well in various tasks.
* It adapts the learning rate for each parameter based on the historical gradients and incorporates weight decay to prevent overfitting.
* Using AdamW as the optimizer is a reasonable choice for training LLMs.

<mark style="color:yellow;">**`adam_beta2: 0.95`**</mark>

* <mark style="color:yellow;">**`adam_beta2`**</mark> is a hyperparameter of the AdamW optimizer that controls the exponential decay rate for the second moment estimates.
* It is typically set to a value close to 1 (e.g., 0.999) to give more importance to recent gradients.
* Setting <mark style="color:yellow;">**`adam_beta2`**</mark> to 0.95 means that the optimizer will give slightly less emphasis to recent gradients compared to the default value.
* The impact of this change may vary depending on the specific task and dataset.

<mark style="color:yellow;">**`adam_epsilon: 0.00001`**</mark>

* <mark style="color:yellow;">**`adam_epsilon`**</mark> is a small constant value used in the denominator of the AdamW optimizer's update rule to prevent division by zero.
* It helps to stabilize the optimization process and avoid numerical instability.
* The default value of <mark style="color:yellow;">**`adam_epsilon`**</mark> is usually 1e-8 or 1e-7.
* Setting it to 0.00001 (1e-5) is slightly larger than the typical default value, which may have a minor impact on the optimization process.

<mark style="color:yellow;">**`max_grad_norm: 1.0`**</mark>

* Gradient clipping is a technique used to prevent exploding gradients in deep neural networks.
* <mark style="color:yellow;">**`max_grad_norm`**</mark> sets the maximum L2 norm of the gradients. If the norm exceeds this value, the gradients are scaled down to meet the constraint.
* Setting `max_grad_norm` to 1.0 means that the gradients will be clipped if their L2 norm exceeds 1.0.
* Gradient clipping helps to stabilize the training process and prevents the gradients from becoming too large, which can lead to unstable or divergent behavior.
* The optimal value of <mark style="color:yellow;">**`max_grad_norm`**</mark> may vary depending on the model architecture and the specific training task.

<mark style="color:yellow;">**`lr_scheduler: cosine`**</mark>

* The learning rate scheduler determines how the learning rate changes over the course of training.
* The cosine scheduler, also known as cosine annealing, gradually decreases the learning rate following a cosine curve.
* It starts with a relatively high learning rate and slowly decreases it until it reaches a minimum value at the end of training.
* Cosine annealing can help the model converge more smoothly and potentially reach better local optima.
* It is a popular choice for learning rate scheduling and has been shown to be effective in various deep learning tasks.

<mark style="color:yellow;">**`learning_rate: 0.000003`**</mark>

* The learning rate determines the step size at which the model's weights are updated during training.
* Setting <mark style="color:yellow;">**`learning_rate`**</mark> to 0.000003 (3e-6) is a relatively small value.
* Smaller learning rates generally result in slower convergence but can lead to more stable and precise updates.
* The optimal learning rate depends on factors such as the model architecture, dataset complexity, and optimizer choice.
* It may be necessary to experiment with different learning rates to find the best value for a specific task.

The combination of AdamW optimizer with a cosine learning rate scheduler and gradient clipping can help stabilize the training process and potentially improve convergence.&#x20;

The micro-batch size and number of epochs should be adjusted based on the available hardware resources and the specific requirements of the training task. It's important to monitor the model's performance during training and make adjustments as needed to achieve the desired results.


# Phi 2.0 - Data and Precision

These configurations allow you to control various aspects of the training process, such as data handling, precision settings, and hardware utilisation.&#x20;

```yaml
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true
```

Let's go through each configuration and explain its purpose and implications:

#### <mark style="color:blue;">`train_on_inputs: false`</mark>

* This configuration determines whether to include or mask out the human's prompt from the training labels.
* When set to `false`, the model will not train on the human's prompt, meaning that the prompt will be excluded from the training labels.
* In other words, the model will only learn from the desired output or response and not from the input prompt.
* This is useful when you want the model to generate responses based on the given prompts without explicitly learning to reproduce the prompts themselves.
* By masking out the human's prompt, the model can focus on learning the mapping between the prompt and the desired output.

#### <mark style="color:blue;">`group_by_length: false`</mark>

* This configuration controls whether to group similarly sized data together to minimise padding during training.
* When set to `false`, the data will not be grouped by length and will be processed in the order it appears in the dataset.
* Grouping data by length can be beneficial when working with variable-length sequences, as it helps to reduce the amount of padding needed.
* Padding is the process of adding dummy tokens to shorter sequences to match the length of the longest sequence in a batch.
* By grouping similarly sized data together, you can minimize the amount of unnecessary padding, which can lead to more efficient memory usage and faster training.
* However, enabling `group_by_length` may result in slower data loading and preprocessing, as it requires downloading and sorting the entire dataset before training.
* It's also worth noting that when `group_by_length` is enabled, the training loss may exhibit an oscillating pattern due to the reordering of the data.

#### <mark style="color:blue;">`bf16: auto`</mark>

* This configuration relates to the use of BFloat16 (BF16) precision during training.
* BFloat16 is a 16-bit floating-point format that offers a wider dynamic range compared to the more common FP16 (Half-precision) format.
* When set to `auto`, the framework will automatically determine whether to use BF16 based on the available hardware and software support.
* If the hardware (e.g., GPU) and software (e.g., PyTorch version) support BF16, it will be used for training.
* BF16 can provide a good balance between computational efficiency and numeric precision, potentially leading to faster training times while maintaining model accuracy.
* However, the actual performance gains may vary depending on the specific hardware and model architecture.

#### <mark style="color:blue;">`fp16:`</mark>

* This configuration is related to the use of FP16 (Half-precision) during training, but in the provided configuration, it is left empty.
* FP16 is a 16-bit floating-point format that offers reduced precision compared to the standard FP32 (Single-precision) format.
* Using FP16 can help to reduce memory usage and accelerate training on certain hardware (e.g., NVIDIA GPUs with Tensor Cores).
* However, the empty value suggests that FP16 is not being explicitly enabled or configured in this case.

#### <mark style="color:blue;">`tf32: true`</mark>

* This configuration is specific to NVIDIA GPUs and relates to the use of TensorFloat-32 (TF32) precision.
* TF32 is a 19-bit floating-point format that is used by default on NVIDIA Ampere architecture GPUs (e.g., NVIDIA A100) for certain operations, such as matrix multiplications and convolutions.
* When set to `true`, TF32 will be used for supported operations on compatible hardware.
* TF32 offers a balance between performance and precision, providing faster computation compared to FP32 while maintaining similar accuracy.
* Enabling TF32 can lead to improved training speeds on NVIDIA Ampere GPUs without significant impact on model quality.


# Phi 2.0 - Optimisations

These configuration options offer various techniques and optimizations to improve the training process&#x20;

Gradient checkpointing and Flash Attention focus on memory efficiency and computational speed, while early stopping and resuming from checkpoints are useful for preventing overfitting and managing the training workflow.&#x20;

Distributed training with `local_rank` enables parallel processing across multiple devices, and adjusting the logging frequency with `logging_steps` helps in monitoring the training progress.

We will use the generic configurations provided by Axolotl.

```yaml
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: True
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
```

### <mark style="color:blue;">Below is an analysis of each of the configurations</mark>

### <mark style="color:purple;">**`gradient_checkpointing: true`**</mark>

* Gradient checkpointing is a technique used to reduce the memory usage during training by trading off computation time.
* Instead of storing all the intermediate activations for the backward pass, gradient checkpointing selectively stores a subset of activations and recomputes the others when needed.
* Setting `gradient_checkpointing` to `true` enables this feature, which can be particularly beneficial when training large models with limited memory.
* By enabling gradient checkpointing, you can potentially train larger models or use larger batch sizes without running out of memory.
* However, it's important to note that gradient checkpointing introduces additional computational overhead, as the model needs to recompute the activations during the backward pass.
* The decision to use gradient checkpointing depends on the available memory and the trade-off between memory usage and training speed.

### <mark style="color:purple;">**`gradient_checkpointing_kwargs:`**</mark>

* This configuration allows you to specify additional keyword arguments for gradient checkpointing.
* In the provided example, `use_reentrant: True` is specified as a keyword argument.
* The `use_reentrant` flag is related to the implementation of gradient checkpointing in PyTorch.
* When set to `True`, it enables the use of reentrant autograd functions, which can provide additional memory savings during gradient checkpointing.
* However, the specific behavior and impact of this flag may depend on the PyTorch version and the model architecture.

### <mark style="color:purple;">`early_stopping_patience:`</mark>

* Early stopping is a technique used to prevent overfitting and improve generalization performance.
* It monitors a validation metric (e.g., validation loss or accuracy) during training and stops the training process if the metric does not improve for a specified number of iterations (patience).
* The `early_stopping_patience` configuration allows you to set the number of iterations to wait before early stopping is triggered.
* For example, if `early_stopping_patience` is set to 3, training will stop if the validation metric does not improve for 3 consecutive iterations.
* Early stopping helps to avoid wasting computational resources on training iterations that do not lead to further improvements and can help prevent the model from overfitting to the training data.

### <mark style="color:purple;">`resume_from_checkpoint:`</mark>

* This configuration allows you to resume training from a specific checkpoint.
* By specifying a checkpoint directory or file path, you can load the model state, optimizer state, and other necessary information to continue training from where it left off.
* Resuming from a checkpoint can be useful in various scenarios, such as when training is interrupted due to system failures, when you want to fine-tune a pre-trained model, or when you want to experiment with different hyperparameters starting from a previously trained model.
* It saves time and resources by avoiding the need to start training from scratch.

### <mark style="color:purple;">`local_rank:`</mark>

* The `local_rank` configuration is related to distributed training, specifically when using techniques like Data Parallel or Distributed Data Parallel.
* In distributed training, multiple GPUs or machines are used to parallelise the training process and speed up computations.
* The `local_rank` represents the unique identifier of a process within a distributed training setup.
* It is typically used to determine the device placement and communication patterns among the processes.
* When using distributed training frameworks like PyTorch's `DistributedDataParallel`, the `local_rank` is automatically set by the framework.

### <mark style="color:purple;">`logging_steps: 1`</mark>

* The `logging_steps` configuration determines the frequency at which training logs and metrics are recorded.
* In this case, setting `logging_steps` to 1 means that logs will be generated after every training step.
* Logging can include information such as the current training loss, learning rate, elapsed time, and other relevant metrics.
* More frequent logging can be useful for monitoring the training progress and identifying any potential issues early on.
* However, generating logs after every step can also introduce overhead and slow down the training process, especially for large datasets or long training runs.
* The logging frequency should be adjusted based on the specific needs and the scale of the training task.

### <mark style="color:purple;">`xformers_attention`</mark>

* The `xformers_attention` configuration is related to the use of the XFormers library, which provides optimized attention implementations for transformers.
* XFormers offers various attention mechanisms, such as memory-efficient attention, that can speed up training and reduce memory usage compared to the standard attention implementation in PyTorch.
* Setting `xformers_attention` to a specific value (not provided in the given configuration) would enable the use of XFormers attention in the model.
* The specific attention mechanism and its parameters would depend on the value provided for `xformers_attention`.
* Using XFormers attention can be beneficial for training large models or when dealing with long sequences, as it can provide computational and memory efficiency improvements.

### <mark style="color:purple;">`flash_attention: true`</mark>

* Flash Attention is a highly optimised attention implementation that can significantly speed up the training of transformers.
* It is designed to be memory-efficient and can handle large sequence lengths and batch sizes.
* Setting `flash_attention` to `true` enables the use of Flash Attention in the model.
* Flash Attention can provide substantial performance improvements, especially for models with a large number of attention heads and long sequences.
* It achieves this by using techniques like kernel fusion, memory optimization, and efficient parallelization.
* Enabling Flash Attention can help reduce training time and allow for training larger models or using larger batch sizes.


# Phi 2.0 - Extra Hyperparameters

This is the balance of the configuration file for the training of Phi 2.0.   We provide a full explanation of each of the configurations below.

```yaml
warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
resize_token_embeddings_to_32x: true
special_tokens:
  pad_token: "<|endoftext|>"
```

### <mark style="color:blue;">**Training Hyperparameters**</mark>

### <mark style="color:purple;">`warmup_steps: 100`</mark>

Warm-up steps are a crucial part of the learning rate scheduling.&#x20;

Over the first 100 training steps, the learning rate incrementally increases to its target value. This gradual increase helps in stabilizing the training early on, preventing the model from making too large updates too quickly.

### <mark style="color:purple;">`evals_per_epoch: 4`</mark>

This setting determines the frequency of evaluations within each training epoch.&#x20;

With a value of 4, the model will be evaluated four times per epoch, providing regular feedback on its performance. Frequent evaluations help in monitoring the model's progress and ensuring it is learning as expected.

### <mark style="color:purple;">`saves_per_epoch: 1`</mark>

To safeguard your training progress, the model's state is saved once every epoch. This checkpointing allows you to resume training from the last saved state in case of interruptions and also provides opportunities for model fine-tuning.

### <mark style="color:blue;">**Deepspeed and FSDP**</mark>

### <mark style="color:purple;">`deepspeed:`</mark>

DeepSpeed integration offers advanced optimisations for accelerating training and reducing memory consumption. Configuring DeepSpeed enhances training efficiency, particularly for large-scale models, by optimizing computational resources and parallelizing the workload.

### <mark style="color:purple;">`fsdp_config:`</mark>

Fully Sharded Data Parallel (FSDP) is a technique to reduce memory consumption and increase the scale of distributed training. The `fsdp_config` allows you to customise FSDP's behavior, optimizing memory usage and computational efficiency across multiple devices.

### <mark style="color:purple;">`weight decay: 1`</mark>

Weight decay is a regularization technique to prevent overfitting by penalising large weights.&#x20;

A weight decay factor of 0.1 helps in moderating the update of weights, encouraging the model to learn more general features rather than overly fitting to the training data.

### <mark style="color:blue;">**Special Tokens and Token Embeddings**</mark>

### <mark style="color:purple;">`resize_token_embeddings_to_32x:`</mark>

This configuration enables the resizing of token embeddings, a feature particularly useful for adapting the model to different vocabulary sizes efficiently.&#x20;

Resizing to 32 times the original size allows for more flexible and scalable model architecture.

### <mark style="color:purple;">`pad_token`</mark>

Special tokens play a pivotal role in how the model processes and understands text. The padding token (`pad_token`) is used to fill out sequences to a uniform length, ensuring consistent input size for the model.


# Phi 2.0 - All Configurations

### <mark style="color:blue;">This is the entire configuration file for the fine tuning run of Phi 2.0</mark>

```yaml
base_model: microsoft/phi-2
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: garage-bAInd/Open-Platypus
    type: alpaca

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./phi-sft-out

sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true

adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_torch
adam_beta2: 0.95
adam_epsilon: 0.00001
max_grad_norm: 1.0
lr_scheduler: cosine
learning_rate: 0.000003

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: True
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
resize_token_embeddings_to_32x: true
special_tokens:
  pad_token: "<|endoftext|>"
```


# Phi 2.0 - Preprocessing

To execute the training run, we will first <mark style="color:yellow;">preprocess the dataset.</mark>&#x20;

Axolotl allows us to optionally <mark style="color:blue;">**pre-tokenize dataset**</mark> before finetuning. This is recommended for large datasets.&#x20;

### <mark style="color:blue;">Populate the config.yaml file</mark>

To execute the preprocessing, we have to make sure the <mark style="color:yellow;">**datasets component**</mark> of the config.yaml is correctly populated. &#x20;

You will need to *<mark style="color:yellow;">**direct Axolotl to where your dataset is**</mark>.*

To do this, find the path to your dataset in VS Code by right clicking on the dataset and asking for the <mark style="color:yellow;">relative path.</mark> &#x20;

Then enter it into the <mark style="color:yellow;">**config YAML file**</mark> below which is located at:

### *your directory*/<mark style="color:blue;">axolotl</mark>/<mark style="color:green;">examples</mark>/<mark style="color:purple;">phi</mark>/<mark style="color:yellow;">phi2-ft.yml</mark>

```bash
your directory/axolotl/examples/phi/phi2-ft.yml
```

When you have located the file, then <mark style="color:yellow;">**populate the configuration file**</mark> below.

To do this, find the <mark style="color:yellow;">**path to your dataset**</mark> in VS Code by <mark style="color:green;">**right clicking**</mark> on the dataset and asking for the relative path. &#x20;

Use this in the command below.  <mark style="color:yellow;">Remember to</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">**include the file name.**</mark>

```yaml
datasets:
  - path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
    type: alpaca
        ds_type: json
    data_files:
    - alpaca_data_cleaned.json
dataset_prepared_path:
val_set_size: 0.20
output_dir: ./phi-out
```

The data set type (ds\_type) is JSON, and we have arbitrarily <mark style="color:yellow;">set the output directory as</mark> <mark style="color:blue;">**./phi-out**</mark>

For a refresher of what the alpaca-cleaned dataset contains:

<details>

<summary><mark style="color:green;">Alpaca Cleaned Dataset - Description</mark></summary>

The Alpaca dataset is a <mark style="color:yellow;">curated and cleaned version of an original dataset</mark> released by Stanford, specifically <mark style="color:yellow;">designed for instruction-tuning of language models</mark>.&#x20;

<mark style="color:green;">**Dataset Description and Improvements**</mark>

1. **Issues Addressed**:
   * <mark style="color:blue;">**Hallucinations**</mark><mark style="color:blue;">:</mark> The original dataset included instructions that led to irrelevant or fabricated responses by the model, particularly in cases where the input was a URL or an ambiguous prompt. These have been cleaned to prevent such issues.
   * <mark style="color:blue;">**Merged Instructions**</mark><mark style="color:blue;">:</mark> Instances where multiple instructions were combined have been separated for clarity and precision.
   * <mark style="color:blue;">**Empty Outputs**</mark><mark style="color:blue;">:</mark> Entries with no outputs have been addressed.
   * <mark style="color:blue;">**Missing Code Examples**</mark><mark style="color:blue;">:</mark> The cleaned dataset ensures that code examples are included where necessary.
   * <mark style="color:blue;">**Instructions for Image Generation**</mark><mark style="color:blue;">:</mark> Removed or modified instructions that required image generation, as this is not feasible for a text-based model.
   * <mark style="color:blue;">**N/A Outputs and Inconsistent Input Fields**</mark><mark style="color:blue;">:</mark> These issues have been standardised or corrected.
   * <mark style="color:blue;">**Incorrect Answers**</mark><mark style="color:blue;">:</mark> The dataset has been reviewed for accuracy, especially in areas like math problems where the original dataset had a high error rate.
   * <mark style="color:blue;">**Unclear Instructions**</mark><mark style="color:blue;">:</mark> Ambiguous instructions have been clarified or rewritten.
   * <mark style="color:blue;">**Extraneous Characters**</mark><mark style="color:blue;">:</mark> Removal of unnecessary escape and control characters.
2. **Original Alpaca Dataset Summary**:
   * The <mark style="color:yellow;">original dataset, Alpaca, contains 52,000 instruction examples</mark> created using OpenAI's `text-davinci-003` engine.
   * It was designed for instruction tuning, making language models better at following specific instructions.
   * The dataset was generated more cost-effectively and is more diverse compared to its predecessors.
3. **Dataset Structure**:
   * The dataset consists of fields like `instruction`, `input`, `output`, and `text`, which combine these elements in a format suitable for model fine-tuning.
   * The `input` field is optional and used when context is necessary for the instruction.
4. **Data Splits**:
   * <mark style="color:yellow;">The dataset contains 52,002 training examples.</mark>
5. **Usage and Applications**:
   * It is intended for instruction training of pre-trained language models, particularly in the text-generation domain.
6. **Languages**:
   * The dataset is exclusively in English.

</details>

Once you have configured the dataset component of the YAML file, then execute the following command which uses the <mark style="color:yellow;">**preprocess.py script**</mark> within the Axolotl library. &#x20;

{% code fullWidth="false" %}

```bash
python -m axolotl.cli.preprocess examples/phi/phi2-ft.yml
```

{% endcode %}

The output should indicate success:

```bash
Success! Preprocessed data path: `dataset_prepared_path: ./phi-out'
```

If you have not set a dataset\_prepared\_path in the configuration file, it will default to:

"preprocess CLI called without dataset\_prepared\_path set, using default path: <mark style="color:yellow;">last\_run\_prepared"</mark>

For an <mark style="color:yellow;">analysis of the output</mark> relating to the dataset preprocessing, see below:

<details>

<summary><mark style="color:green;">Analysis of Output from Pre-Processing</mark></summary>

The numbers:

1. **PID (Process ID)**: `6779`
   * This is the ID of the process running the script.
2. **Token IDs**:
   * End Of Sentence (EOS) Token ID: `2`
   * Beginning Of Sentence (BOS) Token ID: `1`
   * Padding (PAD) Token ID: `0`
   * Unknown (UNK) Token ID: `0`
3. **Data Downloading and Processing**:
   * Number of data files downloaded: `1`
   * Download speed: `12557.80 items/second`
   * Number of data files extracted: `1`
   * Extraction speed: `214.70 items/second`
4. **Dataset Generation**:
   * Number of examples in the train split: `51760`
   * Generation speed: `85351.57 examples/second`
5. **Mapping and Filtering**:
   * Number of examples processed in mapping (num\_proc=12): `51760`
   * Processing speed in mapping: `6035.04 examples/second`
   * Number of examples processed in filtering (num\_proc=12): `51760`
   * Filtering speed: `41994.96 examples/second`
   * Number of examples processed in the second mapping (num\_proc=12): `51760`
   * Second mapping speed: `30435.07 examples/second`
6. **Token Counts**:
   * Total number of tokens: `12104896`
   * Total number of supervised tokens: `8475133`
7. **Efficiency Estimates and Data Loading**:
   * Packing efficiency estimate: `1.0`
   * Total number of tokens per device: `12104896`
   * Data loader length: `1461`
   * Sample packing efficiency estimate across ranks: `0.9798729691644562`
   * Sample packing efficiency estimate: `0.98`
   * Total number of steps: `5844`
8. **Time and Date Information**:
   * Date and time of the log entries: `2023-12-06`
   * Times of various log entries: `05:24:22,428`, `05:24:22,688`, `05:24:32,282`, `05:24:36,028`, `05:24:36,393`, `05:24:41,028`, `05:24:41,029`, `05:24:41,037`
   * Total execution time: `26 seconds`
   * Time when the command prompt was ready again: `05:24:42`

</details>

If you would like information on how the preprocess.py script works you can view the code at   <mark style="color:purple;">ax</mark><mark style="color:purple;">**olotl**</mark>**/**<mark style="color:green;">**src**</mark>**/**<mark style="color:blue;">**axolotl**</mark>**/**<mark style="color:yellow;">**cli.**</mark> &#x20;

Alternatively, take the time to read the analysis of the scripts, and the output it produced:

<details>

<summary><mark style="color:green;">Axolotl preprocess.py -</mark> <mark style="color:yellow;">analysis of script</mark> <mark style="color:green;">and</mark> <mark style="color:yellow;">command output</mark></summary>

The provided <mark style="color:yellow;">**`preprocess.py`**</mark> script is a <mark style="color:yellow;">**command-line interface (CLI) tool**</mark> for preprocessing datasets in the context of training models using the Axolotl platform.&#x20;

<mark style="color:green;">**Imports and Logger Setup**</mark>

* The script imports necessary modules such as <mark style="color:yellow;">**`logging`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`pathlib.Path`**</mark> for file path operations, <mark style="color:yellow;">**`fire`**</mark> for the CLI, and several modules from <mark style="color:yellow;">**`transformers`**</mark> and <mark style="color:yellow;">**`axolotl`**</mark>.
* <mark style="color:yellow;">**`colorama`**</mark>**&#x20;i**s used for coloured console output, enhancing readability.
* A logger <mark style="color:yellow;">**`LOG`**</mark> is set up for logging purposes, using the <mark style="color:yellow;">**`logging`**</mark> module.

<mark style="color:green;">**do\_cli Function**</mark><mark style="color:green;">:</mark>

<mark style="color:blue;">**Load Configurations**</mark>

The main function <mark style="color:yellow;">**`do_cli`**</mark> takes a `config` argument (with a <mark style="color:yellow;">default path of</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">**`"examples/"`**</mark>) and <mark style="color:yellow;">**`**kwargs`**</mark> for additional arguments. &#x20;

These configurations include essential details about the dataset, model, and training setup. This step sets up the parameters for how the dataset will be processed and how the model training will proceed.

<mark style="color:blue;">**Load Datasets**</mark>

The function <mark style="color:yellow;">**`load_datasets`**</mark> is called with the loaded configuration and parsed command-line arguments.&#x20;

This function is responsible for the heavy lifting of loading the dataset (which could include downloading it if not already present locally) and preprocessing it according to the specifications in the configuration. This step is central to preparing the data for model training.

<mark style="color:blue;">**ASCII Art and Configuration Loading**</mark>

It starts by printing ASCII art using <mark style="color:yellow;">**`print_axolotl_text_art`**</mark> for visual appeal. This is more for aesthetic purposes and has no impact on the functional aspects of the scrip

* Then, <mark style="color:yellow;">it loads the configuration file</mark> using <mark style="color:yellow;">**`load_cfg`**</mark>, which sets up the parameters for dataset preprocessing.  &#x20;
* &#x20;Initially, there's a deprecation warning from the <mark style="color:yellow;">**`transformers.deepspeed`**</mark> module. This suggests that in future versions of the Transformers library, you'll <mark style="color:yellow;">need to import DeepSpeed modules differently</mark>.&#x20;
* **Accelerator and User Token Checks**: It ensures that the default accelerator configuration is set and validates the user token for authentication or API access.
* **CLI Arguments Parsing**: The script uses <mark style="color:yellow;">**`transformers.HfArgumentParse`**</mark><mark style="color:yellow;">`r`</mark> to parse additional command-line arguments specific to preprocessing, defined in <mark style="color:yellow;">`PreprocessCliArgs`</mark>.
* **Dataset Preparation Path Check**: It checks if <mark style="color:yellow;">**`dataset_prepared_path`**</mark> is set in the configuration.  If the <mark style="color:yellow;">**`dataset_prepared_pat`**</mark><mark style="color:yellow;">`h`</mark> is not explicitly set in the configuration, the script issues a warning and *<mark style="color:yellow;">assigns a default pat</mark>*<mark style="color:yellow;">h</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`DEFAULT_DATASET_PREPARED_PATH`**</mark><mark style="color:yellow;">**).**</mark> This path is where the script will store the preprocessed dataset, ensuring that there’s always a defined location for this data.

<mark style="color:blue;">**Load and Preprocess Datasets**</mark>

The <mark style="color:yellow;">**`load_datasets`**</mark> function is called with the loaded configuration and parsed CLI arguments, handling the dataset loading and preprocessing.

* The script provides debug <mark style="color:yellow;">information about special tokens</mark> (End Of Sentence, Beginning Of Sentence, Padding, and Unknown) used by the tokenizer. This information is crucial for understanding how the model will interpret different types of tokens in the data.

<mark style="color:blue;">**Logging Success**</mark>

* Upon successful completion, the <mark style="color:yellow;">script logs a message indicating the path</mark> where the preprocessed data is stored. This is done using the <mark style="color:yellow;">**`LOG.info`**</mark> method, and the message is coloured green for visibility.

<mark style="color:blue;">**Main Block**</mark>

This block checks if the script is the main program being run <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`__name__ == "__main__"`**</mark><mark style="color:yellow;">**)**</mark> and not a module imported in another script. If it is the main program, it uses <mark style="color:yellow;">**`fire.Fire(do_cli)`**</mark> to enable the script to be run from the command line, where <mark style="color:yellow;">**`do_cli`**</mark> is the main function being executed.

<mark style="color:blue;">**General Observations**</mark>

* **User-Friendly**: The script is user-friendly, providing clear messages and using colour coding for warnings and success messages.
* **Flexibility**: It is flexible and allows customization through command-line arguments.
* **Error Handling**: The script checks for potential issues like missing configuration parameters and handles them gracefully, providing default values and warnings.
* **Logging**: Effective use of logging helps in tracking the script's execution and diagnosing issues if any arise.

OUTPUT

1. <mark style="color:green;">**Warnings and ASCII Art**</mark><mark style="color:green;">:</mark>
2. <mark style="color:green;">**Token Information**</mark><mark style="color:green;">:</mark> The script provides debug <mark style="color:yellow;">information about special tokens</mark> (End Of Sentence, Beginning Of Sentence, Padding, and Unknown) used by the tokenizer. This information is crucial for understanding how the model will interpret different types of tokens in the data.
3. <mark style="color:green;">**Data File Processing**</mark><mark style="color:green;">:</mark> The script downloads and extracts data files, then generates a training split with 51,760 examples. This is followed by mapping and filtering operations on the dataset, performed in parallel (noted by `num_proc=12`), which is an efficient way to handle large datasets.
4. <mark style="color:green;">**Dataset Merging and Saving**</mark><mark style="color:green;">:</mark> Post-processing, the datasets are merged and saved to disk. This is an essential step to ensure that the processed data is stored in a format ready for model training.
5. <mark style="color:green;">**Token Counts and Sample Packing**</mark><mark style="color:green;">:</mark> The script calculates the total number of tokens and supervised tokens. It also estimates the packing efficiency and total number of steps needed for training, which are critical for understanding how the data will be batched and fed into the model during training.
6. <mark style="color:green;">**Final Information and Path**</mark><mark style="color:green;">:</mark> Finally, the script concludes with a success message and provides the path to the preprocessed data. This is your confirmation that the preprocessing was successful and where the prepared data is stored.

</details>

This script for our Axolotl platform will be:&#x20;

<details>

<summary><mark style="color:green;">The configured</mark> <mark style="color:yellow;">Phi2-ft.yml</mark> <mark style="color:green;">YAML file</mark></summary>

```yaml
base_model: microsoft/phi-2
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false
datasets:
  - path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
    type: alpaca
    ds_type: json
    data_files:
      - alpaca_data_cleaned.json
dataset_prepared_path:
val_set_size: 0.20
output_dir: ./phi-out
sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true
adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_torch
adam_beta2: 0.95
adam_epsilon: 0.00001
max_grad_norm: 1.0
lr_scheduler: cosine
learning_rate: 0.000003
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: True
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
resize_token_embeddings_to_32x: true
special_tokens:
  pad_token: "<|endoftext|>"





```

</details>

{% hint style="warning" %}
Potential Flash Attention Dependency Issues
{% endhint %}

### <mark style="color:red;">Flash Attention Issues</mark>

Some users have had some issue with Flash Attention dependencies. &#x20;

For some reason the axolotl environment keeps reverting to Pytorch 2.01 - Flash Attention needs Pytorch version 2.10

We executed this command in the axolotl library, and it upgraded Pytorch to 2.10.

```bash
pip install flash_attn -U --force-reinstall
```

<details>

<summary><mark style="color:green;">Flash Attention Debugging</mark></summary>

Flash Attention needs Pytorch 2.10 - not 2.0.  I had to force install Flash Attention to make it work:\
\
The error message you're encountering originates from running a Python script that attempts to preprocess a configuration file for a machine learning model using the <mark style="color:yellow;">`axolotl.cli.preprocess`</mark> module.&#x20;

The error is an <mark style="color:yellow;">`Import Error`</mark> associated with the <mark style="color:yellow;">`flash_attn_2_cuda`</mark> library, which is part of the `flash_attn` package. Let's break down the error message line by line:

1. **Command Executed**: <mark style="color:yellow;">`python -m axolotl.cli.preprocess your directory/axolotl/examples/llama-2/lora.yml`</mark>
   * This command attempts to run a Python module (<mark style="color:yellow;">`axolotl.cli.preprocess`</mark>) with a specific configuration file as its argument.
2. **Initial Traceback**:
   * The Python interpreter starts by <mark style="color:yellow;">trying to execute the module but encounters an issue in the process.</mark>
3. **Import Chain**:
   * The error arises in a chain of imports starting from your script's initial import statement. Python attempts to import various modules and packages needed for the script to run.
4. **Specific ImportError**:
   * The final error, <mark style="color:yellow;">`ImportError`</mark>, occurs when Python tries to import <mark style="color:yellow;">`flash_attn_2_cuda`</mark> from the <mark style="color:yellow;">`flash_attn`</mark> package.
   * The error message specifically states `undefined symbol: _ZN3c104cuda9SetDeviceEi`. This indicates a missing or incompatible symbol in the compiled CUDA extension.
5. <https://github.com/Dao-AILab/flash-attention/issues/620>

#### <mark style="color:green;">Troubleshooting Steps</mark>

1. **Verify CUDA Installation**:
   * Ensure that CUDA is correctly installed on your system and is compatible with the versions required by `flash_attn`.
   * Check the CUDA version against the version required by <mark style="color:yellow;">`flash_attn`</mark>. If there's a mismatch, consider upgrading or downgrading CUDA.
2. **Check PyTorch and flash\_attn Compatibility**:
   * Verify that the installed PyTorch version is compatible with `flash_attn`. Incompatibilities between CUDA, PyTorch, and `flash_attn` can lead to such errors.
3. **Reinstall flash\_attn**:
   * Try reinstalling the `flash_attn` package. Sometimes, recompiling the package can resolve symbol mismatch errors.
   * Use the command <mark style="color:blue;">`pip install flash-attn --force-reinstall`</mark> to reinstall.
4. **Check Environment Variables**:
   * Ensure that your <mark style="color:yellow;">`LD_LIBRARY_PATH`</mark> environment variable includes the paths to the CUDA libraries.
5. **Inspect Python Environment**:
   * Verify that you are using the correct Python environment where all required dependencies are installed. Sometimes, conflicts in environments can cause such issues.
6. **Check for System Updates**:
   * Occasionally, system updates or changes to the compiler or related libraries can cause incompatibilities. Ensure your system is up to date.
7. **Consult Documentation or Community Forums**:
   * Check the documentation for `flash_attn` and related packages for any known issues or troubleshooting tips.
   * Seek help from relevant community forums or issue trackers for `flash_attn` and related projects.
8. **Test in a Clean Environment**:
   * If possible, try to run the script in a clean Python environment with only necessary packages installed. This can help rule out conflicts with other packages.

</details>

<details>

<summary><mark style="color:green;">Using the Upload File Function</mark></summary>

The <mark style="color:yellow;">`upload file`</mark> function from the Hugging Face Hub library is used to upload files to a repository on the Hugging Face Hub. This function is quite versatile, supporting various parameters to customize the upload process. Here's a breakdown of the parameters and what they mean:

1. <mark style="color:blue;">**path\_or\_fileobj (str, Path, bytes, or IO)**</mark><mark style="color:blue;">:</mark> This is the source of the file you want to upload. It can be a <mark style="color:yellow;">path to a file on your local machine</mark>, a binary data stream, a file object, or a buffer.
2. <mark style="color:blue;">**path\_in\_repo (str)**</mark><mark style="color:blue;">:</mark> This specifies where in the repository the file should be placed. For example, if you want to place a file in a folder called 'checkpoints', you would use something like `"checkpoints/weights.bin"`.
3. <mark style="color:blue;">**repo\_id (str)**</mark><mark style="color:blue;">:</mark> The identifier for the repository to which you are uploading. This usually follows the format `"username/repository-name"`.
4. <mark style="color:blue;">**token (str, optional)**</mark><mark style="color:blue;">:</mark> Your authentication token for the Hugging Face Hub. If you have already logged in using `HfApi.login`, this will default to the stored token. If not provided, the function will attempt to use a stored token from a previous login.
5. <mark style="color:blue;">**repo\_type (str, optional)**</mark><mark style="color:blue;">:</mark> Indicates the type of repository. It can be `"dataset"`, `"space"`, or `"model"`. If you are uploading to a dataset or a space, specify accordingly. The default is `None`, which is interpreted as a model.
6. <mark style="color:blue;">**revision (str, optional)**</mark><mark style="color:blue;">:</mark> Specifies the git revision (like a branch name or commit hash) from which the commit should be made. The default is the head of the "main" branch.
7. <mark style="color:blue;">**commit\_message (str, optional)**</mark><mark style="color:blue;">:</mark> A short summary or title for the commit you are making. Think of this as the headline of your changes.
8. <mark style="color:blue;">**commit\_description (str, optional)**</mark><mark style="color:blue;">:</mark> A more detailed description of the changes you are committing.
9. <mark style="color:blue;">**create\_pr (boolean, optional)**</mark><mark style="color:blue;">:</mark> Determines whether to create a Pull Request for the commit. Defaults to `False`. If `True`, a PR will be created based on the specified `revision`.
10. <mark style="color:blue;">**parent\_commit (str, optional)**</mark><mark style="color:blue;">:</mark> The hash of the parent commit to which your changes will be added. This is used to ensure that you are committing to the correct version of the repository, especially useful in concurrent environments.
11. <mark style="color:blue;">**run\_as\_future (bool, optional)**</mark><mark style="color:blue;">:</mark> If set to `True`, the upload process will run in the background as a non-blocking action. This returns a `Future` object that can be used to check the status of the upload.

The `path_in_repo` parameter in the `api.upload_file` function from the `huggingface_hub` library specifies the destination path within the repository on the Hugging Face Hub where the file will be uploaded. This is essentially the relative path in the repository where the file will be placed.

In this example command:

```python
api.upload_file(
    path_or_fileobj="/path/to/trained_model_weights.bin",
    path_in_repo="model_weights.bin",
    repo_id="username/my_ml_model",
    repo_type="model"
)
```

* <mark style="color:yellow;">`path_or_fileobj="/path/to/trained_model_weights.bin"`</mark>: This is the path to the file on your local machine. It indicates where the file is currently stored in your local file system.
* <mark style="color:yellow;">`path_in_repo="model_weights.bin"`</mark>: This determines where within your Hugging Face Hub repository the file will be saved. In this case, you are instructing the function to upload your file directly to the root of the repository and <mark style="color:yellow;">name it</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`model_weights.bin`</mark>. If you wanted to place this file inside a folder within your repository, you would specify a path like `"`<mark style="color:yellow;">`folder_name/model_weights.bin`</mark>`"`.
* <mark style="color:yellow;">`repo_id="username/my_ml_model"`</mark>: This identifies the specific repository on the Hugging Face Hub where the file should be uploaded. It's a combination of your username (or organization name) and the repository name.
* <mark style="color:yellow;">`repo_type="model"`</mark><mark style="color:yellow;">:</mark> This indicates the type of repository you are uploading to. In this case, it's a model repository.

</details>


# Phi 2.0 - Training

### <mark style="color:green;">Training Phi 2.0</mark>

To execute the main training script Axolotl provides this command to begin training Phi 2.0

```bash
accelerate launch -m axolotl.cli.train examples/phi/phi2-ft.yml
```

If you have not already done so, you will be asked to enter your Weights and Biases API Key.&#x20;

&#x20;Enter the key at the command line prompt:

```yaml
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

### <mark style="color:blue;">An analysis of the axolotl.clt.train module</mark>

<details>

<summary><mark style="color:green;">Analysis of train.py script</mark></summary>

The <mark style="color:yellow;">**`train.py`**</mark> script in the Axolotl platform is a Command Line Interface (CLI) tool designed for training machine learning models.&#x20;

This script is structured to provide a user-friendly interface for configuring and executing model training. Here's a detailed analysis:

#### <mark style="color:blue;">Script Structure and Functionality</mark>

<mark style="color:green;">**Imports and Logger Setup**</mark>

* Essential modules like <mark style="color:yellow;">**`logging`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`pathlib.Path`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`fire`**</mark>, and <mark style="color:yellow;">**`transformers`**</mark> are imported.
* The script sets up a logger <mark style="color:yellow;">**`LOG`**</mark> using the <mark style="color:yellow;">**`logging`**</mark> module for logging various events and statuses during the script's execution.

<mark style="color:green;">**do\_cli Function**</mark>

* <mark style="color:blue;">**Function Definition**</mark><mark style="color:blue;">:</mark> The <mark style="color:yellow;">**`do_cli`**</mark> function is the main entry point of the script. It accepts a <mark style="color:yellow;">**`config`**</mark> argument (with a default value pointing to an "examples" directory) and <mark style="color:yellow;">**`**kwargs`**</mark> for additional arguments.
* <mark style="color:blue;">**ASCII Art Display**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">**`print_axolotl_text_art()`**</mark> is called to display ASCII art, likely for aesthetic purposes.
* <mark style="color:blue;">**Configuration Loading**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">**`load_cfg`**</mark> loads configuration details from the provided `config` path. These configurations are essential for setting up model training parameters.
* <mark style="color:blue;">**Accelerator and User Token Checks**</mark><mark style="color:blue;">:</mark> The script verifies the default configuration for the accelerator (such as a GPU) and checks the user token. These checks are crucial for ensuring that the hardware is correctly set up and the user is authenticated.
* <mark style="color:blue;">**CLI Arguments Parsing**</mark><mark style="color:blue;">:</mark> It uses <mark style="color:yellow;">**`transformers.HfArgumentParser`**</mark> to parse additional CLI arguments into data classes (<mark style="color:yellow;">**`TrainerCliArgs`**</mark>). This step allows for dynamic customization of training parameters via the command line.
* <mark style="color:blue;">**Dataset Loading**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">**`load_datasets`**</mark> is called with the parsed configuration and CLI arguments. This function is responsible for loading the dataset as per the configuration, which is a critical step in the training process.
* <mark style="color:blue;">**Model Training**</mark><mark style="color:blue;">:</mark> The <mark style="color:yellow;">**`train`**</mark> function is invoked with the loaded configuration, CLI arguments, and dataset metadata. This function likely encompasses the core logic for model training.

<mark style="color:green;">**Main Block**</mark>

* The script checks if it's being run as the main program <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`__name__ == "__main__"`**</mark><mark style="color:yellow;">**)**</mark> and not as a module in another script. If it's the main program, it uses <mark style="color:yellow;">**`fire.Fire(do_cli)`**</mark> to execute the <mark style="color:yellow;">**`do_cli`**</mark> function, enabling the script to be interacted with from the command line.

</details>


# Uploading Models

Please click on the link below for a comprehensive review of uploading models on Hugginface:

{% embed url="<https://huggingface.co/docs/hub/models-uploading#upload-a-pytorch-model-using-huggingfacehub>" %}
Uploading models on HuggingFace
{% endembed %}

## <mark style="color:blue;">Repositories</mark>

Models, Spaces, and Datasets are hosted on the Hugging Face Hub as [Git repositories](https://git-scm.com/about), which means that version control and collaboration are core elements of the Hub.&#x20;

In a nutshell, a repository (also known as a **repo**) is a place where code and assets can be stored to back up your work, share it with the community, and work in a team.

### Creating a repository

Using the Hub’s web interface you can easily create repositories, add files and explore models,&#x20;

There are three kinds of repositories on the Hub, and in this guide you’ll be creating a **model repository** for demonstration purposes.&#x20;

To create a new repository, visit [huggingface.co/new](http://huggingface.co/new)<mark style="color:blue;">:</mark>

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FwDIiA8x4mrY2iw28iY66%2Fimage.png?alt=media&amp;token=c8d14840-4ff7-4697-9ffb-009d80bace45" alt=""><figcaption></figcaption></figure>

Specify the <mark style="color:yellow;">owner of the repository:</mark> this can be either you or any of the organisations you’re affiliated with.

Them enter your model’s name, this will also be the name of the repository.  Then Specify whether you want your model to be public or private.

Specify the license. You can leave the *License* field blank for now. To learn about licenses, visit the [**Licenses**](https://huggingface.co/docs/hub/repositories-licenses) documentation.

After creating your model repository, you should see a page like this:

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FUE8jr1D4LIyFfU3CFM9l%2Fimage.png?alt=media&amp;token=98e7ef1b-d678-4cce-b19a-9fc95881ef92" alt=""><figcaption><p>Blank Model Card</p></figcaption></figure>

Note that the Hub prompts you to create a *Model Card*, which you can learn about in the [**Model Cards documentation**](https://huggingface.co/docs/hub/model-cards).&#x20;

Including a Model Card in your model repo is best practice, but since we’re only making a test repo at the moment we can skip this.

<details>

<summary><mark style="color:green;">Importance of completing a comprehensive model card</mark></summary>

A model card is an essential document that accompanies a model, providing crucial information about the model's characteristics, intended uses, limitations, and performance.&#x20;

Properly filling out a model card is vital for ensuring transparency, reproducibility, and responsible usage of the model.&#x20;

Here's a step-by-step guide on how to fill out a model card, along with explanations of why each piece is important:

<mark style="color:blue;">Model Description</mark>

* Provide a clear and concise description of your model, including its architecture, purpose, and key features.
* Explain the problem domain and the specific task(s) the model is designed to address.
* Specify the model's version and any relevant identifying information.
* Importance: A clear model description helps users understand what the model does and whether it is suitable for their specific use case. It also aids in model discoverability and comparison.

<mark style="color:blue;">Intended Uses & Limitations</mark>

* Clearly state the intended uses of the model and the scenarios for which it was designed.
* Discuss any known limitations, biases, or ethical considerations associated with the model.
* Provide guidance on the appropriate and inappropriate uses of the model.
* Importance: Specifying intended uses and limitations ensures that the model is used responsibly and within its designed scope. It helps prevent misuse and potential harm arising from applying the model to unsuitable tasks or contexts.

<mark style="color:blue;">Training Data</mark>

* Describe the dataset(s) used to train the model, including their sources, size, and any preprocessing steps applied.
* Provide information about the data distribution, such as class balance, demographic information, or any notable biases.
* Discuss any data quality issues, noise, or limitations that may impact the model's performance.
* Importance: Transparency about the training data allows users to assess the model's generalizability and potential biases. It enables reproducibility and helps users understand the model's strengths and weaknesses.

<mark style="color:blue;">Training Procedure</mark>

* Outline the training methodology, including the optimisation algorithm, hyperparameters, and any specific techniques used (e.g., transfer learning, data augmentation).
* Specify the hardware and software environment used for training.
* Provide information about the training duration, number of epochs, and any early stopping criteria.
* Importance: Documenting the training procedure facilitates reproducibility and allows users to understand the factors that influenced the model's performance. It also enables comparisons with other models and aids in debugging and improvement efforts.

<mark style="color:blue;">Evaluation Metrics & Results</mark>

* Clearly define the evaluation metrics used to assess the model's performance, such as accuracy, precision, recall, or domain-specific metrics.
* Present the model's performance results on held-out test data or benchmark datasets.
* Provide confidence intervals, statistical significance, or other measures of uncertainty, if applicable.
* Discuss any limitations or caveats in the evaluation process.
* Importance: Evaluation metrics and results give users an objective assessment of the model's performance and help them determine if the model meets their requirements. It allows for comparative analysis with other models and informs decision-making.

<mark style="color:blue;">Contact Information & Resources</mark>

* Include contact information for the model's creators or maintainers for questions, feedback, or support.
* Provide links to additional resources, such as research papers, documentation, or code repositories.
* Specify the license under which the model is released and any associated terms of use.
* Importance: Contact information and resources enable users to seek further information, report issues, or collaborate with the model's creators. Clear licensing information ensures proper attribution and sets expectations for model usage.

By thoroughly filling out each section of the model card, you provide users with a comprehensive understanding of your model's capabilities, limitations, and considerations. This transparency builds trust, promotes responsible usage, and enables users to make informed decisions when employing your model in their applications.

</details>

**Choose Repository Type**: Decide whether you want to link the repository with an individual user or an organization. Organizations can collect models related to a company, community, or library.

### <mark style="color:blue;">**Upload Methods**</mark>

**There are a number of upload methods.** &#x20;

Because <mark style="color:blue;">model repos</mark> are Git repositories we can use Git to push your model files to the Hub.&#x20;

Follow the guide on Getting Started with Repositories for detailed instructions on using the git CLI to commit and push your models.

* **Custom PyTorch Model**: If your model is a custom PyTorch model, use the `huggingface_hub` Python library, which enables capabilities like `from_pretrained`, `push_to_hub`, and automated download metrics.
* **Web Interface**: Alternatively, you can upload models using the web interface. Visit huggingface.co/new and follow the steps to create a new model repository. You can upload files from your computer and leave a commit message to document your changes.

**Inspect Files and History**: After uploading, you can inspect your repository, view recently added files, explore commits, and see the difference introduced by each commit.

**Add Metadata**: Enhance your model card by adding metadata such as the task type, used library, language, dataset, metrics, license, and more. This helps users understand your model better.

**Model Usage Examples**:

* For libraries with built-in support, like Transformers, use methods provided by the library to push/load from the Hub.
* For custom PyTorch models, leverage the `PyTorchModelHubMixin` class from the `huggingface_hub` library. Define hyperparameters in a config dictionary, create a class that takes the config, and utilize `save_pretrained`, `push_to_hub`, and `from_pretrained` methods.
* Ensure automated download metrics are available for your model, and consider adding a model card to provide additional information about your model.


# Llama2

Meta has introduced the Llama 2 family of large language models (LLMs), spanning from 7 billion to 70 billion parameters.&#x20;

These models, particularly the Llama-2-Chat variants, are fine-tuned for dialogue applications and demonstrate superior performance compared to many open-source chat models, aligning closely with industry leaders like ChatGPT and PaLM in benchmarks of helpfulness and safety.

### <mark style="color:blue;">**Model Variations**</mark>

Llama 2 offers several sizes (7B, 13B, 70B) and includes both pretrained and fine-tuned models, with Llama-2-Chat being the standout for dialogue tasks.

### <mark style="color:blue;">**Architecture and Training**</mark>

Llama 2 employs an auto-regressive transformer architecture. The models are trained using a mix of publicly available data, with the 70B variant incorporating Grouped-Query Attention for enhanced inference scalability. Training occurred between January and July 2023.

### <mark style="color:blue;">Instructions for Accessing and Using Llama v2 Models</mark>

### <mark style="color:green;">**Finding the Models**</mark>

1. **Access HuggingFace Hub**:
   * Visit the [HuggingFace hub](https://huggingface.co/models).
   * Use the search bar to look for "Llama v2" models.
2. **Identify the Correct Models**:
   * Look for models that have 'hf' in their name. For example, a model might be named `llama-v2-hf`.
   * Models with 'hf' in their name are already converted to HuggingFace checkpoints. This means they are ready to use and require no further conversion.
3. **Download or Use Models**:
   * You can download these models directly for offline use, or you can reference them in your code by using their HuggingFace hub path.

### <mark style="color:blue;">Download the model into your LLama2-recipe directory</mark>

First, we must connect to Huggingface.

Create a repository to store a Huggingface token

```bash
git config --global credential.helper store
```

Then connect to Huggingface

```bash
huggingface-cli login
```

When asked to enter the authentication code, use:

```bash
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
```

<details>

<summary><mark style="color:green;">What is a Huggingface User Token</mark></summary>

User Access Tokens on the Hugging Face platform serve as a secure method for authenticating applications and notebooks to access Hugging Face services.&#x20;

#### Purpose of User Access Tokens

* **Preferred Authentication Method**: They are the recommended way to authenticate an application or notebook to Hugging Face services.
* **Management**: Managed through the user's settings on the Hugging Face platform.

#### Scope and Roles

* **Read Role**: Tokens with this role provide read access to both public and private repositories that the user or their organization owns. They are suitable for tasks like <mark style="color:blue;">downloading private models or performing inference.</mark>
* **Write Role**: In addition to read privileges, tokens with the write role can modify content in repositories where the user has write access. This includes <mark style="color:blue;">creating or updating repository content, such as training or modifying a model card</mark>.

#### Organization API Tokens

* **Deprecated**: <mark style="color:blue;">Organization API Tokens have been deprecated. User Access Tokens now encompass permissions based on individual roles within an organization.</mark>

#### Managing User Access Tokens

* **Creation**: Access tokens are created in the user's settings under the Access Tokens tab.
* **Customization**: Users can select a role and name for each token.
* **Control**: Tokens can be deleted or refreshed for security purposes.

#### Usage of User Access Tokens

* **Versatility**: Can be used in various ways, including:
  * <mark style="color:blue;">As a replacement for a password for Git operations or basic authentication.</mark>
  * <mark style="color:blue;">As a bearer token for calling the Inference API.</mark>
  * Within Hugging Face Python libraries, like `transformers` or `datasets`, by passing the token for accessing private models or datasets.
* **Security Warning**: Users are cautioned to safeguard their tokens to prevent unauthorized access to their private repositories.

#### Best Practices

* **Separate Tokens for Different Uses**: Create distinct tokens for different applications or contexts (e.g., local machine, Colab notebook, custom servers).
* **Role Appropriateness**: Assign only the necessary role to each token. If only read access is needed, limit the token to the read role.
* **Token Management**: Regularly rotate and manage tokens to ensure security, especially if a token is suspected to be compromised.

In summary, User Access Tokens are a secure and flexible way to interact with the Hugging Face Hub, allowing for controlled access to resources based on defined roles. Proper management and usage of these tokens are crucial for maintaining security and operational efficiency.

</details>

When asked whether you want to add this as a <mark style="color:green;">git token credential</mark>, say yes.  This should be the output:

```
Token is valid (permission: read).
Your token has been saved in your configured git credential helpers (store).
Your token has been saved to /home/paperspace/.cache/huggingface/token
Login successful
```

The install git large file storage so we can download the model (which is large)

```bash
git lfs install
```

If successful, the following output will be displayed in your terminal

```bash
Updated git hooks.
Git LFS initialized.
```

<details>

<summary><mark style="color:green;"><strong>What is Git large file storage</strong></mark></summary>

\
`Git Large File Storage (LFS)` is an extension for Git that addresses issues related to handling large files and binary data in Git repositories.

#### Why Git LFS is Necessary

1. **Limitations with Large Files in Git**:
   * Standard Git is excellent for handling text files (like code), which are typically small and benefit from Git's delta compression (storing only changes between versions). However, <mark style="color:blue;">Git isn't optimized for large binary files (like images, videos, datasets, etc.).</mark> These files can dramatically increase the size of the repository and degrade performance.
2. **Efficiency and Performance**:
   * Cloning and pulling from a repository with large files can be slow, consume a lot of bandwidth, and require significant storage space on every developer's machine. This inefficiency can hinder collaboration and development speed.
3. **Repository Bloat**:
   * In standard Git, every version of every file is stored in the repository's history. This means that even if a large file is deleted from the current working directory, its history still resides in the Git database, leading to a bloated repository.

#### How Git LFS Works

1. **Pointer Files**:
   * Git LFS replaces large files in your repository with tiny pointer files. When you commit a large file, Git LFS stores a reference (pointer) to that file in your repository, while the actual file data is stored in a separate server-side LFS storage.
2. **LFS Storage**:
   * The large file contents are stored on a remote server configured for LFS, typically alongside your Git repository hosting service (like GitHub, GitLab, Bitbucket, etc.). This storage is separate from your main Git repository storage.
3. **Version Tracking**:
   * Git LFS tracks versions of the large files separately. Every time you push or pull changes, Git LFS uploads or downloads the correct version of the large file from the LFS server, ensuring that you have the correct files in your working copy.
4. **Selective Download**:
   * When cloning or pulling a repository, Git LFS downloads only the versions of large files needed for your current commit, reducing the time and bandwidth compared to downloading the entire history of every file.
5. **Compatibility**:
   * Git LFS is compatible with existing Git services and workflows. It's an extension to Git, so you use the same Git commands. Repositories using Git LFS are still standard Git repos, ensuring backward compatibility.

#### Using Git LFS

1. **Installation**:
   * First, install Git LFS on your machine. It's a separate software package that integrates with Git.
2. **Initialization**:
   * In your Git repository, initialize Git LFS using `git lfs install`. This step configures Git to use LFS.
3. **Tracking Large Files**:
   * Specify which files to track with LFS by file extension or pattern using `git lfs track "*.ext"`, where `*.ext` is the pattern for files you want to track (like `*.mp4` for video files).
4. **Commit and Push**:
   * After tracking, these files are handled by Git LFS. When you commit and push these files, they are stored in the LFS storage.

#### In Summary

Git LFS is an essential tool for teams dealing with large binary files in their repositories. It maintains Git's efficiency and performance while handling large files in a more optimized manner. By storing the actual file data separately and using pointer files in the repository, Git LFS prevents repository bloat, speeds up operations like clone and pull, and makes handling large files more manageable without altering the standard Git workflow.

</details>

Enter the git clone command:

```
git clone https://huggingface.co/meta-llama/Llama-2-7b-chat-hf
```

The outcome should be new folder called LLama2-7b-chat-hf with all the appropriate files in it

```
Cloning Process:

Object Enumeration: The Git remote server started by enumerating objects. 
It found 88 objects in total.
Object Download: All 88 objects were successfully downloaded.
The delta 0 indicates that these objects did not have delta compression 
(differences between similar objects), which is common for initial clones.

Unpacking Objects: Git then unpacked these objects, which totalled 499.00 KiB in size.
The download speed was approximately 4.50 MiB/s.

Filtering Content with Git LFS: The repository uses Git LFS to handle large files. During the cloning process, Git LFS filtered and downloaded large files (totaling 9.10 GiB) at a rate of about 16.73 MiB/s.
Issues with Git LFS Files on Windows:

After cloning, there's a note about encountering issues with two files: 
model-00001-of-00002.safetensors and pytorch_model-00001-of-00002.bin.
These files, likely large due to being managed by Git LFS, may not have been copied 
correctly. This kind of issue can occur due to various reasons like LFS quota limits, network issues, or file system limitations, especially on Windows.

Reference to Git LFS Help:

The message See: 'git lfs help smudge' for more details suggests using the 
Git LFS smudge command for help. The smudge command in Git LFS is involved in converting
pointer files in Git LFS to the actual large files.

Overall Time Taken:

The entire cloning process took 9m17s (9 minutes and 17 seconds).
```

Congratulations, you have now downloaded the models.


# Llama2 - Model Configuration

### <mark style="color:blue;">Model Configuration</mark>

The first configuration block of the Axolotl configuration file is 'model type'.  It comprises three main configurations.

1. base\_model
2. model\_type
3. tokenizer\_type

```yaml
base_model: NousResearch/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true
```

Below is an analysis of the Huggingface Transformer classes that are used in the Axolotl training script:

<details>

<summary>Reference: <mark style="color:yellow;"><strong>LlamaForCausalLM</strong></mark> <mark style="color:green;">- a class within the Huggingface Transformer library</mark></summary>

The <mark style="color:yellow;">**`LlamaForCausalLM`**</mark> class is a high-level interface that allows users to easily use the Llama language model for causal language modelling tasks.&#x20;

Causal language modeling involves predicting the next word or token in a sequence based on the previous words or tokens. This class encapsulates the core functionalities of the Llama model, making it more accessible and user-friendly for developers and researchers working with language models.

It encapsulates the complexities of the underlying model architecture and <mark style="color:yellow;">offers methods for initialization, input/output handling, forward pass, and text generation</mark>.&#x20;

This class enables developers and researchers to easily fine-tune and use the Llama model for various natural language processing applications, such as text completion, content generation, and language understanding. It contributes to the field of LLMs by making the Llama model more accessible and facilitating its integration into real-world applications.

<mark style="color:green;">**Model Initialization**</mark>

* The class takes a configuration object <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`config`**</mark><mark style="color:yellow;">**)**</mark> that specifies the architecture and hyperparameters of the Llama model.
* It initializes the Llama model <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`LlamaModel`**</mark><mark style="color:yellow;">**)**</mark> using the provided configuration.
* It sets up the language modelling head <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`lm_head`**</mark><mark style="color:yellow;">**),**</mark> which is a linear layer that maps the hidden states of the model to the vocabulary size for predicting the next token.

<mark style="color:green;">**Class Inheritance**</mark>

The <mark style="color:yellow;">**`LlamaForCausalLM`**</mark> class inherits from the <mark style="color:yellow;">**`LlamaPreTrainedModel`**</mark> class, which is a base class for all Llama-based pretrained models.

<mark style="color:green;">**Class Attributes**</mark>

The class has a class attribute <mark style="color:yellow;">**`_tied_weights_keys`**</mark> which is a list containing the string <mark style="color:yellow;">**`"lm_head.weight"`**</mark>. This attribute is used for weight tying between the input and output embeddings.

<mark style="color:green;">Initialization</mark> <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`__init__`**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**method)**</mark>

* The <mark style="color:yellow;">**`__init__`**</mark> method takes a <mark style="color:yellow;">`b`</mark> parameter, which is an instance of a configuration class specific to the Llama model.
* It calls the <mark style="color:yellow;">**`super().__init__(config)`**</mark> to initialize the parent class with the provided configuration.
* It creates an instance of the <mark style="color:yellow;">**`LlamaModel`**</mark> class using the provided configuration and assigns it to the `model` attribute.
* It sets the <mark style="color:yellow;">**`vocab_size`**</mark> attribute based on the <mark style="color:yellow;">**`vocab_size`**</mark> from the configuration.
* It creates a linear layer <mark style="color:yellow;">**`lm_head`**</mark> with input size <mark style="color:yellow;">**`config.hidden_size`**</mark> and output size <mark style="color:yellow;">**`config.vocab_size`**</mark>, without bias.
* Finally, it calls the <mark style="color:yellow;">**`post_init()`**</mark> method to perform any necessary post-initialization steps.

<mark style="color:green;">**Embedding Methods**</mark>

The class provides methods to get and set the input and output embeddings:

* <mark style="color:yellow;">**`get_input_embeddings()`**</mark> returns the <mark style="color:yellow;">**`embed_tokens`**</mark> attribute of the `model`.
* <mark style="color:yellow;">**`set_input_embeddings(value)`**</mark> sets the <mark style="color:yellow;">**`embed_tokens`**</mark> attribute of the `model` to the provided `value`.
* <mark style="color:yellow;">**`get_output_embeddings()`**</mark> returns the <mark style="color:yellow;">**`lm_head`**</mark> attribute.
* <mark style="color:yellow;">**`set_output_embeddings(new_embeddings)`**</mark> sets the <mark style="color:yellow;">**`lm_head`**</mark> attribute to the provided `new_embeddings`.

<mark style="color:green;">**Decoder Methods**</mark>

* The class provides <mark style="color:blue;">methods to get and set the decoder:</mark>
  * <mark style="color:yellow;">**`set_decoder(decoder)`**</mark> sets the <mark style="color:yellow;">**`model`**</mark> attribute to the provided <mark style="color:yellow;">**`decoder`**</mark>.
  * <mark style="color:yellow;">**`get_decoder()`**</mark> returns the <mark style="color:yellow;">**`model`**</mark> attribute.

<mark style="color:green;">Forward Pass</mark> <mark style="color:yellow;">**(**</mark><mark style="color:yellow;">**`forward`**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**method)**</mark>

* The <mark style="color:yellow;">`b`</mark> method is decorated with `@add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)` and `@replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)`, which add docstrings and modify the return type documentation.
* It takes <mark style="color:blue;">various input parameters</mark> such as <mark style="color:yellow;">**`input_ids`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`attention_mask`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`position_ids`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`past_key_values`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`inputs_embeds`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`labels`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`use_cache`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`output_attentions`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`output_hidden_states`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`return_dict`**</mark><mark style="color:yellow;">**, and**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`cache_position`**</mark><mark style="color:yellow;">**.**</mark>
* It performs the forward pass of the Llama model by calling the <mark style="color:yellow;">**`model`**</mark> with the provided inputs and configuration options.
* It retrieves the hidden states from the model's output.
* If <mark style="color:yellow;">**`config.pretraining_tp > 1`**</mark>, it splits the <mark style="color:yellow;">`lm_head.weight`</mark> into slices and applies linear transformation to the hidden states using each slice, concatenating the results.
* Otherwise, it applies the <mark style="color:yellow;">**`lm_head`**</mark> linear layer to the hidden states.
* It calculates the language modelling loss if <mark style="color:yellow;">**`labels`**</mark> are provided, using the <mark style="color:yellow;">**`CrossEntropyLoss`**</mark> function.
* Finally, it returns the computed logits and other outputs based on the `return_dict` flag.

<mark style="color:green;">**Input Preparation for Generation (**</mark><mark style="color:green;">**`prepare_inputs_for_generation`**</mark><mark style="color:green;">**&#x20;**</mark><mark style="color:green;">**method)**</mark>

* This method prepares the inputs for the generation process.
* It handles the caching mechanism for past key values and adjusts the input tensors accordingly.
* It also handles the case where <mark style="color:yellow;">**`inputs_embeds`**</mark> are provided instead of <mark style="color:yellow;">**`input_ids`**</mark>.
* It returns a dictionary containing the prepared input tensors and configurations.

<mark style="color:green;">**Cache Reordering (**</mark><mark style="color:green;">**`_reorder_cache`**</mark><mark style="color:green;">**&#x20;**</mark><mark style="color:green;">**static method)**</mark>

* This static method is used to reorder the cache (past key values) based on the provided <mark style="color:yellow;">**`beam_idx`**</mark> during beam search decoding.
* It reorders the past states for each layer using the <mark style="color:yellow;">**`index_select`**</mark> operation.

Overall, the <mark style="color:yellow;">**`LlamaForCausalLM`**</mark> class is a high-level interface for using the Llama model for causal language modelling tasks.&#x20;

</details>

<details>

<summary>Reference: <mark style="color:yellow;"><strong>LlamaTokenizer</strong></mark> <mark style="color:green;">- a class within the Huggingface Transformer library</mark></summary>

Purpose:

* The <mark style="color:yellow;">**`AutoTokenizer`**</mark> class is designed to automatically instantiate the appropriate tokenizer class based on the provided pretrained model name or path.
* It serves as a convenient way to load tokenizers without explicitly specifying the tokenizer class.

<mark style="color:green;">Instantiation</mark>

* The class cannot be instantiated directly using the <mark style="color:yellow;">**`__init__()`**</mark> method. Instead, it raises an <mark style="color:yellow;">**`EnvironmentError`**</mark> to indicate that the class should be instantiated using the <mark style="color:yellow;">**`AutoTokenizer.from_pretrained()`**</mark> class method.

<mark style="color:yellow;">**`from_pretrained()`**</mark> <mark style="color:green;">class method</mark>

* This is the main method used to instantiate the appropriate tokenizer class.
* It takes the <mark style="color:yellow;">**`pretrained_model_name_or_path`**</mark> parameter, which can be a model ID, a path to a directory containing vocabulary files, or a path/URL to a single vocabulary file.
* Additional parameters can be passed to customize the tokenizer's behavior, such as <mark style="color:yellow;">**`use_fast`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`tokenizer_type`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`trust_remote_code`**</mark>, and other tokenizer-specific arguments.
* The method first checks if the <mark style="color:yellow;">**`tokenizer_type`**</mark> is provided and tries to load the corresponding tokenizer class.
* If <mark style="color:yellow;">**`tokenizer_type`**</mark> is not provided, it attempts to load the tokenizer class based on the <mark style="color:yellow;">**`tokenizer_config`**</mark> or <mark style="color:yellow;">**`config`**</mark> associated with the pretrained model.
* If the tokenizer class is not found, it falls back to using the <mark style="color:yellow;">**`model_type`**</mark> derived from the configuration class.

<mark style="color:green;">Configuration handling</mark>

* The class uses the <mark style="color:yellow;">**`PretrainedConfig`**</mark> class to determine the appropriate tokenizer class to instantiate.
* It first tries to load the tokenizer configuration from the <mark style="color:yellow;">**`tokenizer_config`**</mark> file associated with the pretrained model.
* If the <mark style="color:yellow;">**`tokenizer_config`**</mark> is not available, it falls back to using the <mark style="color:yellow;">**`AutoConfig`**</mark> class to load the model configuration.

<mark style="color:green;">Fast tokenizers</mark>

* The class supports loading fast tokenizers, which are implemented in Rust and provide faster tokenization.
* If <mark style="color:yellow;">**`use_fast`**</mark> is set to <mark style="color:yellow;">**`True`**</mark> (default), it tries to load the fast tokenizer version if available.
* If the fast tokenizer is not available, it falls back to the slow (Python-based) tokenizer.

<mark style="color:green;">Trust remote code</mark>

* The class includes a <mark style="color:yellow;">**`trust_remote_code`**</mark> parameter to control whether to allow loading custom models defined on the Hugging Face Hub.
* If set to <mark style="color:yellow;">**`True`**</mark>, it executes code present on the Hub on the local machine, which should only be done for trusted repositories.

<mark style="color:green;">Error handling</mark>

* The class raises appropriate exceptions and provides informative error messages when the requested tokenizer class is not found or when there are inconsistencies in the provided parameters.

<mark style="color:green;">Tokenizer registration</mark>

* The class provides a <mark style="color:yellow;">**`register()`**</mark> method to register new tokenizers in the tokenizer mapping.
* It allows registering a configuration class along with the corresponding slow and fast tokenizer classes.

Overall, the <mark style="color:yellow;">**`AutoTokenizer`**</mark> class provides a convenient and flexible way to load tokenizers based on the pretrained model name or path.

It handles the complexity of determining the appropriate tokenizer class and provides options for customization. The class is well-structured and follows good coding practices, such as raising exceptions for invalid usage and providing clear error messages.

</details>

The next step after determining the model type configurations is to configure the <mark style="color:blue;">d</mark><mark style="color:blue;">**ata loading and processing parameters**</mark>


# Llama2 - Model Quantization

With the model configured, we next have to determine whether whether we will be using quantization in the training process.

**We will be using the&#x20;**<mark style="color:yellow;">**Lora Parameter Efficient Fine Tuning technique**</mark>

### <mark style="color:blue;">Model Configuration</mark>

The first configuration block of the Axolotl configuration file is 'model type'. &#x20;

It comprises three main configurations.

1. base\_model
2. model\_type
3. tokenizer\_type

```yaml
load_in_8bit: true
load_in_4bit: false
strict: false
```

### <mark style="color:blue;">load\_in\_8bit: true or false</mark>

This is configuration flag that determines <mark style="color:yellow;">whether the model should be loaded in</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">**8-bit precision.**</mark>  If it is set to "true," the model will be loaded in 8-bit precision.

<mark style="color:blue;">**Memory Efficiency:**</mark> 8-bit precision reduces the memory footprint of the model compared to higher precision formats (like 16-bit or 32-bit). This is because it requires less memory to store each weight in the model.

Loading a model in 8-bit precision can accelerate model loading and inference times. This is due to the reduced computational load compared to higher precision formats.

While 8-bit precision is more efficient, it can slightly reduce the accuracy of the model compared to full precision (32-bit). This happens because of the reduced resolution in representing the weights and activations.

### <mark style="color:blue;">load\_in\_4bit: true or false</mark>

This is configuration flag that determines whether the model should be loaded in **4-bit precision**.  If it is set to "true," the model will be loaded in 4-bit precision.

4-bit precision takes the concept of memory efficiency further, halving the memory requirements compared to 8-bit. This can be crucial for deploying large models on limited hardware.

Similar to 8-bit, 4-bit precision can lead to even faster loading and inference times due to the further reduced computational requirements.

The trade-off in accuracy might be more pronounced in 4-bit precision. The reduced bit-depth means that the model's ability to represent nuanced information in weights and activations is more limited. This might affect tasks that require high precision or are sensitive to small changes in weights.

### <mark style="color:blue;">strict: true or false</mark>

#### Strict Mode (strict)

* If set to false, default weights will be chosen where missing in adapters.  This is a component of 'bits and bytes' library

<details>

<summary><mark style="color:green;">What is 'bits and bytes'?</mark></summary>

**Overview**

The <mark style="color:yellow;">`bitsandbytes`</mark> repository provides a lightweight wrapper around CUDA custom functions, primarily focusing on 8-bit optimizers, matrix multiplication <mark style="color:yellow;">(</mark><mark style="color:yellow;">`LLM.int8()`</mark><mark style="color:yellow;">),</mark> and quantization functions. This tool is designed to enhance the performance and efficiency of machine learning models, particularly in the context of CUDA-enabled computing environments.

**Key Features**

* **8-bit Optimizers**: Specialized for reducing memory usage and improving computational efficiency.
* **Matrix Multiplication (LLM.int8())**: Offers optimized matrix multiplication capabilities.
* **Quantization Functions**: Includes various methods for quantizing models, contributing to reduced model sizes and potentially faster inference times.

**Requirements**

* Python version 3.8 or higher.
* Linux distribution (Ubuntu, MacOS, etc.) with CUDA version greater than 10.0.
* Note: CUDA 10.0 is deprecated, and future support is focused on CUDA >= 11.0 with release 0.39.0.

**Installation**

* Installable via pip (`pip install bitsandbytes`).
* In cases where compilation from source is necessary, users are encouraged to submit a bug report and follow the provided compilation instructions.

**Usage Highlights**

* **Int8 Inference with HuggingFace Transformers**: Allows models to load in 8-bit for reduced memory usage.
* **8-bit Optimizer Usage**: Users can easily switch to 8-bit optimizers by replacing their existing optimizers with the corresponding 8-bit version from `bitsandbytes`.
* **Mixed 8-bit Training and Int8 Inference**: The library supports both mixed 8-bit training with 16-bit main weights and full 8-bit inference.

**Features**

* Advanced techniques for 8-bit matrix multiplication and LLM.int8() inference.
* A range of 8-bit optimizers including Adam, AdamW, RMSProp, LARS, LAMB, and Lion.
* A stable embedding layer feature for improved stability in NLP models.
* Fast and efficient algorithms for quantile estimation.

**Requirements & Hardware Compatibility**

* Requires Anaconda, cudatoolkit, and PyTorch.
* Compatible with NVIDIA GPUs, specifically Turing or newer for LLM.int8(), and Kepler or newer for 8-bit optimizers and quantization.
* Supports CUDA versions from 10.2 to 12.0.
* Note: The library is currently supported only on Linux distributions.

**Additional Information**

* The repository includes instructions for users of Fairseq, detailing how to implement the Stable Embedding Layer and use the optimizers.
* A section on troubleshooting common errors and solutions is provided.
* The repository also contains detailed instructions for compiling from source if necessary.

**Licensing**

* The majority of `bitsandbytes` is under the MIT license, with some portions (like Pytorch) under different licenses.

**Community Contributions**

* Acknowledgment to Fabio Cannizzo for his work on FastBinarySearch, used for CPU quantization in the project.

</details>

<details>

<summary><mark style="color:green;">Summary of Tim Detmers' Presentation on 8-Bit Methods for Efficient Deep Learning</mark></summary>

His main thesis is that computationally efficient methods will accelerate progress in understanding deep learning.

<mark style="color:green;">**Key Points from Tim's Presentation**</mark>

**8-Bit Methods for Large Models**: Tim highlights the importance of making large models more accessible through quantization, which <mark style="color:yellow;">reduces the memory footprint.</mark>

**Quantization Explained**: He explains quantization as a process of <mark style="color:yellow;">converting floating-point or real representations into discrete buckets</mark>, akin to histogram binning.

**Linear vs. Nonlinear Quantization**: <mark style="color:yellow;">Linear (integer) quantization involves equally wide bins, while nonlinear quantization allows varying bin widths</mark>.

**Error Reduction in Quantization**: Tim illustrates how the <mark style="color:yellow;">choice of bins impacts precision and error distribution in quantized values.</mark>

**4-Bit Inference**: His recent work shows that <mark style="color:yellow;">4-bit inference is highly effective for large transformers</mark>.

**Floating Point Data Types**: The presentation delves into the structure of floating point data types, explaining the roles of exponent bits and fraction bits.

**Dynamic Exponent Data Type**: Tim introduces a <mark style="color:yellow;">unique data type he developed with a dynamic exponent</mark>, which offers flexibility in approximating large and small values with varying precision.

**8-Bit Optimizers**: The focus shifts to 8-bit optimizers, crucial for memory efficiency in training large models, particularly in language modeling.

Tim discusses <mark style="color:yellow;">reducing memory usage by approximately 40% by converting 32-bit Adam optimizer buffers to 8-bit.</mark>

This reduction is significant as it helps make large models more memory-efficient.

<mark style="color:yellow;">Outliers in Adam optimizer buffers cause issues in quantization</mark>, leading to increased error and ineffective 8-bit quantization.

Tim presents an example showing how <mark style="color:yellow;">outliers can skew the data</mark>, leading to a waste of bits and loss of effective representation.

To address the problem of outliers, Tim <mark style="color:yellow;">proposes chunking Adam states into blocks and quantizing each block independently</mark>.

This method isolates the impact of outliers to specific blocks, enhancing the stability of 8-bit optimizers.

The process involves <mark style="color:yellow;">chunking state into blocks</mark>, finding the maximum value for normalization, and storing the index for 8-bit representation.

This method ensures compact yet effective optimization, <mark style="color:yellow;">comparable to 32-bit optimizers</mark>.

This achievement indicates significant memory savings without compromising performance.

8-bit optimizers are efficient in mapping onto hardware, with the <mark style="color:yellow;">main overhead being the dequantization process.</mark>

<mark style="color:yellow;">Outliers become a significant problem in models larger than 6.7 billion parameters</mark>, causing performance drops.

Tim's research identifies <mark style="color:yellow;">systematic outliers that emerge with scale</mark> and become problematic at specific model sizes.

Outliers in large models exhibit systematic and emergent properties, affecting the same dimensions across layers.

These outliers impact all layers in a transformer model once a certain scale is reached.

The emergence of outliers follows an exponential trend, leading to a phase shift-like effect at a certain scale.

Understanding and addressing this exponential trend is key to managing outliers in large models.

A novel approach was developed to identify and process these outliers in 16-bit while handling the rest in 8-bit, effectively maintaining efficiency while addressing the problem.

<mark style="color:green;">**Efficiency of 8-Bit Matrix Multiplication**</mark>

* By applying this method, 99.9% of weights are computed in 8-bit, with a small portion in 16-bit for outliers. This approach achieves performance equivalent to 16-bit computations while halving memory size.
* This makes large models like Llama 65B accessible on consumer hardware, significantly lowering the barrier to entry for working with such models.

<mark style="color:green;">**Few-Shot and Zero-Shot Performance**</mark>

* The few-shot performance of models using 8-bit methods is comparable to 16-bit models. Tim highlighted a strong correlation between zero-shot performance and perplexity in language models, indicating that complexity evaluations can reliably predict zero-shot performance.

<mark style="color:green;">**Understanding Outliers in Transformer Models**</mark>

* Outliers tend to be concentrated in specific columns of the input batch and are more prevalent in larger models. These outliers are crucial for attention mechanisms in transformers.
* They are context-independent, aiding the attention mechanism in focusing on specific values by providing predictable patterns for the model to cancel out unnecessary information.

<mark style="color:green;">**Trade-Offs in Activation Functions**</mark>

* Replacing traditional activation functions like softmax with more stable alternatives can increase stability but may lead to a drop in performance.
* This presents a research challenge in balancing stability with maintaining or enhancing model performance.

<mark style="color:green;">**Impact of Precision and Parameter Count on Model Efficiency**</mark>

* An interesting finding is that models with the same number of bits but different distributions of precision and parameter count (e.g., 8-bit with more parameters vs. 4-bit with fewer parameters) exhibit the same inference latency.
* This equivalence in performance is due to the nature of GPU computations, where memory loading is significantly more costly than computation. Therefore, the memory used during inference, rather than the computational complexity, often dictates the performance.

</details>


# Llama2 - Data Loading and Paths

With the model defined and fully configured, we  will now configure the datasets section.

First off, this configuration file is all about <mark style="color:yellow;">specifying the datasets you want to use for fine-tuning your model and where you would like the fine tuned model stored once training is complete.</mark>

You can provide one or more datasets under the <mark style="color:yellow;">**`datasets`**</mark> section. &#x20;

Each dataset can come from different sources, like a HuggingFace dataset repo, an S3 or GS path, or even a local JSON file.

For each dataset, you can choose the <mark style="color:yellow;">type of prompt</mark> you want to use during training.&#x20;

Axolotl supports a bunch of prompt types, like "alpaca", "sharegpt", "gpteacher", "oasst", and "reflection". You can specify the prompt type using the `type` parameter.

If you're using a file-based dataset, you can specify the data type using the <mark style="color:yellow;">**`ds_type`**</mark> parameter. It supports formats like JSON, Arrow, Parquet, text, and CSV. You can also provide the path to the source data files using the <mark style="color:yellow;">**`data_files`**</mark> parameter.

### <mark style="color:blue;">Basic datasets configuration</mark>

<pre class="language-yaml"><code class="lang-yaml"><strong>datasets:
</strong>  - path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
    type: alpaca
    ds_type: json
    data_files:
  - alpaca_data_cleaned.json
dataset_prepared_path:
val_set_size: 0.20
output_dir: ./llama-out
</code></pre>

This component of the configuration file tells the platform where the dataset it located, and how the dataset should be split up into <mark style="color:yellow;">training versus validation.</mark>

We will only be using a number of these parameters for our fine tuning.  The full number of configurations can be found here:  [<mark style="color:orange;">**data loading and processing configurations**</mark>](/axolotl-configuration-files/data-loading-and-processing)

### <mark style="color:blue;">All configurations</mark>

### <mark style="color:purple;">`path:`</mark>

#### This parameter indicates the <mark style="color:yellow;">path to a dataset that has been prepared for training</mark>.&#x20;

As highlighted, this can be a HuggingFace dataset repository, an AWS S3 storage path (s3://), or Google Cloud Path (gs\://) or "json" for a local dataset.

Here's an example of a Google Cloud Storage path:

```bash
gs://my-bucket/datasets/my_dataset.json
```

In this example, "my-bucket" is the name of the Google Cloud Storage bucket, and "datasets/my\_dataset.json" is the path to the specific file within that bucket.

To determine the source of your data files, right click on the folder containing the data files in VS Code and then select 'copy URL'.&#x20;

```yaml
path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
```

### <mark style="color:purple;">`type:`</mark>

Indicates the type of prompt to use for training, such as <mark style="color:yellow;">"alpaca", "sharegpt", "gpteacher", "oasst", or "reflection".</mark>

We will be using <mark style="color:yellow;">Alpaca in this example</mark>.  For a full explanation of each data 'type' please refer to this documentation: data type

```yaml
type: alpaca
```

### <mark style="color:purple;">`ds_type:`</mark>

(Optional) Specifies the data type when the <mark style="color:yellow;">**`path`**</mark> is a file, such as "json", "arrow", "parquet", "text", or "csv".

Huggingface datasets can come in a number of formats.  You can configure the data loading process to take into account the data type - csv, json, parquet.   <mark style="color:yellow;">We will be using JSON format.</mark>

```yaml
ds_type: json
```

### <mark style="color:purple;">`data_files:`</mark>

(Optional) Specifies the path to the source data files.

### <mark style="color:purple;">`dataset_prepared_path`</mark><mark style="color:purple;">:</mark>

Specifies the relative path where Axolotl attempts to save the prepared dataset as an arrow file after packing the data together. This allows subsequent training attempts to load faster.

```yaml
path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
```

### <mark style="color:purple;">`output_dir:`</mark>&#x20;

This is the path where the full fine tuned model will be saved to:

```bash
output_dir: ./completed-model
```

### <mark style="color:blue;">Advanced: Explanation of all other data related configuration options</mark>

### <mark style="color:purple;">`shards`</mark><mark style="color:purple;">:</mark>&#x20;

Axolotl allows you to split your data into shards, which can be handy for large datasets. You can specify the number of shards using the <mark style="color:yellow;">**`shards`**</mark> parameter.

### <mark style="color:purple;">`name`</mark><mark style="color:purple;">:</mark>&#x20;

If you're using a dataset from a repository, you can give it a name using the <mark style="color:yellow;">**`name`**</mark> parameter. This can help you keep track of which dataset configuration you're using.

### <mark style="color:purple;">`train_on_split`</mark><mark style="color:purple;">:</mark>&#x20;

By default, Axolotl uses the "train" split for training, but you can specify a different split using the `train_on_split` parameter.

### <mark style="color:purple;">`conversation`</mark><mark style="color:purple;">:</mark>&#x20;

If you're using the "sharegpt" prompt type, you can specify the conversation type using the <mark style="color:yellow;">**`conversation`**</mark> parameter. Axolotl uses the FastChat library for conversations, and you can find the available conversation types in the FastChat documentation.

{% embed url="<https://github.com/lm-sys/FastChat/tree/main>" %}

### <mark style="color:purple;">`field_human`</mark> <mark style="color:purple;"></mark><mark style="color:purple;">and</mark> <mark style="color:purple;"></mark><mark style="color:purple;">`field_model`</mark><mark style="color:purple;">: (optional)</mark>

You can also specify the keys to use for the human and assistant roles in the conversation using the <mark style="color:yellow;">`field_human`</mark> and `f`<mark style="color:yellow;">`ield_model`</mark> parameters.

### <mark style="color:purple;">`roles`</mark><mark style="color:purple;">:</mark>&#x20;

If your dataset has additional keys that you want to use as input or output roles, you can specify them under the `roles` section. The `input` role is used for masking, and the `output` role is used for generation.

### <mark style="color:purple;">`input`</mark><mark style="color:purple;">:</mark>

&#x20;(Optional) A list of keys to be masked based on <mark style="color:yellow;">**`train_on_input`**</mark><mark style="color:yellow;">**.**</mark>

### <mark style="color:purple;">`output`</mark><mark style="color:purple;">:</mark>&#x20;

(Optional) A list of keys to be used as output.

### <mark style="color:blue;">Custom user instruction prompt</mark>

Allows defining a custom prompt for each dataset.

### <mark style="color:purple;">`system_prompt`</mark><mark style="color:purple;">:</mark>&#x20;

(Optional) Specifies the system prompt.

### <mark style="color:purple;">`system_format`</mark><mark style="color:blue;">:</mark>

&#x20;(Optional) Specifies the format of the system prompt, with "{system}" as a placeholder.

### <mark style="color:purple;">`field_system`</mark><mark style="color:purple;">,</mark> <mark style="color:purple;"></mark><mark style="color:purple;">`field_instruction`</mark><mark style="color:purple;">,</mark> <mark style="color:purple;"></mark><mark style="color:purple;">`field_input`</mark><mark style="color:purple;">,</mark> <mark style="color:purple;"></mark><mark style="color:purple;">`field_output`</mark><mark style="color:purple;">:</mark>&#x20;

(Optional) Specify the column names for the respective fields in the dataset.

### <mark style="color:purple;">`format`</mark><mark style="color:purple;">:</mark>&#x20;

Specifies the format of the prompt, which can include placeholders for instruction and input.

### <mark style="color:purple;">`no_input_format`</mark><mark style="color:blue;">:</mark>

&#x20;Specifies the format of the prompt when there is no input, excluding the "{input}" placeholder.

1. <mark style="color:blue;">**`field`**</mark><mark style="color:blue;">**:**</mark> (Optional) For "completion" datasets only, specifies the field to use instead of the "text" column.
2. <mark style="color:blue;">**`shuffle_merged_datasets`**</mark><mark style="color:blue;">**:**</mark> (Optional) By default, Axolotl shuffles the merged datasets, but you can disable this behavior by setting <mark style="color:yellow;">**`shuffle_merged_datasets`**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**to**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`false`**</mark>.

### <mark style="color:blue;">Datasets for Evaluation</mark>

### <mark style="color:purple;">`test_datasets`</mark>

A list of one or more datasets used for evaluating the model.

* <mark style="color:blue;">**`path`**</mark><mark style="color:blue;">**:**</mark> Specifies the path to the test dataset file.
* <mark style="color:blue;">**`ds_type`**</mark><mark style="color:blue;">**:**</mark> Specifies the data type of the test dataset, such as "json".
* <mark style="color:blue;">**`split`**</mark><mark style="color:blue;">**:**</mark> Specifies the split to use for the test dataset. For "json" datasets, the default split is called "train".
* <mark style="color:blue;">**`type`**</mark><mark style="color:blue;">**:**</mark> Specifies the type of the test dataset, such as "completion".
* <mark style="color:blue;">**`data_files`**</mark><mark style="color:blue;">**:**</mark> Specifies the list of data files for the test dataset.

Note: You can use either <mark style="color:blue;">**`test_datasets`**</mark> or <mark style="color:blue;">**`val_set_size`**</mark>, but not both.

### <mark style="color:purple;">`dataset_prepared_path`</mark><mark style="color:purple;">:</mark>

Specifies the relative path where Axolotl attempts to save the prepared dataset as an arrow file after packing the data together. This allows subsequent training attempts to load faster.

### <mark style="color:purple;">`push_dataset_to_hub`</mark>

If you want to push the prepared dataset to the Hugging Face Hub, you can specify the repository path using the <mark style="color:yellow;">**`push_dataset_to_hub`**</mark> parameter.

### <mark style="color:purple;">`hub_model_id`</mark>

You can also push the fine-tuned model checkpoints to the Hugging Face Hub by specifying the private repository path using the <mark style="color:yellow;">**`hub_model_id`**</mark> parameter.

### <mark style="color:purple;">`hub_strategy`</mark>

The <mark style="color:yellow;">**`hub_strategy`**</mark> parameter allows you to control how the checkpoints are pushed to the hub.   Refer to the Hugging Face Trainer documentation for more information.


# Llama2 - Sequence Configuration

Before training or fine-tuning begins, the <mark style="color:yellow;">input data must be correctly formatted and prepared</mark>.

We will be configuring the:

Sequence Length

1. Sequence Length
2. Sample Packing
3. Padding to Sequence

```yaml
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
```

### <mark style="color:blue;">sequence\_len</mark>

This parameter sets the <mark style="color:yellow;">maximum allowable length for input sequences</mark>. Sequences longer than this length may be truncated or split during training.

&#x20;This limit is essential since transformers process input data in fixed-size blocks.

Sequences longer than this length are either truncated or split.&#x20;

Truncation means cutting off the part of the sequence that exceeds the limit, while splitting involves dividing a long sequence into smaller segments, each within the maximum length.

The choice of sequence length affects memory usage and computational requirements. Longer sequences can capture more context but require more computational resources.

Axolotl recommend the maximum length should typically be less than 2048 as most models have a token/context limit of 2048

### <mark style="color:blue;">sample\_packing</mark>

A flag that determines whether sample packing should be used.&#x20;

This is a method to optimise the training process by packing multiple shorter sequences into a single training example (batch).  It can increases training efficiency by reducing padding needs and better utilizing GPU memory. This technique is particularly useful when dealing with variable-length sequences.

**Implementation**: If set to <mark style="color:yellow;">**`true`**</mark>, sequences that are shorter than <mark style="color:yellow;">**`sequence_len`**</mark> are concatenated with others to form a packed batch. This process continues until the maximum sequence length is reached or no more sequences are available for packing.

### <mark style="color:blue;">pad\_to\_sequence\_len</mark>

This is a flag that controls whether sequences should be padded to match the specified sequence length.

This ensures that <mark style="color:yellow;">**all sequences in a batch are of the same length**</mark>, which is necessary for parallel processing by the model.   Shorter sequences are extended (padded) with special tokens (usually <mark style="color:yellow;">**`[PAD]`**</mark>) to reach the defined maximum sequence length.

Padding is a standard practice in training neural networks on sequences of varying lengths, but it can introduce additional computational overhead, especially with longer sequence lengths.

If set to "true," input sequences will be padded with special tokens to reach the maximum sequence length defined earlier.


# Llama2 - Lora Configuration

This is the default Lora Configuration. &#x20;

```yaml
adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
```

<mark style="color:yellow;">**`lora_r`**</mark>

This parameter determines the rank of the low-rank matrices used in LoRA.&#x20;

It controls the capacity and expressiveness of the LoRA adaptation. A higher value of <mark style="color:yellow;">**`lora_r`**</mark> allows for more fine-grained adaptations but also increases the number of trainable parameters.&#x20;

In this configuration, **`lora_r`** is set to 32.

<mark style="color:yellow;">**`lora_alpha`**</mark>

This parameter controls the scaling factor applied to the LoRA adaptation.&#x20;

It determines the contribution of the LoRA matrices to the original model's weights.

A higher value of **`lora_alpha`** gives more importance to the LoRA adaptation. In your configuration, **`lora_alpha`** is set to 16.

<mark style="color:yellow;">**`lora_dropout`**</mark>

This parameter specifies the dropout rate applied to the LoRA matrices during training.&#x20;

Dropout is a regularization technique that helps prevent overfitting.&#x20;

A value of 0.05 means that 5% of the elements in the LoRA matrices will be randomly set to zero during training.

<mark style="color:yellow;">**`lora_target_modules`**</mark>

This parameter specifies the names of the modules in the model architecture where LoRA will be applied.&#x20;

In this  configuration, LoRA is applied to the <mark style="color:yellow;">**`q_proj`**</mark> and <mark style="color:yellow;">**`v_proj`**</mark> modules, which are likely the query and value projection matrices in the attention mechanism.

We have commented out other potential target modules like <mark style="color:yellow;">**`k_proj`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`o_proj`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`gate_proj`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`down_proj`**</mark>, and <mark style="color:yellow;">**`up_proj`**</mark>.

<mark style="color:yellow;">**`lora_target_linear`**</mark>

This parameter is not set in your configuration.&#x20;

If set to `true`, LoRA will be applied to all linear modules in the model.

<mark style="color:yellow;">**`peft_layers_to_transform`**</mark>

This parameter allows you to specify the indices of the layers to which LoRA should be applied.

If not specified, LoRA will be applied to all layers by default.

<mark style="color:yellow;">**`lora_modules_to_save`**</mark>

This parameter is relevant when you have added new tokens to the tokenizer. In such cases, you may need to save certain LoRA modules that are aware of the new tokens.&#x20;

For LLaMA and Mistral models, you typically need to save <mark style="color:yellow;">**`embed_tokens`**</mark> and <mark style="color:yellow;">**`lm_head`**</mark> modules. <mark style="color:yellow;">**`embed_tokens`**</mark> converts tokens to embeddings, and <mark style="color:yellow;">**`lm_head`**</mark> converts embeddings to token probabilities.&#x20;

In this configuration, these modules are commented out.

<mark style="color:yellow;">**`lora_fan_in_fan_out`**</mark>

This parameter determines the structure of the LoRA matrices. If set to <mark style="color:yellow;">**`true`**</mark>, it uses a more efficient implementation of LoRA that reduces the number of additional parameters. In your configuration, it is set to <mark style="color:yellow;">**`false`**</mark>.

These hyperparameters allow you to control various aspects of the LoRA adaptation during fine-tuning.&#x20;

The optimal values for these hyperparameters may vary depending on your specific task, dataset, and model architecture.&#x20;

It's recommended to experiment with different configurations and monitor the performance to find the best settings for your use case.


# Llama2 - Logging

We will be using Weights and Biases to log and monitor our fine tuning runs.

If you are new to Weights and Biases, please take the time to review their quickstart guide:

{% embed url="<https://docs.wandb.ai/quickstart>" %}
Weights and Biases Quickstart
{% endembed %}

&#x20;A [run](https://docs.wandb.ai/guides/runs) is the basic building block of W\&B. You will use them often to [track metrics](https://docs.wandb.ai/guides/track), [create logs](https://docs.wandb.ai/guides/artifacts), [create jobs](https://docs.wandb.ai/guides/launch).

Once you have created a Weights and Biases account, login to  your Weights and Biases account by entering the following command at the command prompt:

```bash
wandb login
```

You will be asked to enter your Weights and Biases account details as per below:

<mark style="color:blue;">Username:</mark>&#x20;

<mark style="color:blue;">Password:</mark> &#x20;

<mark style="color:blue;">API Token:</mark>&#x20;

### <mark style="color:blue;">Configuration of Logging Component</mark>

<mark style="color:green;">Project</mark>

The project naming convention can be simplistic but informational.   Name your project whatever you like.

<mark style="color:green;">Entity</mark>

The <mark style="color:yellow;">entity</mark> is simply \<your-organisation>.

The configuration file should look like the below script:

```yaml
wandb_project: <your project name>
wandb_entity: <your-organisation> 
wandb_name: Set the name of your wandb run
wandb_run_id: Set the ID of your wandb run
```

<details>

<summary>Reference: <mark style="color:green;">The Weights and Biases Script in the Axolotl Library</mark></summary>

<mark style="color:yellow;">src/axolotl/utils/wandb\_.py</mark>

The script defines a function `setup_wandb_env_vars` that configures environment variables for wandb based on a given configuration. Here's a detailed explanation of what this function does:

<mark style="color:green;">**Module Description**</mark>

* The comment `"""Module for wandb utilities"""` indicates that this Python file is intended to provide utility functions for working with wandb.

<mark style="color:green;">**Import Statements**</mark>

* The script imports the `os` module, which provides functions for interacting with the operating system, including managing environment variables.
* It imports `DictDefault` from `axolotl.utils.dict`, which is a custom dictionary utility, likely providing some default behavior for dictionary operations.

<mark style="color:green;">**Function Definition**</mark>

`setup_wandb_env_vars(cfg: DictDefault)`:

* The function takes a single parameter `cfg`, which is expected to be an instance of `DictDefault`. This parameter likely contains configuration settings.

<mark style="color:green;">**Setting Environment Variables for wandb**</mark>

* The function iterates through all keys in the `cfg` dictionary.
* If a key starts with `"wandb_"`, the function retrieves its value.
* If the value is a non-empty string, the function sets an environment variable with the name of the key in uppercase and assigns it the value from `cfg`. For example, if `cfg` contains `{"wandb_api_key": "your_key"}`, it sets an environment variable `WANDB_API_KEY` with the value `"your_key"`.

<mark style="color:green;">**Enabling wandb Integration**</mark>

* The function checks if `cfg.wandb_project` exists and is a non-empty string.
* If `cfg.wandb_project` is set, `cfg.use_wandb` is set to `True`, and any existing `WANDB_DISABLED` environment variable is removed. This implies that wandb should be enabled and operational for the current session.
* If `cfg.wandb_project` is not set, the function sets the environment variable `WANDB_DISABLED` to `"true"`, effectively disabling wandb integration.

<mark style="color:green;">**Usage Context**</mark>

* This script is useful for dynamically configuring wandb based on a set of configuration parameters, especially in scenarios where different wandb settings are needed for different runs or experiments.
* It automates the process of setting up wandb environment variables, enabling or disabling wandb tracking based on the provided configuration.

In summary, the script is a utility for configuring wandb environment variables in a Python environment based on a provided configuration, enabling or disabling wandb tracking as needed. This is particularly useful in machine learning workflows where experiment tracking needs to be managed programmatically.

</details>


# Llama2 - Training Configuration

This is the default training configuration.  We will leave it as is for the time being.

```yaml
gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_torch
adam_beta2: 0.95
adam_epsilon: 0.00001
max_grad_norm: 1.0
lr_scheduler: cosine
learning_rate: 0.000003
```

### <mark style="color:blue;">Explanation of each configurations</mark>

<mark style="color:yellow;">**`gradient_accumulation_steps: 1`**</mark>

* Gradient accumulation is a technique used to simulate larger batch sizes without increasing memory usage.
* It allows the model to accumulate gradients over multiple forward passes before performing a single backward pass and updating the weights.
* Setting <mark style="color:yellow;">**`gradient_accumulation_steps`**</mark> to 1 means that the model will update its weights after every batch, effectively not using gradient accumulation.
* This is suitable when the available memory is sufficient to process the desired batch size without the need for accumulation.
* However, if memory is limited and you want to simulate a larger batch size, increasing this value can be beneficial.

<mark style="color:yellow;">**`micro_batch_size: 2`**</mark>

* The micro-batch size determines the number of samples processed in each forward pass of the model.
* In this case, setting <mark style="color:yellow;">**`micro_batch_size`**</mark> to 2 means that the model will process 2 samples at a time.
* Smaller micro-batch sizes can be useful when dealing with limited memory resources, as they require less memory per forward pass.
* However, smaller micro-batch sizes may result in slower training times and potentially noisier gradients.
* The optimal micro-batch size depends on the available hardware and the specific requirements of the training task.

<mark style="color:yellow;">**`num_epochs: 4`**</mark>

* The number of epochs represents the number of times the model will iterate over the entire training dataset.
* Setting <mark style="color:yellow;">**`num_epochs`**</mark> to 4 means that the model will go through the training data 4 times.
* Increasing the number of epochs can allow the model to learn more from the data and potentially improve its performance.
* However, training for too many epochs may lead to overfitting, where the model becomes too specialized to the training data and fails to generalize well to unseen data.
* The optimal number of epochs depends on factors such as the size and complexity of the dataset, the model architecture, and the learning rate.

<mark style="color:yellow;">**`optimizer: adamw_torch`**</mark>

* The optimizer is responsible for updating the model's weights based on the computed gradients.
* AdamW (Adam with Weight Decay) is an optimization algorithm that extends the original Adam optimizer by adding L2 regularization.
* AdamW is well-suited for training deep learning models and has been shown to perform well in various tasks.
* It adapts the learning rate for each parameter based on the historical gradients and incorporates weight decay to prevent overfitting.
* Using AdamW as the optimizer is a reasonable choice for training LLMs.

<mark style="color:yellow;">**`adam_beta2: 0.95`**</mark>

* <mark style="color:yellow;">**`adam_beta2`**</mark> is a hyperparameter of the AdamW optimizer that controls the exponential decay rate for the second moment estimates.
* It is typically set to a value close to 1 (e.g., 0.999) to give more importance to recent gradients.
* Setting <mark style="color:yellow;">**`adam_beta2`**</mark> to 0.95 means that the optimizer will give slightly less emphasis to recent gradients compared to the default value.
* The impact of this change may vary depending on the specific task and dataset.

<mark style="color:yellow;">**`adam_epsilon: 0.00001`**</mark>

* <mark style="color:yellow;">**`adam_epsilon`**</mark> is a small constant value used in the denominator of the AdamW optimizer's update rule to prevent division by zero.
* It helps to stabilize the optimization process and avoid numerical instability.
* The default value of <mark style="color:yellow;">**`adam_epsilon`**</mark> is usually 1e-8 or 1e-7.
* Setting it to 0.00001 (1e-5) is slightly larger than the typical default value, which may have a minor impact on the optimization process.

<mark style="color:yellow;">**`max_grad_norm: 1.0`**</mark>

* Gradient clipping is a technique used to prevent exploding gradients in deep neural networks.
* <mark style="color:yellow;">**`max_grad_norm`**</mark> sets the maximum L2 norm of the gradients. If the norm exceeds this value, the gradients are scaled down to meet the constraint.
* Setting `max_grad_norm` to 1.0 means that the gradients will be clipped if their L2 norm exceeds 1.0.
* Gradient clipping helps to stabilize the training process and prevents the gradients from becoming too large, which can lead to unstable or divergent behavior.
* The optimal value of <mark style="color:yellow;">**`max_grad_norm`**</mark> may vary depending on the model architecture and the specific training task.

<mark style="color:yellow;">**`lr_scheduler: cosine`**</mark>

* The learning rate scheduler determines how the learning rate changes over the course of training.
* The cosine scheduler, also known as cosine annealing, gradually decreases the learning rate following a cosine curve.
* It starts with a relatively high learning rate and slowly decreases it until it reaches a minimum value at the end of training.
* Cosine annealing can help the model converge more smoothly and potentially reach better local optima.
* It is a popular choice for learning rate scheduling and has been shown to be effective in various deep learning tasks.

<mark style="color:yellow;">**`learning_rate: 0.000003`**</mark>

* The learning rate determines the step size at which the model's weights are updated during training.
* Setting <mark style="color:yellow;">**`learning_rate`**</mark> to 0.000003 (3e-6) is a relatively small value.
* Smaller learning rates generally result in slower convergence but can lead to more stable and precise updates.
* The optimal learning rate depends on factors such as the model architecture, dataset complexity, and optimizer choice.
* It may be necessary to experiment with different learning rates to find the best value for a specific task.

The combination of AdamW optimizer with a cosine learning rate scheduler and gradient clipping can help stabilize the training process and potentially improve convergence.&#x20;

The micro-batch size and number of epochs should be adjusted based on the available hardware resources and the specific requirements of the training task. It's important to monitor the model's performance during training and make adjustments as needed to achieve the desired results.


# Llama2 - Data and Precision

These configurations allow you to control various aspects of the training process, such as data handling, precision settings, and hardware utilisation.&#x20;

```yaml
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: true
```

Let's go through each configuration and explain its purpose and implications:

#### <mark style="color:yellow;">`train_on_inputs: false`</mark>

* This configuration determines whether to include or mask out the human's prompt from the training labels.
* When set to <mark style="color:yellow;">**`false`**</mark>, the model will not train on the human's prompt, meaning that the prompt will be excluded from the training labels.
* In other words, the model will only learn from the desired output or response and not from the input prompt.
* This is useful when you want the model to generate responses based on the given prompts without explicitly learning to reproduce the prompts themselves.
* By masking out the human's prompt, the model can focus on learning the mapping between the prompt and the desired output.

#### <mark style="color:yellow;">`group_by_length: false`</mark>

* This configuration controls whether to group similarly sized data together to minimise padding during training.
* When set to <mark style="color:yellow;">**`false`**</mark>, the data will not be grouped by length and will be processed in the order it appears in the dataset.
* Grouping data by length can be beneficial when working with variable-length sequences, as it helps to reduce the amount of padding needed.
* Padding is the process of adding dummy tokens to shorter sequences to match the length of the longest sequence in a batch.
* By grouping similarly sized data together, you can minimize the amount of unnecessary padding, which can lead to more efficient memory usage and faster training.
* However, enabling <mark style="color:yellow;">**`group_by_length`**</mark> may result in slower data loading and preprocessing, as it requires downloading and sorting the entire dataset before training.
* It's also worth noting that when <mark style="color:yellow;">**`group_by_length`**</mark> is enabled, the training loss may exhibit an oscillating pattern due to the reordering of the data.

#### <mark style="color:yellow;">`bf16: auto`</mark>

* This configuration relates to the use of BFloat16 (BF16) precision during training.
* BFloat16 is a 16-bit floating-point format that offers a wider dynamic range compared to the more common FP16 (Half-precision) format.
* When set to <mark style="color:yellow;">**`auto`**</mark>, the framework will automatically determine whether to use BF16 based on the available hardware and software support.
* If the hardware (e.g., GPU) and software (e.g., PyTorch version) support BF16, it will be used for training.
* BF16 can provide a good balance between computational efficiency and numeric precision, potentially leading to faster training times while maintaining model accuracy.
* However, the actual performance gains may vary depending on the specific hardware and model architecture.

#### <mark style="color:yellow;">`fp16:`</mark>

* This configuration is related to the use of FP16 (Half-precision) during training, but in the provided configuration, it is left empty.
* FP16 is a 16-bit floating-point format that offers reduced precision compared to the standard FP32 (Single-precision) format.
* Using FP16 can help to reduce memory usage and accelerate training on certain hardware (e.g., NVIDIA GPUs with Tensor Cores).
* However, the empty value suggests that FP16 is not being explicitly enabled or configured in this case.

#### <mark style="color:yellow;">`tf32: true`</mark>

* This configuration is specific to NVIDIA GPUs and relates to the use of TensorFloat-32 (TF32) precision.
* TF32 is a 19-bit floating-point format that is used by default on NVIDIA Ampere architecture GPUs (e.g., NVIDIA A100) for certain operations, such as matrix multiplications and convolutions.
* When set to `true`, TF32 will be used for supported operations on compatible hardware.
* TF32 offers a balance between performance and precision, providing faster computation compared to FP32 while maintaining similar accuracy.
* Enabling TF32 can lead to improved training speeds on NVIDIA Ampere GPUs without significant impact on model quality.


# Llama2 - Optimisations

These configuration options offer various techniques and optimisations to improve the training process&#x20;

<mark style="color:blue;">**Gradient checkpointing**</mark> and <mark style="color:blue;">**Flash Attention**</mark> focus on memory efficiency and computational speed, while early stopping and resuming from checkpoints are useful for preventing overfitting and managing the training workflow.&#x20;

Distributed training with <mark style="color:yellow;">**`local_rank`**</mark> enables parallel processing across multiple devices, and adjusting the logging frequency with <mark style="color:yellow;">**`logging_steps`**</mark> helps in monitoring the training progress.

We will use the generic configurations provided by Axolotl.

```yaml
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: True
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
```

### <mark style="color:blue;">Below is an analysis of each of the configurations</mark>

<mark style="color:yellow;">**`gradient_checkpointing: true`**</mark>

* Gradient checkpointing is a technique used to reduce the memory usage during training by trading off computation time.
* Instead of storing all the intermediate activations for the backward pass, gradient checkpointing selectively stores a subset of activations and recomputes the others when needed.
* Setting <mark style="color:yellow;">**`gradient_checkpointing`**</mark> to <mark style="color:yellow;">**`true`**</mark> enables this feature, which can be particularly beneficial when training large models with limited memory.
* By enabling gradient checkpointing, you can potentially train larger models or use larger batch sizes without running out of memory.
* However, it's important to note that gradient checkpointing introduces additional computational overhead, as the model needs to recompute the activations during the backward pass.
* The decision to use gradient checkpointing depends on the available memory and the trade-off between memory usage and training speed.

<mark style="color:yellow;">**`gradient_checkpointing_kwargs:`**</mark>

* This configuration allows you to specify additional keyword arguments for gradient checkpointing.
* In the provided example, <mark style="color:yellow;">**`use_reentrant: True`**</mark> is specified as a keyword argument.
* The <mark style="color:yellow;">**`use_reentrant`**</mark> flag is related to the implementation of gradient checkpointing in PyTorch.
* When set to <mark style="color:yellow;">**`True`**</mark>, it enables the use of reentrant autograd functions, which can provide additional memory savings during gradient checkpointing.
* However, the specific behavior and impact of this flag may depend on the PyTorch version and the model architecture.

<mark style="color:yellow;">**`early_stopping_patience:`**</mark>

* Early stopping is a technique used to prevent overfitting and improve generalization performance.
* It monitors a validation metric (e.g., validation loss or accuracy) during training and stops the training process if the metric does not improve for a specified number of iterations (patience).
* The <mark style="color:yellow;">**`early_stopping_patience`**</mark> configuration allows you to set the number of iterations to wait before early stopping is triggered.
* For example, if <mark style="color:yellow;">**`early_stopping_patience`**</mark> is set to 3, training will stop if the validation metric does not improve for 3 consecutive iterations.
* Early stopping helps to avoid wasting computational resources on training iterations that do not lead to further improvements and can help prevent the model from overfitting to the training data.

<mark style="color:yellow;">**`resume_from_checkpoint:`**</mark>

* This configuration allows you to resume training from a specific checkpoint.
* By specifying a checkpoint directory or file path, you can load the model state, optimizer state, and other necessary information to continue training from where it left off.
* Resuming from a checkpoint can be useful in various scenarios, such as when training is interrupted due to system failures, when you want to fine-tune a pre-trained model, or when you want to experiment with different hyperparameters starting from a previously trained model.
* It saves time and resources by avoiding the need to start training from scratch.

<mark style="color:yellow;">**`local_rank:`**</mark>

* The <mark style="color:yellow;">**`local_rank`**</mark> configuration is related to distributed training, specifically when using techniques like Data Parallel or Distributed Data Parallel.
* In distributed training, multiple GPUs or machines are used to parallelise the training process and speed up computations.
* The <mark style="color:yellow;">**`local_rank`**</mark> represents the unique identifier of a process within a distributed training setup.
* It is typically used to determine the device placement and communication patterns among the processes.
* When using distributed training frameworks like PyTorch's <mark style="color:yellow;">**`DistributedDataParallel`**</mark>, the <mark style="color:yellow;">**`local_rank`**</mark> is automatically set by the framework.

<mark style="color:yellow;">**`logging_steps: 1`**</mark>

* The <mark style="color:yellow;">**`logging_steps`**</mark> configuration determines the frequency at which training logs and metrics are recorded.
* In this case, setting <mark style="color:yellow;">**`logging_steps`**</mark> to 1 means that logs will be generated after every training step.
* Logging can include information such as the current training loss, learning rate, elapsed time, and other relevant metrics.
* More frequent logging can be useful for monitoring the training progress and identifying any potential issues early on.
* However, generating logs after every step can also introduce overhead and slow down the training process, especially for large datasets or long training runs.
* The logging frequency should be adjusted based on the specific needs and the scale of the training task.

<mark style="color:yellow;">**`xformers_attention`**</mark>

* The `xformers_attention` configuration is related to the use of the XFormers library, which provides optimized attention implementations for transformers.
* XFormers offers various attention mechanisms, such as memory-efficient attention, that can speed up training and reduce memory usage compared to the standard attention implementation in PyTorch.
* Setting <mark style="color:yellow;">**`xformers_attention`**</mark> to a specific value (not provided in the given configuration) would enable the use of XFormers attention in the model.
* The specific attention mechanism and its parameters would depend on the value provided for <mark style="color:yellow;">**`xformers_attention`**</mark>.
* Using XFormers attention can be beneficial for training large models or when dealing with long sequences, as it can provide computational and memory efficiency improvements.

<mark style="color:yellow;">**`flash_attention: true`**</mark>

* Flash Attention is a highly optimised attention implementation that can significantly speed up the training of transformers.
* It is designed to be memory-efficient and can handle large sequence lengths and batch sizes.
* Setting <mark style="color:yellow;">**`flash_attention`**</mark> to <mark style="color:yellow;">**`true`**</mark> enables the use of Flash Attention in the model.
* Flash Attention can provide substantial performance improvements, especially for models with a large number of attention heads and long sequences.
* It achieves this by using techniques like kernel fusion, memory optimization, and efficient parallelization.
* Enabling Flash Attention can help reduce training time and allow for training larger models or using larger batch sizes.


# Llama2 - Extra Hyperparameters

This is the balance of the configuration file for the training of Phi 2.0.   We provide a full explanation of each of the configurations below.

```yaml
warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
resize_token_embeddings_to_32x: true
special_tokens:
  pad_token: "<|endoftext|>"
```

### <mark style="color:blue;">**Training Hyperparameters**</mark>

### <mark style="color:purple;">`warmup_steps: 100`</mark>

Warm-up steps are a crucial part of the learning rate scheduling.&#x20;

Over the first 100 training steps, the learning rate incrementally increases to its target value. This gradual increase helps in stabilizing the training early on, preventing the model from making too large updates too quickly.

### <mark style="color:purple;">`evals_per_epoch: 4`</mark>

This setting determines the frequency of evaluations within each training epoch.&#x20;

With a value of 4, the model will be evaluated four times per epoch, providing regular feedback on its performance. Frequent evaluations help in monitoring the model's progress and ensuring it is learning as expected.

### <mark style="color:purple;">`saves_per_epoch: 1`</mark>

To safeguard your training progress, the model's state is saved once every epoch. This checkpointing allows you to resume training from the last saved state in case of interruptions and also provides opportunities for model fine-tuning.

### <mark style="color:blue;">**Deepspeed and FSDP**</mark>

### <mark style="color:purple;">`deepspeed:`</mark>

DeepSpeed integration offers advanced optimisations for accelerating training and reducing memory consumption. Configuring DeepSpeed enhances training efficiency, particularly for large-scale models, by optimizing computational resources and parallelizing the workload.

### <mark style="color:purple;">`fsdp_config:`</mark>

Fully Sharded Data Parallel (FSDP) is a technique to reduce memory consumption and increase the scale of distributed training. The `fsdp_config` allows you to customise FSDP's behavior, optimizing memory usage and computational efficiency across multiple devices.

### <mark style="color:purple;">`weight decay: 1`</mark>

Weight decay is a regularization technique to prevent overfitting by penalising large weights.&#x20;

A weight decay factor of 0.1 helps in moderating the update of weights, encouraging the model to learn more general features rather than overly fitting to the training data.

### <mark style="color:blue;">**Special Tokens and Token Embeddings**</mark>

### <mark style="color:purple;">`resize_token_embeddings_to_32x:`</mark>

This configuration enables the resizing of token embeddings, a feature particularly useful for adapting the model to different vocabulary sizes efficiently.&#x20;

Resizing to 32 times the original size allows for more flexible and scalable model architecture.

### <mark style="color:purple;">`pad_token`</mark>

Special tokens play a pivotal role in how the model processes and understands text. The padding token (`pad_token`) is used to fill out sequences to a uniform length, ensuring consistent input size for the model.


# Llama2- All Configurations

This is a summary of the entire configuration file for the fine tuning run of Llama2

```yaml
base_model: NousResearch/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer

load_in_8bit: true
load_in_4bit: false
strict: false

datasets:
  - path: mhenrichsen/alpaca_2k_test
    type: alpaca
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./lora-out

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention:

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
```


# Llama2 - Training Configuration

This is the default training configuration.  We will leave it as is for the time being.

```yaml
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
```


# Llama2 - Preprocessing

To execute the training run, we will first <mark style="color:yellow;">preprocess the dataset.</mark>&#x20;

Axolotl allows us to optionally <mark style="color:blue;">**pre-tokenize dataset**</mark> before finetuning. This is recommended for large datasets.&#x20;

### <mark style="color:blue;">Populate the config.yaml file</mark>

To execute the preprocessing, we have to make sure the <mark style="color:yellow;">datasets component</mark> of the config.yaml is correctly populated. &#x20;

You will need to <mark style="color:yellow;">**direct Axolotl to where your dataset is**</mark>.

To do this, find the path to your dataset in VS Code by right clicking on the dataset and asking for the <mark style="color:yellow;">relative path.</mark>  Then enter it into the config YAML file below which is located at:

### *your directory*/<mark style="color:blue;">axolotl</mark>/<mark style="color:green;">examples</mark>/<mark style="color:purple;">llama-2</mark>/<mark style="color:yellow;">lora.yml</mark>

```bash
your directory/axolotl/examples/llama-2/lora.yml
```

When you have located the file, then <mark style="color:yellow;">populate the configuration file</mark> below.

To do this, find the <mark style="color:yellow;">path to your dataset</mark> in VS Code by right clicking on the dataset and asking for the relative URL.  Use this in the command below.  <mark style="color:yellow;">Remember to include the file name.</mark>

```yaml
datasets:
  - path: datasets/alpaca-cleaned/alpaca_data_cleaned.json
    type: alpaca
    ds_type: json
    data_files:
    - alpaca_data_cleaned.json
dataset_prepared_path:
val_set_size: 0.20
output_dir: ./llama-out
```

The data set type (ds\_type) is JSON, and we have arbitrarily <mark style="color:yellow;">set the output directory as ./llama-out</mark>

For a refresher of what the alpaca-cleaned dataset contains:

<details>

<summary><mark style="color:green;">Alpaca Cleaned Dataset</mark></summary>

The Alpaca dataset is a curated and cleaned version of an original dataset released by Stanford, specifically <mark style="color:yellow;">designed for instruction-tuning of language models</mark>. The analysis of the dataset's characteristics and the improvements made over the original version can be summarized as follows:

#### Dataset Description and Improvements

1. **Issues Addressed**:
   * **Hallucinations**: The original dataset included instructions that led to irrelevant or fabricated responses by the model, particularly in cases where the input was a URL or an ambiguous prompt. These have been cleaned to prevent such issues.
   * **Merged Instructions**: Instances where multiple instructions were combined have been separated for clarity and precision.
   * **Empty Outputs**: Entries with no outputs have been addressed.
   * **Missing Code Examples**: The cleaned dataset ensures that code examples are included where necessary.
   * **Instructions for Image Generation**: Removed or modified instructions that required image generation, as this is not feasible for a text-based model.
   * **N/A Outputs and Inconsistent Input Fields**: These issues have been standardized or corrected.
   * **Incorrect Answers**: The dataset has been reviewed for accuracy, especially in areas like math problems where the original dataset had a high error rate.
   * **Unclear Instructions**: Ambiguous instructions have been clarified or rewritten.
   * **Extraneous Characters**: Removal of unnecessary escape and control characters.
2. **Original Alpaca Dataset Summary**:
   * The <mark style="color:yellow;">original dataset, Alpaca, contains 52,000 instruction examples</mark> created using OpenAI's `text-davinci-003` engine.
   * It was designed for instruction tuning, making language models better at following specific instructions.
   * The dataset was generated more cost-effectively and is more diverse compared to its predecessors.
3. **Dataset Structure**:
   * The dataset consists of fields like `instruction`, `input`, `output`, and `text`, which combine these elements in a format suitable for model fine-tuning.
   * The `input` field is optional and used when context is necessary for the instruction.
4. **Data Splits**:
   * <mark style="color:yellow;">The dataset contains 52,002 training examples.</mark>
5. **Usage and Applications**:
   * It is intended for instruction training of pre-trained language models, particularly in the text-generation domain.
6. **Languages**:
   * The dataset is exclusively in English.

</details>

Once you have configured the dataset component of the YAML file, then execute the following command which uses the <mark style="color:yellow;">preprocess.py script</mark> within the Axolotl library. &#x20;

{% code fullWidth="false" %}

```bash
python -m axolotl.cli.preprocess examples/llama-2/lora.yml
```

{% endcode %}

The output should indicate success:

```bash
Success! Preprocessed data path: `dataset_prepared_path: ./llama-data`
```

If you have not set a dataset\_prepared\_path in the configuration file, it will default to:

"preprocess CLI called without dataset\_prepared\_path set, using default path: <mark style="color:yellow;">last\_run\_prepared"</mark>

For an <mark style="color:yellow;">analysis of the output</mark> relating to the dataset preprocessing, see below:

<details>

<summary><mark style="color:green;">Analysis of Output</mark></summary>

The numbers:

1. **PID (Process ID)**: `6779`
   * This is the ID of the process running the script.
2. **Token IDs**:
   * End Of Sentence (EOS) Token ID: `2`
   * Beginning Of Sentence (BOS) Token ID: `1`
   * Padding (PAD) Token ID: `0`
   * Unknown (UNK) Token ID: `0`
3. **Data Downloading and Processing**:
   * Number of data files downloaded: `1`
   * Download speed: `12557.80 items/second`
   * Number of data files extracted: `1`
   * Extraction speed: `214.70 items/second`
4. **Dataset Generation**:
   * Number of examples in the train split: `51760`
   * Generation speed: `85351.57 examples/second`
5. **Mapping and Filtering**:
   * Number of examples processed in mapping (num\_proc=12): `51760`
   * Processing speed in mapping: `6035.04 examples/second`
   * Number of examples processed in filtering (num\_proc=12): `51760`
   * Filtering speed: `41994.96 examples/second`
   * Number of examples processed in the second mapping (num\_proc=12): `51760`
   * Second mapping speed: `30435.07 examples/second`
6. **Token Counts**:
   * Total number of tokens: `12104896`
   * Total number of supervised tokens: `8475133`
7. **Efficiency Estimates and Data Loading**:
   * Packing efficiency estimate: `1.0`
   * Total number of tokens per device: `12104896`
   * Data loader length: `1461`
   * Sample packing efficiency estimate across ranks: `0.9798729691644562`
   * Sample packing efficiency estimate: `0.98`
   * Total number of steps: `5844`
8. **Time and Date Information**:
   * Date and time of the log entries: `2023-12-06`
   * Times of various log entries: `05:24:22,428`, `05:24:22,688`, `05:24:32,282`, `05:24:36,028`, `05:24:36,393`, `05:24:41,028`, `05:24:41,029`, `05:24:41,037`
   * Total execution time: `26 seconds`
   * Time when the command prompt was ready again: `05:24:42`

</details>

If you would like information on how the preprocess.py script works you can view the code at   <mark style="color:purple;">axolotl</mark>/<mark style="color:green;">src</mark>/<mark style="color:blue;">axolotl</mark>/<mark style="color:yellow;">cli.</mark> &#x20;

Alternatively, take the time to read our analysis of the scripts, and the output it produced:

<details>

<summary><mark style="color:green;">Axolotl preprocess.py -</mark> <mark style="color:yellow;">analysis of script</mark> <mark style="color:green;">and</mark> <mark style="color:yellow;">command output</mark></summary>

The provided `preprocess.py` script is a <mark style="color:yellow;">command-line interface (CLI) tool</mark> for preprocessing datasets in the context of training models using the Axolotl platform.&#x20;

1. <mark style="color:blue;">**Imports and Logger Setup**</mark><mark style="color:blue;">:</mark>
   * The script imports necessary modules such as <mark style="color:yellow;">`logging`</mark><mark style="color:yellow;">,</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`pathlib.Path`</mark> for file path operations, <mark style="color:yellow;">`fire`</mark> for the CLI, and several modules from <mark style="color:yellow;">`transformers`</mark> and <mark style="color:yellow;">`axolotl`</mark>.
   * <mark style="color:yellow;">`colorama`</mark> is used for colored console output, enhancing readability.
   * A logger `LOG` is set up for logging purposes, using the `logging` module.
2. <mark style="color:green;">**do\_cli Function**</mark><mark style="color:green;">:</mark>
   * <mark style="color:blue;">**Load Configurations**</mark><mark style="color:blue;">:</mark>   The main function <mark style="color:green;">`do_cli`</mark> takes a `config` argument (with a <mark style="color:yellow;">default path of</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`"examples/"`</mark>) and <mark style="color:yellow;">`**kwargs`</mark> for additional arguments.  These configurations include essential details about the dataset, model, and training setup. This step is crucial as it sets up the parameters for how the dataset will be processed and how the model training will proceed.
   * The function <mark style="color:yellow;">`load_datasets`</mark> is called with the loaded configuration and parsed command-line arguments. This function is responsible for the heavy lifting of loading the dataset (which could include downloading it if not already present locally) and preprocessing it according to the specifications in the configuration. This step is central to preparing the data for model training.
   * **ASCII Art and Configuration Loading**: It starts by printing ASCII art using <mark style="color:yellow;">`print_axolotl_text_art`</mark> for visual appeal. This is more for aesthetic purposes and has no impact on the functional aspects of the scrip
   * Then, <mark style="color:yellow;">it loads the configuration file</mark> using <mark style="color:yellow;">`load_cfg`</mark>, which sets up the parameters for dataset preprocessing.  &#x20;
   * &#x20;Initially, there's a deprecation warning from the <mark style="color:yellow;">`transformers.deepspeed`</mark> module. This suggests that in future versions of the Transformers library, you'll <mark style="color:yellow;">need to import DeepSpeed modules differently</mark>.&#x20;
   * **Accelerator and User Token Checks**: It ensures that the default accelerator configuration is set and validates the user token for authentication or API access.
   * **CLI Arguments Parsing**: The script uses <mark style="color:yellow;">`transformers.HfArgumentParser`</mark> to parse additional command-line arguments specific to preprocessing, defined in <mark style="color:yellow;">`PreprocessCliArgs`</mark>.
   * **Dataset Preparation Path Check**: It checks if <mark style="color:yellow;">`dataset_prepared_path`</mark> is set in the configuration.  If the <mark style="color:yellow;">`dataset_prepared_path`</mark> is not explicitly set in the configuration, the script issues a warning and *<mark style="color:yellow;">assigns a default pat</mark>*<mark style="color:yellow;">h (</mark><mark style="color:yellow;">`DEFAULT_DATASET_PREPARED_PATH`</mark><mark style="color:yellow;">).</mark> This path is where the script will store the preprocessed dataset, ensuring that there’s always a defined location for this data.
3. * **Load and Preprocess Datasets**: The <mark style="color:yellow;">`load_datasets`</mark> function is called with the loaded configuration and parsed CLI arguments, handling the dataset loading and preprocessing.
   * The script provides debug <mark style="color:yellow;">information about special tokens</mark> (End Of Sentence, Beginning Of Sentence, Padding, and Unknown) used by the tokenizer. This information is crucial for understanding how the model will interpret different types of tokens in the data.
4. **Logging Success**:
   * Upon successful completion, the <mark style="color:yellow;">script logs a message indicating the path</mark> where the preprocessed data is stored. This is done using the <mark style="color:yellow;">`LOG.info`</mark> method, and the message is colored green for visibility.
5. **Main Block**:
   * This block checks if the script is the main program being run <mark style="color:yellow;">(</mark><mark style="color:yellow;">`__name__ == "__main__"`</mark><mark style="color:yellow;">)</mark> and not a module imported in another script. If it is the main program, it uses <mark style="color:yellow;">`fire.Fire(do_cli)`</mark> to enable the script to be run from the command line, where <mark style="color:yellow;">`do_cli`</mark> is the main function being executed.
6. **General Observations**:
   * **User-Friendly**: The script is user-friendly, providing clear messages and using color coding for warnings and success messages.
   * **Flexibility**: It is flexible and allows customization through command-line arguments.
   * **Error Handling**: The script checks for potential issues like missing configuration parameters and handles them gracefully, providing default values and warnings.
   * **Logging**: Effective use of logging helps in tracking the script's execution and diagnosing issues if any arise.

OUTPUT

1. <mark style="color:green;">**Warnings and ASCII Art**</mark><mark style="color:green;">:</mark>
2. <mark style="color:green;">**Token Information**</mark><mark style="color:green;">:</mark> The script provides debug <mark style="color:yellow;">information about special tokens</mark> (End Of Sentence, Beginning Of Sentence, Padding, and Unknown) used by the tokenizer. This information is crucial for understanding how the model will interpret different types of tokens in the data.
3. <mark style="color:green;">**Data File Processing**</mark><mark style="color:green;">:</mark> The script downloads and extracts data files, then generates a training split with 51,760 examples. This is followed by mapping and filtering operations on the dataset, performed in parallel (noted by `num_proc=12`), which is an efficient way to handle large datasets.
4. <mark style="color:green;">**Dataset Merging and Saving**</mark><mark style="color:green;">:</mark> Post-processing, the datasets are merged and saved to disk. This is an essential step to ensure that the processed data is stored in a format ready for model training.
5. <mark style="color:green;">**Token Counts and Sample Packing**</mark><mark style="color:green;">:</mark> The script calculates the total number of tokens and supervised tokens. It also estimates the packing efficiency and total number of steps needed for training, which are critical for understanding how the data will be batched and fed into the model during training.
6. <mark style="color:green;">**Final Information and Path**</mark><mark style="color:green;">:</mark> Finally, the script concludes with a success message and provides the path to the preprocessed data. This is your confirmation that the preprocessing was successful and where the prepared data is stored.

</details>

This script for our Axolotl platform will be:&#x20;

<details>

<summary><mark style="color:green;">Configure this YAML File</mark></summary>

```yaml
base_model: NousResearch/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true

load_in_8bit: true
load_in_4bit: false
strict: false

datasets:
  - path: axolotl/datasets/alpaca-cleaned
    type: alpaca
dataset_prepared_path: ./llama-ou
val_set_size: 0.05
output_dir: ./lora-out

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
eval_steps: 0.05
eval_table_size:
eval_table_max_new_tokens: 128
save_steps:
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"

```

</details>

{% hint style="warning" %}
Flash Attention Dependency Issues
{% endhint %}

### <mark style="color:red;">Flash Attention Issues</mark>

We had some issue with Flash Attention dependencies.  For some reason the axolotl environment keep reverting to Pytorch 2.01 - Flash Attention needs Pytorch version 2.10

We executed this command in the axolotl library, and it upgraded Pytorch to 2.10.

```bash
pip install flash_attn -U --force-reinstall
```

<details>

<summary>Flash Attention Debugging</summary>

Flash Attention needs Pytorch 2.10 - not 2.0.  I had to force install Flash Attention to make it work:\
\
The error message you're encountering originates from running a Python script that attempts to preprocess a configuration file for a machine learning model using the <mark style="color:yellow;">`axolotl.cli.preprocess`</mark> module. The error is an <mark style="color:yellow;">`ImportError`</mark> associated with the `flash_attn_2_cuda` library, which is part of the `flash_attn` package. Let's break down the error message line by line:

1. **Command Executed**: <mark style="color:yellow;">`python -m axolotl.cli.preprocess your directory/axolotl/examples/llama-2/lora.yml`</mark>
   * This command attempts to run a Python module (<mark style="color:yellow;">`axolotl.cli.preprocess`</mark>) with a specific configuration file as its argument.
2. **Initial Traceback**:
   * The Python interpreter starts by <mark style="color:yellow;">trying to execute the module but encounters an issue in the process.</mark>
3. **Import Chain**:
   * The error arises in a chain of imports starting from your script's initial import statement. Python attempts to import various modules and packages needed for the script to run.
4. **Specific ImportError**:
   * The final error, <mark style="color:yellow;">`ImportError`</mark>, occurs when Python tries to import <mark style="color:yellow;">`flash_attn_2_cuda`</mark> from the <mark style="color:yellow;">`flash_attn`</mark> package.
   * The error message specifically states `undefined symbol: _ZN3c104cuda9SetDeviceEi`. This indicates a missing or incompatible symbol in the compiled CUDA extension.
5. <https://github.com/Dao-AILab/flash-attention/issues/620>

#### Troubleshooting Steps:

1. **Verify CUDA Installation**:
   * Ensure that CUDA is correctly installed on your system and is compatible with the versions required by `flash_attn`.
   * Check the CUDA version against the version required by <mark style="color:yellow;">`flash_attn`</mark>. If there's a mismatch, consider upgrading or downgrading CUDA.
2. **Check PyTorch and flash\_attn Compatibility**:
   * Verify that the installed PyTorch version is compatible with `flash_attn`. Incompatibilities between CUDA, PyTorch, and `flash_attn` can lead to such errors.
3. **Reinstall flash\_attn**:
   * Try reinstalling the `flash_attn` package. Sometimes, recompiling the package can resolve symbol mismatch errors.
   * Use the command <mark style="color:blue;">`pip install flash-attn --force-reinstall`</mark> to reinstall.
4. **Check Environment Variables**:
   * Ensure that your <mark style="color:yellow;">`LD_LIBRARY_PATH`</mark> environment variable includes the paths to the CUDA libraries.
5. **Inspect Python Environment**:
   * Verify that you are using the correct Python environment where all required dependencies are installed. Sometimes, conflicts in environments can cause such issues.
6. **Check for System Updates**:
   * Occasionally, system updates or changes to the compiler or related libraries can cause incompatibilities. Ensure your system is up to date.
7. **Consult Documentation or Community Forums**:
   * Check the documentation for `flash_attn` and related packages for any known issues or troubleshooting tips.
   * Seek help from relevant community forums or issue trackers for `flash_attn` and related projects.
8. **Test in a Clean Environment**:
   * If possible, try to run the script in a clean Python environment with only necessary packages installed. This can help rule out conflicts with other packages.

</details>

<details>

<summary><mark style="color:green;">Using the Upload File Function</mark></summary>

The <mark style="color:yellow;">`upload_file`</mark> function from the Hugging Face Hub library is used to upload files to a repository on the Hugging Face Hub. This function is quite versatile, supporting various parameters to customize the upload process. Here's a breakdown of the parameters and what they mean:

1. <mark style="color:blue;">**path\_or\_fileobj (str, Path, bytes, or IO)**</mark><mark style="color:blue;">:</mark> This is the source of the file you want to upload. It can be a <mark style="color:yellow;">path to a file on your local machine</mark>, a binary data stream, a file object, or a buffer.
2. <mark style="color:blue;">**path\_in\_repo (str)**</mark><mark style="color:blue;">:</mark> This specifies where in the repository the file should be placed. For example, if you want to place a file in a folder called 'checkpoints', you would use something like `"checkpoints/weights.bin"`.
3. <mark style="color:blue;">**repo\_id (str)**</mark><mark style="color:blue;">:</mark> The identifier for the repository to which you are uploading. This usually follows the format `"username/repository-name"`.
4. <mark style="color:blue;">**token (str, optional)**</mark><mark style="color:blue;">:</mark> Your authentication token for the Hugging Face Hub. If you have already logged in using `HfApi.login`, this will default to the stored token. If not provided, the function will attempt to use a stored token from a previous login.
5. <mark style="color:blue;">**repo\_type (str, optional)**</mark><mark style="color:blue;">:</mark> Indicates the type of repository. It can be `"dataset"`, `"space"`, or `"model"`. If you are uploading to a dataset or a space, specify accordingly. The default is `None`, which is interpreted as a model.
6. <mark style="color:blue;">**revision (str, optional)**</mark><mark style="color:blue;">:</mark> Specifies the git revision (like a branch name or commit hash) from which the commit should be made. The default is the head of the "main" branch.
7. <mark style="color:blue;">**commit\_message (str, optional)**</mark><mark style="color:blue;">:</mark> A short summary or title for the commit you are making. Think of this as the headline of your changes.
8. <mark style="color:blue;">**commit\_description (str, optional)**</mark><mark style="color:blue;">:</mark> A more detailed description of the changes you are committing.
9. <mark style="color:blue;">**create\_pr (boolean, optional)**</mark><mark style="color:blue;">:</mark> Determines whether to create a Pull Request for the commit. Defaults to `False`. If `True`, a PR will be created based on the specified `revision`.
10. <mark style="color:blue;">**parent\_commit (str, optional)**</mark><mark style="color:blue;">:</mark> The hash of the parent commit to which your changes will be added. This is used to ensure that you are committing to the correct version of the repository, especially useful in concurrent environments.
11. <mark style="color:blue;">**run\_as\_future (bool, optional)**</mark><mark style="color:blue;">:</mark> If set to `True`, the upload process will run in the background as a non-blocking action. This returns a `Future` object that can be used to check the status of the upload.

The `path_in_repo` parameter in the `api.upload_file` function from the `huggingface_hub` library specifies the destination path within the repository on the Hugging Face Hub where the file will be uploaded. This is essentially the relative path in the repository where the file will be placed.

In this example command:

```python
api.upload_file(
    path_or_fileobj="/path/to/trained_model_weights.bin",
    path_in_repo="model_weights.bin",
    repo_id="username/my_ml_model",
    repo_type="model"
)
```

* <mark style="color:yellow;">`path_or_fileobj="/path/to/trained_model_weights.bin"`</mark>: This is the path to the file on your local machine. It indicates where the file is currently stored in your local file system.
* <mark style="color:yellow;">`path_in_repo="model_weights.bin"`</mark>: This determines where within your Hugging Face Hub repository the file will be saved. In this case, you are instructing the function to upload your file directly to the root of the repository and <mark style="color:yellow;">name it</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`model_weights.bin`</mark>. If you wanted to place this file inside a folder within your repository, you would specify a path like `"`<mark style="color:yellow;">`folder_name/model_weights.bin`</mark>`"`.
* <mark style="color:yellow;">`repo_id="username/my_ml_model"`</mark>: This identifies the specific repository on the Hugging Face Hub where the file should be uploaded. It's a combination of your username (or organization name) and the repository name.
* <mark style="color:yellow;">`repo_type="model"`</mark><mark style="color:yellow;">:</mark> This indicates the type of repository you are uploading to. In this case, it's a model repository.

</details>


# Llama2 - Training

### <mark style="color:blue;">LLama2</mark>

To execute the main training script Axolotl provides this command to begin training Phi 2.0

```bash
accelerate launch -m axolotl.cli.train examples/llama-2/lora.yml
```

If you have not already done so, you will be asked to enter your Weights and Biases API Key.&#x20;

&#x20;Enter the key at the command line prompt:

```yaml
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

### <mark style="color:blue;">An analysis of the axolotl.clt.train module</mark>

<details>

<summary><mark style="color:green;">Analysis of train.py script</mark></summary>

The <mark style="color:yellow;">`train.py`</mark> script in the Axolotl platform is a Command Line Interface (CLI) tool designed for training machine learning models.&#x20;

This script is structured to provide a user-friendly interface for configuring and executing model training. Here's a detailed analysis:

#### <mark style="color:blue;">Script Structure and Functionality</mark>

<mark style="color:green;">**Imports and Logger Setup**</mark>

* Essential modules like `logging`, `pathlib.Path`, `fire`, and `transformers` are imported.
* The script sets up a logger `LOG` using the `logging` module for logging various events and statuses during the script's execution.

<mark style="color:green;">**do\_cli Function**</mark>

* <mark style="color:blue;">**Function Definition**</mark><mark style="color:blue;">:</mark> The <mark style="color:yellow;">`do_cli`</mark> function is the main entry point of the script. It accepts a <mark style="color:yellow;">`config`</mark> argument (with a default value pointing to an "examples" directory) and `**kwargs` for additional arguments.
* <mark style="color:blue;">**ASCII Art Display**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">`print_axolotl_text_art()`</mark> is called to display ASCII art, likely for aesthetic purposes.
* <mark style="color:blue;">**Configuration Loading**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">`load_cfg`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">l</mark>oads configuration details from the provided `config` path. These configurations are essential for setting up model training parameters.
* <mark style="color:blue;">**Accelerator and User Token Checks**</mark><mark style="color:blue;">:</mark> The script verifies the default configuration for the accelerator (such as a GPU) and checks the user token. These checks are crucial for ensuring that the hardware is correctly set up and the user is authenticated.
* <mark style="color:blue;">**CLI Arguments Parsing**</mark><mark style="color:blue;">:</mark> It uses <mark style="color:yellow;">`transformers.HfArgumentParser`</mark> to parse additional CLI arguments into data classes (<mark style="color:yellow;">`TrainerCliArgs`</mark>). This step allows for dynamic customization of training parameters via the command line.
* <mark style="color:blue;">**Dataset Loading**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">`load_datasets`</mark> is called with the parsed configuration and CLI arguments. This function is responsible for loading the dataset as per the configuration, which is a critical step in the training process.
* <mark style="color:blue;">**Model Training**</mark><mark style="color:blue;">:</mark> The <mark style="color:yellow;">`train`</mark> function is invoked with the loaded configuration, CLI arguments, and dataset metadata. This function likely encompasses the core logic for model training.

<mark style="color:green;">**Main Block**</mark>

* The script checks if it's being run as the main program <mark style="color:yellow;">(</mark><mark style="color:yellow;">`__name__ == "__main__"`</mark><mark style="color:yellow;">)</mark> and not as a module in another script. If it's the main program, it uses <mark style="color:yellow;">`fire.Fire(do_cli)`</mark> to execute the <mark style="color:yellow;">`do_cli`</mark> function, enabling the script to be interacted with from the command line.

</details>


# Llama3

### <mark style="color:blue;">Introduction</mark>

Meta Llama 3 is the next generation of Meta's open-source large language model (LLM) series.&#x20;

This model demonstrates exceptional performance on various industry benchmarks and offers new capabilities such as improved reasoning and code generation.&#x20;

Llama 3 models are available in 8B and 70B parameter sizes, with plans to release larger models up to 400B parameters in the coming months.

#### <mark style="color:green;">Model Architecture</mark>&#x20;

Decoder-only Transformer Llama 3 uses a standard <mark style="color:yellow;">decoder-only transformer architecture</mark>, with improvements over its predecessor, Llama 2.&#x20;

The model employs a tokenizer with a <mark style="color:yellow;">vocabulary of 128K tokens</mark>, enabling more efficient language encoding and substantially improved model performance.

#### <mark style="color:green;">Grouped Query Attention (GQA)</mark>

To enhance inference efficiency, Llama 3 models adopt Grouped Query Attention (GQA) across both the 8B and 70B sizes. The models are trained on sequences of 8,192 tokens, using a mask to prevent self-attention from crossing document boundaries.

#### <mark style="color:green;">Training Data</mark>

Llama 3 is pretrained on over <mark style="color:yellow;">15T tokens</mark> collected from publicly available sources.&#x20;

The training dataset is *<mark style="color:yellow;">**seven times larger than that used for Llama 2**</mark>*, with four times more code.&#x20;

Over 5% of the pretraining dataset consists of high-quality non-English data covering more than 30 languages, preparing for upcoming multilingual use cases.

#### <mark style="color:green;">Data Filtering</mark>&#x20;

To ensure the highest quality of training data, Meta developed data-filtering pipelines that include heuristic filters, NSFW filters, semantic deduplication approaches, and text classifiers to predict data quality.&#x20;

Llama 2 was used to generate training data for the text-quality classifiers powering Llama 3.

Scaling and Performance

#### <mark style="color:green;">Scaling Laws</mark>

Meta developed detailed scaling laws for downstream benchmark evaluations, enabling optimal data mix selection and informed decisions on training compute allocation.&#x20;

These scaling laws allow performance prediction for the largest models on key tasks before training, ensuring strong performance across various use cases and capabilities.

#### <mark style="color:green;">Parallelization</mark>

Llama 3's largest models combine data parallelization, model parallelization, and pipeline parallelization.&#x20;

The most efficient implementation achieves a compute utilization of over 400 TFLOPS per GPU when trained on 16K GPUs simultaneously.

#### <mark style="color:green;">Instruction Fine-Tuning</mark>

Llama 3's post-training approach combines supervised fine-tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct policy optimization (DPO).&#x20;

The quality of prompts used in SFT and preference rankings used in PPO and DPO significantly influences the performance of aligned models. Training on preference rankings enables the model to learn how to select the right answer in reasoning and coding tasks.

#### <mark style="color:green;">Deployment and Availability</mark>&#x20;

Llama 3 will soon be available on major platforms, including cloud providers, model API providers, and more.&#x20;

The improved tokenizer efficiency and GQA contribute to maintaining the inference efficiency of the 8B model on par with Llama 2 7B, despite having 1B more parameters.

#### <mark style="color:green;">Future Developments</mark>

Meta plans to release larger Llama 3 models with over 400B parameters in the coming months.&#x20;

These models will offer new capabilities, including multimodality, multilingual conversation, longer context windows, and stronger overall capabilities. A detailed research paper will be published once the training of Llama 3 is complete.


# Downloading the model

### <mark style="color:blue;">**Access HuggingFace Hub**</mark>

* Visit the [HuggingFace hub](https://huggingface.co/models).
* Use the search bar to look for "Llama v3" models.

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FZdZMW5lsfCKlrGbQ24oi%2Fimage.png?alt=media&amp;token=510872ae-f064-4f41-9da8-0e1f8adc5792" alt=""><figcaption><p>The full suite of Meta models</p></figcaption></figure>

### <mark style="color:blue;">**Identify the Correct Models**</mark>

* Click on the Meta Llama3 collection

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FSpbY6K9FvBwRER8xx9VC%2Fimage.png?alt=media&amp;token=09ad777b-9976-49a5-aa7e-e6b2d51f0638" alt=""><figcaption></figcaption></figure>

We are downloading Meta-Llama-3-8B.

{% hint style="info" %}
If you have not done already, you will have to get permission to download the Llama3 models.  It is a simple process.
{% endhint %}

We note that this model has not yet been converted to Huggingface weights.

* Models with 'hf' in their name are already converted to HuggingFace checkpoints. This means they are ready to use and require no further conversion.

### <mark style="color:blue;">Download the model into your model directory</mark>

First, we must connect to Huggingface.

If you have not done already, create a repository to store a Huggingface token

```bash
git config --global credential.helper store
```

Then connect to Huggingface

```bash
huggingface-cli login
```

When asked to enter the authentication code, use:

```bash
Your Huggingface token....
```

<details>

<summary><mark style="color:green;">What is a Huggingface User Token</mark></summary>

User Access Tokens on the Hugging Face platform serve as a secure method for authenticating applications and notebooks to access Hugging Face services.&#x20;

#### <mark style="color:green;">Purpose of User Access Tokens</mark>

* **Preferred Authentication Method**: They are the recommended way to authenticate an application or notebook to Hugging Face services.
* **Management**: Managed through the user's settings on the Hugging Face platform.

#### <mark style="color:green;">Scope and Roles</mark>

* **Read Role**: Tokens with this role provide read access to both public and private repositories that the user or their organization owns. They are suitable for tasks like <mark style="color:yellow;">downloading private models or performing inference.</mark>
* **Write Role**: In addition to read privileges, tokens with the write role can modify content in repositories where the user has write access. This includes creating or updating repository content, such as training or modifying a model card<mark style="color:yellow;">.</mark>

#### <mark style="color:green;">Managing User Access Tokens</mark>

* **Creation**: Access tokens are created in the user's settings under the Access Tokens tab.
* **Customization**: Users can select a role and name for each token.
* **Control**: Tokens can be deleted or refreshed for security purposes.

#### <mark style="color:green;">Usage of User Access Tokens</mark>

* **Versatility**: Can be used in various ways, including:
  * <mark style="color:blue;">As a replacement for a password for Git operations or basic authentication.</mark>
  * <mark style="color:blue;">As a bearer token for calling the Inference API.</mark>
  * Within Hugging Face Python libraries, like `transformers` or `datasets`, by passing the token for accessing private models or datasets.
* **Security Warning**: Users are cautioned to safeguard their tokens to prevent unauthorized access to their private repositories.

#### <mark style="color:green;">Best Practices</mark>

* **Separate Tokens for Different Uses**: Create distinct tokens for different applications or contexts (e.g., local machine, Colab notebook, custom servers).
* **Role Appropriateness**: Assign only the necessary role to each token. If only read access is needed, limit the token to the read role.
* **Token Management**: Regularly rotate and manage tokens to ensure security, especially if a token is suspected to be compromised.

</details>

When asked whether you want to add your Huggingface token  as a <mark style="color:yellow;">**git token credential,**</mark> say yes.  This should be the output:

```
Token is valid (permission: read).
Your token has been saved in your configured git credential helpers (store).
Your token has been saved to /home/paperspace/.cache/huggingface/token
Login successful
```

The install git large file storage so we can download the model (which is large)

```bash
git lfs install
```

If successful, the following output will be displayed in your terminal

```
Updated git hooks.
Git LFS initialized.
```

<details>

<summary><mark style="color:green;"><strong>What is Git large file storage</strong></mark></summary>

\
`Git Large File Storage (LFS)` is an extension for Git that addresses issues related to handling large files and binary data in Git repositories.

#### <mark style="color:green;">Why Git LFS is Necessary</mark>

1. **Limitations with Large Files in Git**:
   * Standard Git is excellent for handling text files (like code), which are typically small and benefit from Git's delta compression (storing only changes between versions). However, <mark style="color:yellow;">Git isn't optimized for large binary files (like images, videos, datasets, etc.)</mark><mark style="color:blue;">.</mark> These files can dramatically increase the size of the repository and degrade performance.
2. **Efficiency and Performance**:
   * Cloning and pulling from a repository with large files can be slow, consume a lot of bandwidth, and require significant storage space on every developer's machine. This inefficiency can hinder collaboration and development speed.
3. **Repository Bloat**:
   * In standard Git, every version of every file is stored in the repository's history. This means that even if a large file is deleted from the current working directory, its history still resides in the Git database, leading to a bloated repository.

#### <mark style="color:green;">How Git LFS Works</mark>

1. **Pointer Files**:
   * Git LFS replaces large files in your repository with tiny pointer files. When you commit a large file, Git LFS stores a reference (pointer) to that file in your repository, while the actual file data is stored in a separate server-side LFS storage.
2. **LFS Storage**:
   * The large file contents are stored on a remote server configured for LFS, typically alongside your Git repository hosting service (like GitHub, GitLab, Bitbucket, etc.). This storage is separate from your main Git repository storage.
3. **Version Tracking**:
   * Git LFS tracks versions of the large files separately. Every time you push or pull changes, Git LFS uploads or downloads the correct version of the large file from the LFS server, ensuring that you have the correct files in your working copy.
4. **Selective Download**:
   * When cloning or pulling a repository, Git LFS downloads only the versions of large files needed for your current commit, reducing the time and bandwidth compared to downloading the entire history of every file.
5. **Compatibility**:
   * Git LFS is compatible with existing Git services and workflows. It's an extension to Git, so you use the same Git commands. Repositories using Git LFS are still standard Git repos, ensuring backward compatibility.

</details>

Create a models directory

```bash
mkdir models
```

Move into the Axolotl models directory:

```bash
cd models
```

Enter the git clone command:

```bash
git clone https://huggingface.co/meta-llama/Meta-Llama-3-8B
```

The outcome should be new folder called LLama3-8b with all the appropriate files in it.

This is a screen cut from VS Code, showing all of the model files.

<figure><img src="https://148429626-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FcgmygEk0ifLuns7P9aMW%2Fuploads%2FtLBAYCaQxdlLQWkyjXGc%2Fimage.png?alt=media&amp;token=bb149a76-2190-49be-91bc-2e7f6b7acc19" alt=""><figcaption><p>Snippet from VS Code showing the downloaded model files</p></figcaption></figure>


# Analysis of model files

The provided file structure represents the various components and files of the Llama3 language model. Let's explain each file and how they relate to one another:

#### <mark style="color:green;">original</mark>

This folder  contains the original checkpoint of the Llama3 model. It serves as a backup or reference to the initial state of the model.

#### <mark style="color:green;">config.json</mark>

This file contains the configuration parameters for the Llama3 model. It defines the model architecture, hyperparameters, and other settings used during training and inference.

#### <mark style="color:green;">generation\_config.json</mark>

This file specifies the configuration for text generation using the Llama3 model. It includes settings like maximum sequence length, temperature, top-k sampling, and other generation-related parameters.

#### <mark style="color:green;">model-00001-of-00004.safetensors to model-00004-of-00004.safetensors</mark>

These files contain the actual weights and parameters of the Llama3 model.&#x20;

The model is divided into multiple files (in this case, four files) due to its large size.&#x20;

The ".safetensors" extension indicates that the files are stored in a memory-mapped format for efficient loading and inference.

#### <mark style="color:green;">model.safetensors.index.json</mark>

This file serves as an index for the ".safetensors" files.

It maps the model's parameters to their corresponding locations in the ".safetensors" files, allowing the model to be loaded efficiently during inference.

#### <mark style="color:green;">special\_tokens\_map.json</mark>

This file defines the mapping between special token names and their corresponding token IDs.&#x20;

Special tokens are used to represent specific entities or control sequences in the input and output of the model.

#### <mark style="color:green;">tokenizer.json</mark>

This file contains the vocabulary and mappings used by the tokenizer to convert input text into token IDs. It defines the mapping between words, subwords, or characters and their corresponding unique token IDs.

#### <mark style="color:green;">tokenizer\_config.json</mark>

This file specifies the configuration settings for the tokenizer.&#x20;

It defines the behavior and properties of the tokenizer, such as the special tokens, maximum sequence length, and input tensor names.

### <mark style="color:blue;">The files in this structure work together to define and use Llama3</mark>

* The model weights and parameters are stored in the ".safetensors" files, with the "model.safetensors.index.json" file serving as an index for efficient loading.
* The "config.json" and "generation\_config.json" files provide the necessary configuration settings for the model architecture and text generation.
* The "tokenizer.json" and "tokenizer\_config.json" files are used by the tokenizer to convert input text into token IDs and handle special tokens.
* The "special\_tokens\_map.json" file defines the mapping between special token names and their IDs.
* The "README.md", "LICENSE", and "USE\_POLICY.md" files provide documentation, licensing information, and usage guidelines for the model.

During inference, the model weights are loaded from the ".safetensors" files, and the configuration files are used to set up the model architecture and generation settings.&#x20;

The tokenizer, guided by the "tokenizer.json" and "tokenizer\_config.json" files, converts the input text into token IDs, which are then fed into the model for processing and generating outputs.


# Model Analysis - Configuration Parameters

### <mark style="color:blue;">This is an analysis of the config.json file</mark>

These configuration parameters define the architecture and hyperparameters of the Llama3 language model.&#x20;

The model uses a decoder-only transformer architecture with <mark style="color:blue;">**32 hidden layers**</mark>, each having <mark style="color:blue;">**32 attention heads**</mark> and a <mark style="color:blue;">**hidden size of 4096**</mark>.&#x20;

The model can <mark style="color:yellow;">**process sequences up to 8192 tokens long**</mark> and uses <mark style="color:yellow;">**SiLU activation**</mark> in the hidden layers.&#x20;

The weights are initialised with a range of 0.02, and the model uses <mark style="color:yellow;">**bfloat16 precision**</mark> for its weights and activations.&#x20;

The <mark style="color:yellow;">**vocabulary size is set to 128,256 tokens**</mark>, with specific IDs assigned for the beginning-of-sequence and end-of-sequence tokens.

<table><thead><tr><th width="194" align="center">Parameter</th><th width="139" align="center">Value</th><th align="center">Explanation</th></tr></thead><tbody><tr><td align="center">architectures</td><td align="center"><p> Llama</p><p>ForCausalLM</p></td><td align="center">Specifies the architecture class used for the language model. This class is designed for causal language modeling tasks, where the model predicts the next token based on the previous tokens.</td></tr><tr><td align="center">attention_bias</td><td align="center"><em>false</em></td><td align="center">Indicates that attention bias, which can be used to mask out certain positions or modify attention weights, is not used in the model.</td></tr><tr><td align="center">attention_dropout</td><td align="center">0.0</td><td align="center">The dropout probability for the attention layers. Dropout is a regularization technique that randomly sets some input units to 0 during training. In this case, dropout is not applied to the attention layers.</td></tr><tr><td align="center">bos_token_id</td><td align="center">128,000</td><td align="center">The ID of the special token used to indicate the beginning of a sequence. This token is added to the start of the input sequence.</td></tr><tr><td align="center">eos_token_id</td><td align="center">128,001</td><td align="center">The ID of the special token used to indicate the end of a sequence. This token is added to the end of the input sequence.</td></tr><tr><td align="center">hidden_act</td><td align="center">"silu"</td><td align="center">The activation function used in the hidden layers of the model. In this case, the Sigmoid Linear Unit (SiLU) function is used, which is defined as x * sigmoid(x).</td></tr><tr><td align="center">hidden_size</td><td align="center">4,096</td><td align="center">The dimensionality of the hidden states in the model. This determines the size of the vectors that the model uses to represent the input and output at each layer.</td></tr><tr><td align="center">initializer_range</td><td align="center">0.02</td><td align="center">The range used for initialising the model's weights. The weights are initialised randomly within this range to break symmetry and encourage diverse representations.</td></tr><tr><td align="center">intermediate_size</td><td align="center">14,336</td><td align="center">The dimensionality of the intermediate (feed-forward) layer in the model. This layer is applied after the attention layer and expands the hidden state size before projecting it back to the original hidden_size.</td></tr><tr><td align="center"><p>max_position</p><p>embeddings</p></td><td align="center">8,192</td><td align="center">The maximum sequence length that the model can process, in tokens. This limits the context size that the model can attend to.</td></tr><tr><td align="center">model_type</td><td align="center">"llama"</td><td align="center">Specifies the type of the language model. In this case, it is the "llama" model, which is a large language model developed by Meta.</td></tr><tr><td align="center">num_attention_heads</td><td align="center">32</td><td align="center">The number of attention heads in each attention layer. Attention heads allow the model to attend to different parts of the input sequence simultaneously, capturing different relationships and patterns.</td></tr><tr><td align="center">num_hidden_layers</td><td align="center">32</td><td align="center">The number of hidden layers (transformer blocks) in the model. Each layer consists of an attention mechanism followed by a feed-forward network.</td></tr><tr><td align="center"><p>num_key_value</p><p>heads</p></td><td align="center">8</td><td align="center">The number of key-value heads in each attention layer. This is used in the query-key-value attention mechanism, where the key-value heads are used to compute the attention weights.</td></tr><tr><td align="center">pretraining_tp</td><td align="center">1</td><td align="center">The tensor parallelism used during pretraining. Tensor parallelism allows for parallel computation across multiple devices or nodes.</td></tr><tr><td align="center">rms_norm_eps</td><td align="center">1e-05</td><td align="center">The epsilon value used for RMSNorm (Root Mean Square Normalization) layers. RMSNorm is a normalization technique that normalizes the activations based on their root mean square values. The epsilon is added for numerical stability.</td></tr><tr><td align="center">rope_scaling</td><td align="center"><em>null</em></td><td align="center">Indicates that RoPE (Rotary Position Embedding) scaling is not used. RoPE is a technique for encoding positional information in the attention mechanism.</td></tr><tr><td align="center">rope_theta</td><td align="center">500,000</td><td align="center">The scaling factor for RoPE. This value determines the frequency of the rotary position embeddings.</td></tr><tr><td align="center">tie_word_embeddings</td><td align="center"><em>false</em></td><td align="center">Indicates that word embeddings are not tied with the output layer. Tying word embeddings means sharing the weights between the input embedding layer and the output softmax layer.</td></tr><tr><td align="center">torch_dtype</td><td align="center">"bfloat16"</td><td align="center">The data type used for the model's weights and activations. In this case, Brain Floating Point 16-bit (bfloat16) is used, which is a 16-bit floating-point format that offers better performance and memory usage compared to the standard 32-bit float.</td></tr><tr><td align="center">transformers_version</td><td align="center">4.40.0.dev0</td><td align="center">The version of the Transformers library used. This specifies the version of the library that the model configuration is compatible with.</td></tr><tr><td align="center">use_cache</td><td align="center"><em>true</em></td><td align="center">Enables caching of key-value pairs during inference for faster generation. Caching stores the computed key-value pairs for each layer, avoiding redundant computations when generating sequences.</td></tr><tr><td align="center">vocab_size</td><td align="center">128,256</td><td align="center">The size of the model's vocabulary, in tokens. This represents the number of unique tokens that the model can recognize and generate, including regular words, subwords, and special tokens.</td></tr></tbody></table>

<details>

<summary><mark style="color:green;"><strong>config.json</strong></mark></summary>

The provided code snippet is a JSON configuration file for the Llama3 language model.&#x20;

1. "architectures": This parameter specifies the architecture class used for the language model, which is "LlamaForCausalLM". This indicates that the model is designed for causal language modeling tasks.
2. "attention\_bias": Set to `false`, indicating that attention bias is not used in the model.
3. "attention\_dropout": The dropout probability for the attention layers, set to 0.0, meaning no dropout is applied.
4. "bos\_token\_id": The ID of the beginning-of-sequence token, set to 128000.
5. "eos\_token\_id": The ID of the end-of-sequence token, set to 128001.
6. "hidden\_act": The activation function used in the hidden layers, set to "silu" (Sigmoid Linear Unit).
7. "hidden\_size": The dimensionality of the hidden states in the model, set to 4096.
8. "initializer\_range": The range used for initializing the model's weights, set to 0.02.
9. "intermediate\_size": The dimensionality of the intermediate (feed-forward) layer in the model, set to 14336.
10. "max\_position\_embeddings": The maximum sequence length that the model can process, set to 8192 tokens.
11. "model\_type": Specifies the type of the language model, set to "llama".
12. "num\_attention\_heads": The number of attention heads in each attention layer, set to 32.
13. "num\_hidden\_layers": The number of hidden layers (transformer blocks) in the model, set to 32.
14. "num\_key\_value\_heads": The number of key-value heads in each attention layer, set to 8.
15. "pretraining\_tp": The tensor parallelism used during pretraining, set to 1.
16. "rms\_norm\_eps": The epsilon value used for RMSNorm (Root Mean Square Normalization) layers, set to 1e-05.
17. "rope\_scaling": Set to `null`, indicating that RoPE (Rotary Position Embedding) scaling is not used.
18. "rope\_theta": The scaling factor for RoPE, set to 500000.0.
19. "tie\_word\_embeddings": Set to `false`, indicating that word embeddings are not tied with the output layer.
20. "torch\_dtype": The data type used for the model's weights and activations, set to "bfloat16" (Brain Floating Point 16-bit).
21. "transformers\_version": The version of the Transformers library used, set to "4.40.0.dev0".
22. "use\_cache": Set to `true`, enabling caching of key-value pairs during inference for faster generation.
23. "vocab\_size": The size of the model's vocabulary, set to 128256 tokens.

These configuration parameters define the architecture and hyperparameters of the Llama3 language model.&#x20;

The model uses a decoder-only transformer architecture with 32 hidden layers, each having 32 attention heads and a hidden size of 4096.&#x20;

The model can process sequences up to 8192 tokens long and uses SiLU activation in the hidden layers.&#x20;

The weights are initialised with a range of 0.02, and the model uses bfloat16 precision for its weights and activations. The vocabulary size is set to 128256 tokens, with specific IDs assigned for the beginning-of-sequence and end-of-sequence tokens.

</details>


# Model Analysis - Safetensors

### <mark style="color:blue;">File: model.safetensors.index.json</mark>

This JSON file serves as an index or a map that provides information about the storage and organisation of the model's weights across multiple files.

It helps the deep learning framework locate and load the appropriate weight files when the model is being used for inference or further training.

Now, let's break down the contents of the file and explain each part in more detail:

### <mark style="color:blue;">Metadata</mark>

This section contains metadata about the model.

### <mark style="color:blue;">total\_size</mark>

Specifies the total size of the model weights in bytes. In this case it is <mark style="color:yellow;">approximately 16 GB.</mark>

### <mark style="color:blue;">weight\_map</mark>

This section is the main part of the index file.&#x20;

It <mark style="color:yellow;">maps each model parameter to the corresponding file where its weights are stored</mark>.

* Each entry in the "weight\_map" follows the format: "<mark style="color:orange;">**parameter\_name**</mark>": "<mark style="color:purple;">**file\_name**</mark>".
* The "<mark style="color:orange;">**parameter\_name**</mark>" represents the <mark style="color:yellow;">name of a specific parameter in the model architecture.</mark> For example, "model.embed\_tokens.weight" refers to the embedding matrix used to map input tokens to their corresponding vector representations.
* The "<mark style="color:purple;">**file\_name**</mark>" indicates the <mark style="color:yellow;">name of the file where the weights for that specific parameter are stored.</mark> In this case, the weights are distributed across four files named "model-00001-of-00004.safetensors" to "model-00004-of-00004.safetensors".

The "<mark style="color:blue;">**weight\_map**</mark>" section is <mark style="color:yellow;">organised in a hierarchical manner</mark>, following the structure of the model architecture.&#x20;

Let's break it down further:

* "<mark style="color:green;">model.embed\_tokens.weight</mark>": Refers to the embedding matrix weights.
* "<mark style="color:green;">model.layers.0.input\_layernorm.weight</mark>" to "model.layers.31.self\_attn.v\_proj.weight": These entries correspond to the weights of the 32 transformer layers in the model.
  * Each layer has several components, such as "input\_layernorm" (input layer normalization), "mlp" (multilayer perceptron), "self\_attn" (self-attention mechanism), and "post\_attention\_layernorm" (post-attention layer normalization).
  * Within each component, there are specific weights. For example, "self\_attn.k\_proj.weight" refers to the weights of the key projection matrix in the self-attention mechanism.
* "model.norm.weight": Represents the weights of the final layer normalization applied to the model's output.

### <mark style="color:blue;">Why safetensor?</mark>

The safetensors format is used to store the model weights.

It is a file format designed for efficient storage and loading of large tensors, which is particularly useful for deep learning models.

By distributing the weights across multiple files, the model can be loaded more efficiently, especially when dealing with large models like Llama3.&#x20;

The index file provides a roadmap for the deep learning framework to locate and load the necessary weights for each parameter during model usage.

When you want to use the Llama3 model, the <mark style="color:yellow;">deep learning framework will read this index file</mark>, identify the required weight files, load them into memory, and reconstruct the model architecture by assigning the weights to their corresponding parameters.


# Tokenizer Configuration Files

The <mark style="color:blue;">**tokenizer\_config.json**</mark> and <mark style="color:blue;">**tokenizer.json**</mark> files serve different purposes in the tokenization process of the Llama3 language model.&#x20;

Let's clarify the difference between the two and how they interact:

### <mark style="color:blue;">tokenizer\_config.json</mark>

* This file contains the <mark style="color:yellow;">configuration settings for the tokenizer</mark>
* It defines the <mark style="color:yellow;">behaviour and properties of the tokenizer</mark>, such as the special tokens, maximum sequence length, and input tensor names.
* The tokenizer\_config.json file <mark style="color:yellow;">specifies how the tokenizer should handle and interpret the input text during the tokenization process.</mark>
* It includes settings like the beginning-of-sequence (BOS) token, end-of-sequence (EOS) token, and whether to clean up extra spaces during tokenization.
* The tokenizer\_config.json file also defines the mapping between special token IDs and their corresponding token content in the "added\_tokens\_decoder" section.

### <mark style="color:blue;">tokenizer.json</mark>

* This file <mark style="color:yellow;">contains the actual vocabulary and mappings used by the tokenizer to convert input text into token IDs.</mark>
* It defines the mapping between each word, subword, or character in the vocabulary and its corresponding unique token ID.
* The tokenizer.json file is used during the tokenization process to look up the token IDs for each word or subword in the input text.
* It is a crucial component of the tokenizer and is loaded by the tokenizer implementation to perform the actual tokenization.

### <mark style="color:blue;">Interaction between the two files</mark>

* The <mark style="color:blue;">**tokenizer\_config.json**</mark>**&#x20;file** provides the configuration settings for the tokenizer, specifying how it should behave and handle special tokens.
* The <mark style="color:blue;">**tokenizer.json**</mark> file contains the actual vocabulary and mappings used by the tokenizer to convert input text into token IDs.
* During the tokenization process, the tokenizer implementation loads both files:
  * It uses the tokenizer\_config.json file to configure its behavior and special token handling.
  * It uses the tokenizer.json file to look up the token IDs for each word or subword in the input text.
* The tokenizer applies the configuration settings from tokenizer\_config.json while utilizing the vocabulary and mappings from tokenizer.json to perform the tokenization.


# Model Analysis - tokenizer.json

tokenizer.json

### <mark style="color:blue;">Tokenizer.json Structure</mark>

* The tokenizer.json file contains a <mark style="color:yellow;">mapping between characters or subwords and their corresponding token IDs.</mark>
* Each entry in the file consists of a character or subword as the key and its assigned token ID as the value.
* The token IDs are unique integers that represent each character or subword in the vocabulary.
* The entries are typically structured in ascending order of token IDs.

### <mark style="color:blue;">Lowest Token IDs</mark>

* These token IDs <mark style="color:yellow;">correspond to the most basic and common characters in the vocabulary.</mark>
* The lowest token IDs usually start from 1 and are assigned to characters like punctuation marks, digits, and lowercase and uppercase letters.
* For example, the token ID 1 is assigned to the double quote character ("), token ID 2 is assigned to the hash character (#), and so on.
* The assignment of token IDs to characters follows a specific order, which is determined during the tokenizer training process.

### <mark style="color:blue;">Tokenizer Training</mark>

* The tokenizer is trained on a large corpus of text data to <mark style="color:yellow;">learn the vocabulary and assign token IDs to each character or subword.</mark>
* During the training process, the tokenizer analyzes the frequency and distribution of characters or subwords in the training data.
* It then assigns token IDs to each character or subword based on their frequency and importance in the vocabulary.
* The most common and basic characters, such as punctuation marks and letters, are usually assigned the lowest token IDs.
* As the token IDs increase, they are assigned to less common characters, subwords, or special tokens.

### <mark style="color:blue;">Tokenization Process</mark>

* When an input text is fed into the Llama3 language model, it undergoes a tokenization process.
* The tokenizer uses the mappings defined in the tokenizer.json file to convert the input text into a sequence of token IDs.
* Each character or subword in the input text is replaced by its corresponding token ID from the tokenizer.json file.
* This process converts the human-readable text into a numerical representation that the language model can understand and process.

### <mark style="color:blue;">Role in Training and Inference</mark>

* The tokenizer.json file plays a crucial role in both the training and inference phases of the Llama3 language model.
* During training, the <mark style="color:yellow;">tokenizer is used to preprocess the training data and convert it into token IDs</mark> before feeding it into the model.
* The language model learns to <mark style="color:yellow;">predict the next token ID based on the previous token IDs</mark> in the sequence.
* During inference, when a user provides an input text, the tokenizer is used to convert the text into token IDs, which are then passed to the trained model for generating predictions or responses.

### <mark style="color:blue;">Integration with Software Packages</mark>

* The tokenizer.json file is typically used in conjunction with deep learning frameworks and libraries such as PyTorch, TensorFlow, or Hugging Face's Transformers library.
* These libraries provide built-in functionalities to load the tokenizer.json file and use it for tokenization during training and inference.
* The tokenizer is often packaged together with the pre-trained language model weights, allowing developers to easily load and use the model for various natural language processing tasks.

In summary, the tokenizer.json file <mark style="color:yellow;">defines the mapping between characters or subwords and their corresponding token IDs.</mark>&#x20;

### <mark style="color:blue;">Summary</mark>

The file is structured in a way that assigns the lowest token IDs to the most common and basic characters, with increasing IDs for less common ones.&#x20;

The tokenizer is used during both training and inference to convert input text into token IDs, enabling the language model to process and generate human-readable text.&#x20;

The tokenizer.json file is tightly integrated with deep learning frameworks and libraries, making it easy to load and use the Llama3 model for various natural language processing tasks.

### <mark style="color:blue;">Structure</mark>

* The file contains a JSON object with various properties and arrays.
* Each entry in the file represents a token and its associated information.
* The structure of each token entry is as follows:

```json
{
  "id": <token_id>,
  "content": "<token_content>",
  "single_word": <true/false>,
  "lstrip": <true/false>,
  "rstrip": <true/false>,
  "normalized": <true/false>,
  "special": <true/false>
}
```

* <mark style="color:yellow;">`id`</mark><mark style="color:yellow;">:</mark> The unique integer ID assigned to the token.
* <mark style="color:yellow;">`content`</mark><mark style="color:yellow;">:</mark> The actual content of the token, which can be a word, subword, or special token.
* <mark style="color:yellow;">`single_word`</mark><mark style="color:yellow;">:</mark> Indicates whether the token represents a single word or not.
* <mark style="color:yellow;">`lstrip`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">and</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`rstrip`</mark><mark style="color:yellow;">:</mark> Specify whether leading or trailing whitespace should be stripped from the token.
* <mark style="color:yellow;">`normalized`</mark><mark style="color:yellow;">:</mark> Indicates whether the token has been normalized (e.g., lowercase, remove diacritics).
* <mark style="color:yellow;">`special`</mark><mark style="color:yellow;">:</mark> Indicates whether the token is a special token used for specific purposes.

### <mark style="color:blue;">Special Tokens</mark>

* The provided code snippet focuses on <mark style="color:yellow;">special tokens</mark>, which are <mark style="color:yellow;">predefined tokens used for specific purposes in the model.</mark>
* Special tokens are identified by their `special` property set to `true`.
* Examples of special tokens in the snippet include:
  * `<|reserved_special_token_1|>` to `<|reserved_special_token_33|>`: Reserved special tokens for future use.
  * `<|start_header_id|>` and `<|end_header_id|>`: Special tokens to mark the start and end of a header section.
  * `<|eot_id|>`: End-of-text token, indicating the end of the input sequence.
* Special tokens are assigned unique token IDs within the vocabulary range.

### <mark style="color:blue;">Vocabulary</mark>

* The remaining entries in the tokenizer.json file represent the actual vocabulary of the Llama3 model.
* Each entry maps a word or subword to a unique token ID.
* For example, `"ĠForm": 3459` maps the subword "ĠForm" to the token ID 3459.
* The vocabulary is typically sorted by frequency, <mark style="color:yellow;">with more frequent words having lower token IDs.</mark>


# Model Analysis - Special Tokens

tokenizer\_config.json

The configuration file for the tokenizer used in the Llama3 language model.&#x20;

### <mark style="color:blue;">added\_tokens\_decoder</mark>

* This section defines a <mark style="color:yellow;">mapping between special token IDs and their corresponding token content.</mark>
* Each entry in this object represents a special token with its ID as the key and its properties as the value.
* The properties of each special token include:
  * "content": The actual content of the special token, usually enclosed in angle brackets (e.g., "<|begin\_of\_text|>").
  * "lstrip" and "rstrip": Boolean values indicating whether leading or trailing whitespace should be stripped from the token.
  * "normalized": Indicates whether the token has been normalized (e.g., lowercase, remove diacritics).
  * "single\_word": Specifies if the token represents a single word or not.
  * "special": Indicates that the token is a special token used for specific purposes.

### <mark style="color:blue;">Special Tokens</mark>

* The configuration file includes a large number of special tokens, starting from token ID 128,000 up to 128,038.
* These special tokens serve various purposes in the tokenization process and the language model's functioning. Some common special tokens include:
  * <mark style="color:green;">"<|begin\_of\_text|>"</mark> (ID 128000): Marks the beginning of the input text.
  * <mark style="color:green;">"<|end\_of\_text|>"</mark> (ID 128001): Marks the end of the input text.
  * <mark style="color:green;">"<|start\_header\_id|>"</mark> (ID 128006) and "<|end\_header\_id|>" (ID 128007): Used to indicate the start and end of a header section in the input text.
  * <mark style="color:green;">"<|eot\_id|>"</mark> (ID 128009): Represents the end of the token sequence.
  * <mark style="color:green;">"<|reserved\_special\_token\_X|>"</mark> (IDs 128002-128005, 128008, 128010-128038): Reserved special tokens for future use or custom purposes.
* The large number of special tokens allows for flexibility and extensibility in the tokenization process and the language model's behavior.

### <mark style="color:blue;">Placeholders for special tokens in the tokenizer\_config.json file</mark>

The large number of special token placeholders (128002-128038) in the tokenizer\_config.json file provides flexibility and extensibility for the Llama3 language model.&#x20;

These placeholders allow for the definition and use of custom special tokens for various purposes. Here are a few ideas on how these special tokens could be utilized:

<mark style="color:green;">Domain-specific tokens</mark>

Special tokens can be used to represent domain-specific concepts or entities. For example, in a medical domain, special tokens could be defined for medical terms, drugs, or anatomical parts. These tokens can help the model better understand and generate text related to that specific domain.

<mark style="color:green;">Task-specific tokens</mark>

Special tokens can be employed to indicate specific tasks or instructions for the language model. For instance, special tokens could be used to specify the type of text generation task, such as "<|summarize|>" for summarization or "<|translate|>" for translation. The model can then interpret these tokens and perform the corresponding task accordingly.

<mark style="color:green;">Formatting and structure tokens</mark>

Special tokens can be utilized to represent formatting or structural elements in the input text. For example, tokens like "<|title|>", "<|paragraph|>", or "<|code\_block|>" can be used to indicate the presence of titles, paragraphs, or code blocks within the text. The model can learn to generate text with the appropriate formatting based on these special tokens.

<mark style="color:green;">Multi-turn conversation tokens</mark>

In conversational AI systems, special tokens can be employed to represent different speakers or turns in a conversation. Tokens like "<|user|>" and "<|system|>" can be used to differentiate between user input and system responses, enabling the model to generate more coherent and context-aware conversations.

<mark style="color:green;">Language-specific tokens</mark>

Special tokens can be defined to indicate the language of the input text or the desired output language. For example, tokens like "<|en|>" for English or "<|fr|>" for French can be used to specify the language context for the model.

These are just a few examples of how the special token placeholders in the tokenizer\_config.json file can be utilized. The flexibility provided by these placeholders allows developers and researchers to customize the tokenizer and language model behavior based on their specific requirements and use cases.

It's important to note that the actual use of these special tokens depends on how the language model is trained and fine-tuned. The model needs to be trained with the special tokens appropriately incorporated into the training data to learn their intended meanings and behaviors.

### <mark style="color:blue;">Other Configuration Settings</mark>

* <mark style="color:green;">"bos\_token":</mark> Specifies the special token used for the beginning of a sequence ("<|begin\_of\_text|>").
* <mark style="color:green;">"clean\_up\_tokenization\_spaces":</mark> Indicates whether to clean up extra spaces during tokenization (set to `true`).
* <mark style="color:green;">"eos\_token":</mark> Specifies the special token used for the end of a sequence ("<|end\_of\_text|>").
* <mark style="color:green;">"model\_input\_names":</mark> Defines the names of the input tensors expected by the language model ("input\_ids" and "attention\_mask").
* <mark style="color:green;">"model\_max\_length":</mark> Sets the maximum length of the input sequence that the model can process (an extremely large value in this case).
* <mark style="color:green;">"tokenizer\_class":</mark> Specifies the class of the tokenizer ("PreTrainedTokenizerFast").

The purpose of having a large number of special tokens is to provide a wide range of control and flexibility in the tokenization process and the language model's behavior.&#x20;

Special tokens can be used to mark specific segments of the input text, indicate the beginning or end of sequences, represent out-of-vocabulary words, or serve as placeholders for custom purposes.

By defining these special tokens in the tokenizer configuration file, the Llama3 language model can recognise and handle them appropriately during the tokenization process.&#x20;

The model can use these special tokens to guide its understanding of the input text structure, control the generation process, and perform specific tasks based on the presence or absence of certain special tokens.

The <mark style="color:yellow;">tokenizer\_config.json</mark> file works in conjunction with the tokenizer implementation to ensure consistent tokenization and interpretation of the input text.&#x20;

The tokenizer uses this configuration to map the input text to the corresponding token IDs, including the special tokens, which are then fed into the language model for processing and generation tasks.


# Llama3 - Model Configuration

### <mark style="color:blue;">Model Configuration</mark>

The first configuration block of the Axolotl configuration file is 'model type'.  It comprises three main configurations.

1. base\_model
2. model\_type
3. tokenizer\_type

```yaml
base_model: meta-llama/Meta-Llama-3-8B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
```

Below are explanations of the Huggingface Transformer classes that are used within the Axolotl training script specific to the model and tokenizer type.

<details>

<summary>Reference: <mark style="color:yellow;">AutoModelForCausalLM</mark> <mark style="color:green;">- class within the Huggingface Transformer library</mark></summary>

The <mark style="color:yellow;">`AutoModelForCausalLM`</mark> class is part of the Hugging Face Transformers library and is designed to provide a convenient way to instantiate and work with pre-trained models for causal language modeling tasks.

Causal language modeling, also known as autoregressive language modeling, is a type of language modeling task where the model predicts the next token in a sequence based on the previous tokens.&#x20;

In other words, given a sequence of tokens, the model learns to predict the probability distribution of the next token. This is commonly used for tasks like text generation, where the model generates text by predicting one token at a time based on the previously generated tokens.

The <mark style="color:yellow;">`AutoModelForCausalLM`</mark> class is a subclass of <mark style="color:yellow;">`_BaseAutoModelClass`</mark>, which is a <mark style="color:blue;">base class for all the auto model classes in the Transformers library</mark>.&#x20;

The purpose of the auto model classes is to provide a unified interface for loading and using pre-trained models for various tasks.

Here's how the <mark style="color:yellow;">`AutoModelForCausalLM`</mark> class works:

1. It has a class attribute <mark style="color:yellow;">`_model_mapping`</mark> that maps the model names to their corresponding classes for causal language modeling. This mapping allows the class to automatically determine the appropriate model class based on the provided model name or path.
2. When you call the <mark style="color:yellow;">`from_pretrained()`</mark> method of <mark style="color:yellow;">`AutoModelForCausalLM`</mark> and provide a pre-trained model name or path, it automatically retrieves the corresponding model class from the <mark style="color:yellow;">`_model_mapping`</mark> based on the model name.
3. It then initializes and returns an instance of the retrieved model class, which can be used for causal language modeling tasks.

The advantage of using <mark style="color:yellow;">`AutoModelForCausalLM`</mark> is that you *<mark style="color:yellow;">**don't need to know the specific model class for a given pre-trained model.**</mark>* You can simply provide the model name or path, and the class will handle the instantiation of the appropriate model class for you.

For example, if you have a pre-trained GPT-2 model and want to use it for causal language modeling, you can do the following:

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")
```

This code will <mark style="color:blue;">automatically load the GPT-2 model</mark> and return an instance of the appropriate model class for causal language modeling.

Overall, the <mark style="color:yellow;">`AutoModelForCausalLM`</mark> class provides a convenient and flexible way to work with pre-trained models for causal language modeling tasks, abstracting away the need to know the specific model classes and allowing you to focus on using the models for your desired task.

</details>

<details>

<summary>Reference: <mark style="color:yellow;"><strong>AutoTokenizer</strong></mark> <mark style="color:green;">- a class within the Huggingface Transformer library</mark></summary>

The <mark style="color:yellow;">`AutoTokenizer`</mark> class is a powerful and versatile tool in the Hugging Face Transformers library that simplifies the process of instantiating the appropriate tokenizer for a given pretrained model.

It serves as a high-level interface that automatically selects and initializes the correct tokenizer class based on the provided pretrained model name or path.

Let's dive into the key aspects of the <mark style="color:yellow;">`AutoTokenizer`</mark> class and understand how it enhances the usability and flexibility of tokenization in natural language processing tasks:

<mark style="color:green;">Automatic Tokenizer Selection</mark>

* The primary purpose of the `AutoTokenizer` class is to *<mark style="color:yellow;">**automatically determine the appropriate tokenizer class to use based on the pretrained model.**</mark>*
* It eliminates the need for users to manually specify the tokenizer class, saving time and reducing the chances of errors.
* The class leverages various methods to infer the tokenizer class, such as examining the model's configuration, using pattern matching on the model name or path, or utilizing a tokenizer configuration file.

<mark style="color:green;">Pretrained Model Support</mark>

* The `AutoTokenizer` class seamlessly integrates with pretrained models available on the Hugging Face Model Hub or locally saved models.
* It accepts a <mark style="color:yellow;">`pretrained_model_name_or_path`</mark> parameter, which can be a model identifier, a path to a directory containing the necessary files, or a URL to a specific file.
* This flexibility allows users to easily load tokenizers associated with a wide range of pretrained models, enabling quick experimentation and transfer learning.

<mark style="color:green;">Tokenizer Instantiation</mark>

* The <mark style="color:yellow;">`from_pretrained()`</mark> class method is the primary entry point for instantiating tokenizers using the <mark style="color:yellow;">`AutoTokenizer`</mark><mark style="color:yellow;">.</mark>
* It takes care of downloading and caching the required files, such as vocabulary files, if they are not already present locally.
* The method accepts various parameters to customize the tokenizer's behavior, such as specifying the tokenizer type, using a fast tokenizer variant, or providing additional keyword arguments.

<mark style="color:green;">**Fast Tokenizers**</mark>

* The <mark style="color:yellow;">`AutoTokenizer`</mark> class supports the use of fast tokenizers, which are implemented in Rust and offer improved performance compared to their Python counterparts.
* By setting the <mark style="color:yellow;">`use_fast`</mark> parameter to <mark style="color:yellow;">`True`</mark> (default), the class automatically selects the fast tokenizer variant if available for the given model.
* If a fast tokenizer is not available, it gracefully falls back to the standard Python-based tokenizer.

<mark style="color:green;">**Trust Remote Code**</mark>

* The <mark style="color:yellow;">`trust_remote_code`</mark> parameter allows users to control whether the <mark style="color:yellow;">`AutoTokenizer`</mark> should trust and execute custom tokenization code defined in the model's repository.
* This feature is useful for models that require specific tokenization logic but should be used with caution and only for trusted repositories.

<mark style="color:green;">**Tokenizer Configuration**</mark>

* The <mark style="color:yellow;">`AutoTokenizer`</mark> class utilizes configuration objects <mark style="color:yellow;">(</mark><mark style="color:yellow;">`PretrainedConfig`</mark><mark style="color:yellow;">)</mark> to determine the appropriate tokenizer class to instantiate.
* It first attempts to load the tokenizer configuration from a dedicated file <mark style="color:yellow;">(</mark><mark style="color:yellow;">`tokenizer_config`</mark><mark style="color:yellow;">) a</mark>ssociated with the pretrained model.
* If the tokenizer configuration is not available, it falls back to using the model's configuration <mark style="color:yellow;">(</mark><mark style="color:yellow;">`AutoConfig`</mark><mark style="color:yellow;">) t</mark>o infer the tokenizer class.

<mark style="color:green;">**Tokenizer Registration**</mark>

* The `AutoTokenizer` class provides a <mark style="color:yellow;">`register()`</mark> method that allows users to register new tokenizer classes.
* This feature is particularly useful for extending the <mark style="color:yellow;">`AutoTokenizer`</mark> to support custom or newly developed tokenizers.
* By registering a configuration class along with the corresponding slow and fast tokenizer classes, users can seamlessly integrate their own tokenizers into the <mark style="color:yellow;">`AutoTokenizer`</mark> ecosystem.

In summary, the <mark style="color:yellow;">`AutoTokenizer`</mark> class is a powerful tool that simplifies the process of initializing tokenizers for pretrained models.&#x20;

It abstracts away the complexities of manually selecting and instantiating tokenizer classes, allowing users to focus on their natural language processing tasks. With its automatic tokenizer selection, support for pretrained models, fast tokenizer variants, and extensibility through registration, the <mark style="color:yellow;">`AutoTokenizer`</mark> class greatly enhances the usability and flexibility of tokenization in the Hugging Face Transformers library.

The AutoTokenizer class from the Hugging Face Transformers library is a generic tokenizer class that automatically instantiates the appropriate tokenizer for a given pretrained model.&#x20;

It provides a convenient way to load and use tokenizers without explicitly specifying the tokenizer class.

</details>

The next step after determining the model type configurations is to configure the <mark style="color:blue;">d</mark><mark style="color:blue;">**ata loading and processing parameters**</mark>


# Llama3 - Model Quantization

With the model configured, we next have to determine whether whether we will be using quantization in the training process.

&#x20;The first configuration is whether we will be <mark style="color:blue;">**training the model**</mark> using the Lora or QLora technique - that is training the model with **4 or 8 bit quantization.**

**We will be using the&#x20;**<mark style="color:yellow;">**Lora Parameter Efficient Fine Tuning Technique**</mark>

### <mark style="color:blue;">Model Configuration</mark>

The first configuration block of the Axolotl configuration file is 'model type'.  It comprises three main configurations.

1. base\_model
2. model\_type
3. tokenizer\_type

```yaml
load_in_8bit: true
load_in_4bit: false
strict: false
```

### <mark style="color:blue;">load\_in\_8bit: true or false</mark>

This is configuration flag that determines <mark style="color:yellow;">whether the model should be loaded in</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">**8-bit precision.**</mark>  If it is set to "true," the model will be loaded in 8-bit precision.

<mark style="color:green;">**Memory Efficiency**</mark>

8-bit precision reduces the memory footprint of the model compared to higher precision formats (like 16-bit or 32-bit). This is because it requires less memory to store each weight in the model.

Loading a model in 8-bit precision can accelerate model loading and inference times. This is due to the reduced computational load compared to higher precision formats.

While 8-bit precision is more efficient, it can slightly reduce the accuracy of the model compared to full precision (32-bit). This happens because of the reduced resolution in representing the weights and activations.

### <mark style="color:blue;">load\_in\_4bit: true or false</mark>

This is configuration flag that determines whether the model should be loaded in **4-bit precision**.  If it is set to "true," the model will be loaded in 4-bit precision.

4-bit precision takes the concept of memory efficiency further, halving the memory requirements compared to 8-bit. This can be crucial for deploying large models on limited hardware.

Similar to 8-bit, 4-bit precision can lead to even faster loading and inference times due to the further reduced computational requirements.

The trade-off in accuracy might be more pronounced in 4-bit precision.&#x20;

The reduced bit-depth means that the model's ability to represent nuanced information in weights and activations is more limited. This might affect tasks that require high precision or are sensitive to small changes in weights.

### <mark style="color:blue;">strict: true or false</mark>

#### Strict Mode (strict)

* If set to false, default weights will be chosen where missing in adapters.  This is a component of 'bits and bytes' library

<details>

<summary><mark style="color:green;">What is 'bits and bytes'?</mark></summary>

**Overview**

The <mark style="color:yellow;">`bitsandbytes`</mark> repository provides a lightweight wrapper around CUDA custom functions, primarily focusing on 8-bit optimizers, matrix multiplication <mark style="color:yellow;">(</mark><mark style="color:yellow;">`LLM.int8()`</mark><mark style="color:yellow;">),</mark> and quantization functions. This tool is designed to enhance the performance and efficiency of machine learning models, particularly in the context of CUDA-enabled computing environments.

**Key Features**

* **8-bit Optimizers**: Specialized for reducing memory usage and improving computational efficiency.
* **Matrix Multiplication (LLM.int8())**: Offers optimized matrix multiplication capabilities.
* **Quantization Functions**: Includes various methods for quantizing models, contributing to reduced model sizes and potentially faster inference times.

**Requirements**

* Python version 3.8 or higher.
* Linux distribution (Ubuntu, MacOS, etc.) with CUDA version greater than 10.0.
* Note: CUDA 10.0 is deprecated, and future support is focused on CUDA >= 11.0 with release 0.39.0.

**Installation**

* Installable via pip (`pip install bitsandbytes`).
* In cases where compilation from source is necessary, users are encouraged to submit a bug report and follow the provided compilation instructions.

**Usage Highlights**

* **Int8 Inference with HuggingFace Transformers**: Allows models to load in 8-bit for reduced memory usage.
* **8-bit Optimizer Usage**: Users can easily switch to 8-bit optimizers by replacing their existing optimizers with the corresponding 8-bit version from `bitsandbytes`.
* **Mixed 8-bit Training and Int8 Inference**: The library supports both mixed 8-bit training with 16-bit main weights and full 8-bit inference.

**Features**

* Advanced techniques for 8-bit matrix multiplication and LLM.int8() inference.
* A range of 8-bit optimizers including Adam, AdamW, RMSProp, LARS, LAMB, and Lion.
* A stable embedding layer feature for improved stability in NLP models.
* Fast and efficient algorithms for quantile estimation.

**Requirements & Hardware Compatibility**

* Requires Anaconda, cudatoolkit, and PyTorch.
* Compatible with NVIDIA GPUs, specifically Turing or newer for LLM.int8(), and Kepler or newer for 8-bit optimizers and quantization.
* Supports CUDA versions from 10.2 to 12.0.
* Note: The library is currently supported only on Linux distributions.

**Additional Information**

* The repository includes instructions for users of Fairseq, detailing how to implement the Stable Embedding Layer and use the optimizers.
* A section on troubleshooting common errors and solutions is provided.
* The repository also contains detailed instructions for compiling from source if necessary.

**Licensing**

* The majority of `bitsandbytes` is under the MIT license, with some portions (like Pytorch) under different licenses.

**Community Contributions**

* Acknowledgment to Fabio Cannizzo for his work on FastBinarySearch, used for CPU quantization in the project.

</details>

<details>

<summary><mark style="color:green;">Summary of Tim Detmers' Presentation on 8-Bit Methods for Efficient Deep Learning</mark></summary>

His main thesis is that computationally efficient methods will accelerate progress in understanding deep learning.

<mark style="color:green;">**Key Points from Tim's Presentation**</mark>

**8-Bit Methods for Large Models**: Tim highlights the importance of making large models more accessible through quantization, which <mark style="color:yellow;">reduces the memory footprint.</mark>

**Quantization Explained**: He explains quantization as a process of <mark style="color:yellow;">converting floating-point or real representations into discrete buckets</mark>, akin to histogram binning.

**Linear vs. Nonlinear Quantization**: <mark style="color:yellow;">Linear (integer) quantization involves equally wide bins, while nonlinear quantization allows varying bin widths</mark>.

**Error Reduction in Quantization**: Tim illustrates how the <mark style="color:yellow;">choice of bins impacts precision and error distribution in quantized values.</mark>

**4-Bit Inference**: His recent work shows that <mark style="color:yellow;">4-bit inference is highly effective for large transformers</mark>.

**Floating Point Data Types**: The presentation delves into the structure of floating point data types, explaining the roles of exponent bits and fraction bits.

**Dynamic Exponent Data Type**: Tim introduces a <mark style="color:yellow;">unique data type he developed with a dynamic exponent</mark>, which offers flexibility in approximating large and small values with varying precision.

**8-Bit Optimizers**: The focus shifts to 8-bit optimizers, crucial for memory efficiency in training large models, particularly in language modeling.

Tim discusses <mark style="color:yellow;">reducing memory usage by approximately 40% by converting 32-bit Adam optimizer buffers to 8-bit.</mark>

This reduction is significant as it helps make large models more memory-efficient.

<mark style="color:yellow;">Outliers in Adam optimizer buffers cause issues in quantization</mark>, leading to increased error and ineffective 8-bit quantization.

Tim presents an example showing how <mark style="color:yellow;">outliers can skew the data</mark>, leading to a waste of bits and loss of effective representation.

To address the problem of outliers, Tim <mark style="color:yellow;">proposes chunking Adam states into blocks and quantizing each block independently</mark>.

This method isolates the impact of outliers to specific blocks, enhancing the stability of 8-bit optimizers.

The process involves <mark style="color:yellow;">chunking state into blocks</mark>, finding the maximum value for normalization, and storing the index for 8-bit representation.

This method ensures compact yet effective optimization, <mark style="color:yellow;">comparable to 32-bit optimizers</mark>.

This achievement indicates significant memory savings without compromising performance.

8-bit optimizers are efficient in mapping onto hardware, with the <mark style="color:yellow;">main overhead being the dequantization process.</mark>

<mark style="color:yellow;">Outliers become a significant problem in models larger than 6.7 billion parameters</mark>, causing performance drops.

Tim's research identifies <mark style="color:yellow;">systematic outliers that emerge with scale</mark> and become problematic at specific model sizes.

Outliers in large models exhibit systematic and emergent properties, affecting the same dimensions across layers.

These outliers impact all layers in a transformer model once a certain scale is reached.

The emergence of outliers follows an exponential trend, leading to a phase shift-like effect at a certain scale.

Understanding and addressing this exponential trend is key to managing outliers in large models.

A novel approach was developed to identify and process these outliers in 16-bit while handling the rest in 8-bit, effectively maintaining efficiency while addressing the problem.

<mark style="color:green;">**Efficiency of 8-Bit Matrix Multiplication**</mark>

* By applying this method, 99.9% of weights are computed in 8-bit, with a small portion in 16-bit for outliers. This approach achieves performance equivalent to 16-bit computations while halving memory size.
* This makes large models like Llama 65B accessible on consumer hardware, significantly lowering the barrier to entry for working with such models.

<mark style="color:green;">**Few-Shot and Zero-Shot Performance**</mark>

* The few-shot performance of models using 8-bit methods is comparable to 16-bit models. Tim highlighted a strong correlation between zero-shot performance and perplexity in language models, indicating that complexity evaluations can reliably predict zero-shot performance.

<mark style="color:green;">**Understanding Outliers in Transformer Models**</mark>

* Outliers tend to be concentrated in specific columns of the input batch and are more prevalent in larger models. These outliers are crucial for attention mechanisms in transformers.
* They are context-independent, aiding the attention mechanism in focusing on specific values by providing predictable patterns for the model to cancel out unnecessary information.

<mark style="color:green;">**Trade-Offs in Activation Functions**</mark>

* Replacing traditional activation functions like softmax with more stable alternatives can increase stability but may lead to a drop in performance.
* This presents a research challenge in balancing stability with maintaining or enhancing model performance.

<mark style="color:green;">**Impact of Precision and Parameter Count on Model Efficiency**</mark>

* An interesting finding is that models with the same number of bits but different distributions of precision and parameter count (e.g., 8-bit with more parameters vs. 4-bit with fewer parameters) exhibit the same inference latency.
* This equivalence in performance is due to the nature of GPU computations, where memory loading is significantly more costly than computation. Therefore, the memory used during inference, rather than the computational complexity, often dictates the performance.

</details>


# Llama3 - Data Loading and Paths

## <mark style="color:green;">datasets</mark>

<pre class="language-yaml"><code class="lang-yaml"><strong>datasets:
</strong>  - path: datasets/alpagasus/data/train-00000-of-00001-0c59455170918204.parquet
    type: alpaca
    ds_type: parquet
    data_files:
  - train-00000-of-00001-0c59455170918204.parquet
dataset_prepared_path:
val_set_size: 0.10
output_dir: ./llama3-out
</code></pre>

This component of the configuration file tells the platform where the dataset it located, and how the dataset should be split up into <mark style="color:yellow;">training versus validation.</mark>

We will only be using a number of these parameters for our fine tuning.  The full number of configurations can be found here:  [<mark style="color:orange;">**data loading and processing configurations**</mark>](/axolotl-configuration-files/data-loading-and-processing)

### <mark style="color:blue;">**We will be using the Alpgasus dataset**</mark>

{% embed url="<https://arxiv.org/abs/2307.08701>" %}

<details>

<summary><mark style="color:green;"><strong>Alpgasus README file</strong></mark></summary>

This README.md file describes a dataset hosted on Huggingface, which is based on the AlpaGasus paper.&#x20;

AlpaGasus is an unofficial implementation of a dataset that aims to improve upon the original Alpaca dataset. The key points about this dataset are:

<mark style="color:green;">Features: The dataset consists of three features:</mark>

* `instruction`: The instruction or prompt given to the model (data type: string)
* `input`: The input text associated with the instruction (data type: string)
* `output`: The expected output or response from the model (data type: string)

<mark style="color:green;">Splits: The dataset has a single split called "train," which contains:</mark>

* Number of bytes: 3,918,129
* Number of examples: 9,229
* Download size: 2,486,877 bytes
* Dataset size: 3,918,129 bytes

<mark style="color:green;">Configurations:</mark> There is only one configuration named "default," which uses the data files located at "data/train-\*" paths.

1. License: The dataset is licensed under the GPL-3.0 license.
2. Task Categories: The dataset is categorized under the "text-generation" task.
3. Tags: The dataset is tagged with "alpaca" and "llama."
4. Size Categories: The dataset falls under the "1K\<n<10K" size category, indicating that it has between 1,000 and 10,000 examples.

The AlpaGasus dataset is a filtered version of the original Alpaca dataset, where GPT-4 acts as a judge to select high-quality samples.&#x20;

The authors of the AlpaGasus paper demonstrated that models trained on this filtered dataset with only 9,000 samples can outperform models trained on the original 52,000 samples of the Alpaca dataset.

The key takeaways are:

1. AlpaGasus is an unofficial implementation that aims to improve the Alpaca dataset by filtering it using GPT-4.
2. The dataset contains 9,229 high-quality examples, significantly fewer than the original Alpaca dataset.
3. Despite having fewer examples, models trained on the AlpaGasus dataset can outperform those trained on the larger Alpaca dataset.
4. The dataset is designed for text generation tasks and is compatible with models like Alpaca and Llama.
5. The dataset is available on Huggingface and is licensed under GPL-3.0.

</details>

### <mark style="color:blue;">Tips on Parquet files</mark>

Here are some interesting insights about using Parquet files when training large language models:

Axolotl supports loading datasets in various formats, including Parquet files.&#x20;

This allows for flexibility in how the training data is stored and accessed.

Parquet files are a popular choice for storing training datasets due to their efficiency and ability to handle large amounts of data. They are columnar storage format that provides fast querying and retrieval of specific columns.

When specifying the dataset in the Axolotl configuration file (YAML), you can provide the path to the Parquet files or directories containing the Parquet files. For example:

```yaml
datasets:
  - path: /path/to/parquet/files/
    type: alpaca
    ds_type: parquet
```

Parquet files can be stored locally or in remote storage systems like Amazon S3 or Google Cloud Storage (GCS).&#x20;

Axolotl supports loading datasets from these remote storage systems by specifying the appropriate path format, such as `s3://path/to/data.parquet` or `gs://path/to/parquet/dir/`.

If the dataset is separated into multiple Parquet files on Hugging Face or other platforms, you can specify the number of shards to split the data into using the <mark style="color:yellow;">**`shards`**</mark> parameter in the dataset configuration.

Preprocessing the data and converting it into Parquet or Arrow format can be beneficial for efficient storage and faster loading during training.  It allows you to filter out unnecessary data and optimise the dataset for your specific training needs.

The size of the Parquet files can vary depending on the dataset and the specific requirements of the training task.

When loading Parquet files, you can specify the <mark style="color:yellow;">**`ds_type`**</mark> as "parquet" in the dataset configuration to explicitly indicate the data type.

Overall, using Parquet files provides a scalable and efficient way to store and load training datasets for large language models.  Axolotl's support for Parquet files, along with other formats like JSON, Arrow, and CSV, offers flexibility in dataset management and allows for seamless integration with various storage systems.

### <mark style="color:blue;">fine tuning data set location</mark>

#### <mark style="color:green;">dataset\_prepared\_path:</mark>

This parameter indicates the <mark style="color:yellow;">path to a dataset that has been prepared for training</mark>.&#x20;

#### <mark style="color:green;">ds\_type:</mark>

Huggingface datasets can come in a number of formats.  You can configure the data loading process to take into account the data type - csv, json, parquet.   We will be using Parquet format.

### <mark style="color:blue;">validation Set</mark>

#### <mark style="color:green;">val\_set\_size: 0.10</mark>

**Description:** Specifies the <mark style="color:yellow;">size of the validation set as a percentage of the total dataset.</mark>

**Meaning:** This parameter determines the proportion of the dataset that will be used for validation during the training process. In this case, the default is set to 5% of the total dataset.

### <mark style="color:blue;">output files</mark>

#### <mark style="color:green;">output\_dir: ./output</mark>

Description: Specifies the directory where <mark style="color:yellow;">output files should be saved.</mark>

Meaning: This parameter designates the directory where various output files, such as trained model checkpoints or logs, should be stored.


# Llama3 - Sequence Configuration

Before training or fine-tuning begins, the <mark style="color:yellow;">input data must be correctly formatted and prepared</mark>.

We will be configuring the:

Sequence Length

1. Sequence Length
2. Sample Packing
3. Padding to Sequence

```yaml
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
```

### <mark style="color:blue;">sequence\_len</mark>

This parameter sets the <mark style="color:yellow;">maximum allowable length for input sequences</mark>. Sequences longer than this length may be truncated or split during training.

This limit is essential since transformers process input data in fixed-size blocks.

Sequences longer than this length are either truncated or split.&#x20;

Truncation means cutting off the part of the sequence that exceeds the limit, while splitting involves dividing a long sequence into smaller segments, each within the maximum length.

The choice of sequence length affects memory usage and computational requirements. Longer sequences can capture more context but require more computational resources.

### <mark style="color:blue;">sample\_packing</mark>

A flag that determines whether sample packing should be used.&#x20;

This is a method to optimise the training process by packing multiple shorter sequences into a single training example (batch).   It can increases training efficiency by reducing padding needs and better utilizing GPU memory.  This technique is particularly &#x75;*<mark style="color:yellow;">**seful when dealing with variable-length sequences.**</mark>*

**Implementation**: If set to <mark style="color:yellow;">**`true`**</mark>, sequences that are shorter than <mark style="color:yellow;">**`sequence_len`**</mark> are concatenated with others to form a packed batch. This process continues until the maximum sequence length is reached or no more sequences are available for packing.

### <mark style="color:blue;">pad\_to\_sequence\_len</mark>

This is a flag that controls whether sequences should be padded to match the specified sequence length.

This ensures that <mark style="color:yellow;">all sequences in a batch are of the same length</mark>, which is necessary for parallel processing by the model.  &#x20;

Shorter sequences are extended (padded) with special tokens (usually `[PAD]`) to reach the defined maximum sequence length.

Padding is a standard practice in training neural networks on sequences of varying lengths, but it can introduce additional computational overhead, especially with longer sequence lengths.

If set to "true," input sequences will be padded with special tokens to reach the maximum sequence length defined earlier.


# Llama3 - Lora Configuration

This is the default Lora Configuration for Llama3

```yaml
adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
```

#### <mark style="color:yellow;">`lora_r`</mark>

This parameter determines the rank of the low-rank matrices used in LoRA.&#x20;

It controls the capacity and expressiveness of the LoRA adaptation. A higher value of `lora_r` allows for more fine-grained adaptations but also increases the number of trainable parameters.&#x20;

In this configuration, <mark style="color:yellow;">**`lora_r`**</mark> is set to 32.

#### <mark style="color:yellow;">`lora_alpha`</mark>

This parameter controls the scaling factor applied to the LoRA adaptation.&#x20;

It determines the contribution of the LoRA matrices to the original model's weights.

A higher value of <mark style="color:yellow;">**`lora_alpha`**</mark> gives more importance to the LoRA adaptation. In our configuration, <mark style="color:yellow;">**`lora_alpha`**</mark> is set to 16.

#### <mark style="color:yellow;">`lora_dropout`</mark>

This parameter specifies the dropout rate applied to the LoRA matrices during training.&#x20;

Dropout is a regularization technique that helps prevent overfitting.&#x20;

A value of 0.05 means that 5% of the elements in the LoRA matrices will be randomly set to zero during training.

#### <mark style="color:yellow;">`lora_target_modules`</mark>

This parameter specifies the names of the modules in the model architecture where LoRA will be applied.&#x20;

In this  configuration, LoRA is applied to the `q_proj` and `v_proj` modules, which are likely the query and value projection matrices in the attention mechanism.

We have commented out other potential target modules like `k_proj`, `o_proj`, `gate_proj`, `down_proj`, and `up_proj`.

#### <mark style="color:yellow;">`lora_target_linear`</mark>

This parameter is not set in your configuration.&#x20;

If set to `true`, LoRA will be applied to all linear modules in the model.

#### <mark style="color:yellow;">`peft_layers_to_transform`</mark>

This parameter allows you to specify the indices of the layers to which LoRA should be applied.

If not specified, LoRA will be applied to all layers by default.

#### <mark style="color:yellow;">`lora_modules_to_save`</mark>

This parameter is relevant when you have added new tokens to the tokenizer. In such cases, you may need to save certain LoRA modules that are aware of the new tokens.&#x20;

For LLaMA and Mistral models, you typically need to save `embed_tokens` and `lm_head` modules. `embed_tokens` converts tokens to embeddings, and `lm_head` converts embeddings to token probabilities.&#x20;

In this configuration, these modules are commented out.

#### <mark style="color:yellow;">`lora_fan_in_fan_out`</mark>

This parameter determines the structure of the LoRA matrices. If set to <mark style="color:yellow;">**`true`**</mark>, it uses a more efficient implementation of LoRA that reduces the number of additional parameters. In your configuration, it is set to <mark style="color:yellow;">**`false`**</mark>.

These hyperparameters allow you to control various aspects of the LoRA adaptation during fine-tuning.&#x20;

The optimal values for these hyperparameters may vary depending on your specific task, dataset, and model architecture. It's recommended to experiment with different configurations and monitor the performance to find the best settings for your use case.


# Llama3 - Logging

We will be using Weights and Biases to log and monitor our fine tuning runs.

If you are new to Weights and Biases, please take the time to review their quickstart guide:

{% embed url="<https://docs.wandb.ai/quickstart>" %}
Weights and Biases Quickstart
{% endembed %}

&#x20;A [run](https://docs.wandb.ai/guides/runs) is the basic building block of W\&B. You will use them often to [track metrics](https://docs.wandb.ai/guides/track), [create logs](https://docs.wandb.ai/guides/artifacts), [create jobs](https://docs.wandb.ai/guides/launch).

Once you have created a Weights and Biases account, login to  your Weights and Biases account by entering the following command at the command prompt:

```bash
wandb login
```

You will be asked to enter your Weights and Biases account details as per below:

<mark style="color:blue;">Username:</mark>&#x20;

<mark style="color:blue;">Password:</mark> &#x20;

<mark style="color:blue;">API Token:</mark>&#x20;

### <mark style="color:blue;">Configuration of Logging Component</mark>

<mark style="color:green;">Project</mark>

The project naming convention can be simplistic but informational.   Name your project whatever you like.

<mark style="color:green;">Entity</mark>

The <mark style="color:yellow;">entity</mark> is simply \<your-organisation>.

The configuration file should look like the below script:

```yaml
wandb_project: <your project name>
wandb_entity: <your-organisation> 
wandb_name: Set the name of your wandb run
wandb_run_id: Set the ID of your wandb run
```

<details>

<summary>Reference: <mark style="color:green;">The Weights and Biases Script in the Axolotl Library</mark></summary>

<mark style="color:yellow;">src/axolotl/utils/wandb\_.py</mark>

The script defines a function `setup_wandb_env_vars` that configures environment variables for wandb based on a given configuration. Here's a detailed explanation of what this function does:

<mark style="color:green;">**Module Description**</mark>

* The comment `"""Module for wandb utilities"""` indicates that this Python file is intended to provide utility functions for working with wandb.

<mark style="color:green;">**Import Statements**</mark>

* The script imports the `os` module, which provides functions for interacting with the operating system, including managing environment variables.
* It imports `DictDefault` from `axolotl.utils.dict`, which is a custom dictionary utility, likely providing some default behavior for dictionary operations.

<mark style="color:green;">**Function Definition**</mark>

`setup_wandb_env_vars(cfg: DictDefault)`:

* The function takes a single parameter `cfg`, which is expected to be an instance of `DictDefault`. This parameter likely contains configuration settings.

<mark style="color:green;">**Setting Environment Variables for wandb**</mark>

* The function iterates through all keys in the `cfg` dictionary.
* If a key starts with `"wandb_"`, the function retrieves its value.
* If the value is a non-empty string, the function sets an environment variable with the name of the key in uppercase and assigns it the value from `cfg`. For example, if `cfg` contains `{"wandb_api_key": "your_key"}`, it sets an environment variable `WANDB_API_KEY` with the value `"your_key"`.

<mark style="color:green;">**Enabling wandb Integration**</mark>

* The function checks if `cfg.wandb_project` exists and is a non-empty string.
* If `cfg.wandb_project` is set, `cfg.use_wandb` is set to `True`, and any existing `WANDB_DISABLED` environment variable is removed. This implies that wandb should be enabled and operational for the current session.
* If `cfg.wandb_project` is not set, the function sets the environment variable `WANDB_DISABLED` to `"true"`, effectively disabling wandb integration.

<mark style="color:green;">**Usage Context**</mark>

* This script is useful for dynamically configuring wandb based on a set of configuration parameters, especially in scenarios where different wandb settings are needed for different runs or experiments.
* It automates the process of setting up wandb environment variables, enabling or disabling wandb tracking based on the provided configuration.

In summary, the script is a utility for configuring wandb environment variables in a Python environment based on a provided configuration, enabling or disabling wandb tracking as needed. This is particularly useful in machine learning workflows where experiment tracking needs to be managed programmatically.

</details>


# Llama3 - Training Configuration

This is the default training configuration.  We will leave it as is for the time being.

```yaml
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
```

### <mark style="color:green;">Explanation of each configurations</mark>

<mark style="color:yellow;">`gradient_accumulation_steps: 1`</mark>

* Gradient accumulation is a technique used to simulate larger batch sizes without increasing memory usage.
* It allows the model to accumulate gradients over multiple forward passes before performing a single backward pass and updating the weights.
* Setting `gradient_accumulation_steps` to 1 means that the model will update its weights after every batch, effectively not using gradient accumulation.
* This is suitable when the available memory is sufficient to process the desired batch size without the need for accumulation.
* However, if memory is limited and you want to simulate a larger batch size, increasing this value can be beneficial.

<mark style="color:yellow;">`micro_batch_size: 2`</mark>

* The micro-batch size determines the number of samples processed in each forward pass of the model.
* In this case, setting `micro_batch_size` to 2 means that the model will process 2 samples at a time.
* Smaller micro-batch sizes can be useful when dealing with limited memory resources, as they require less memory per forward pass.
* However, smaller micro-batch sizes may result in slower training times and potentially noisier gradients.
* The optimal micro-batch size depends on the available hardware and the specific requirements of the training task.

<mark style="color:yellow;">`num_epochs: 4`</mark>

* The number of epochs represents the number of times the model will iterate over the entire training dataset.
* Setting `num_epochs` to 4 means that the model will go through the training data 4 times.
* Increasing the number of epochs can allow the model to learn more from the data and potentially improve its performance.
* However, training for too many epochs may lead to overfitting, where the model becomes too specialized to the training data and fails to generalize well to unseen data.
* The optimal number of epochs depends on factors such as the size and complexity of the dataset, the model architecture, and the learning rate.

<mark style="color:yellow;">`optimizer: adamw_torch`</mark>

* The optimizer is responsible for updating the model's weights based on the computed gradients.
* AdamW (Adam with Weight Decay) is an optimization algorithm that extends the original Adam optimizer by adding L2 regularization.
* AdamW is well-suited for training deep learning models and has been shown to perform well in various tasks.
* It adapts the learning rate for each parameter based on the historical gradients and incorporates weight decay to prevent overfitting.
* Using AdamW as the optimizer is a reasonable choice for training LLMs.

<mark style="color:yellow;">`adam_beta2: 0.95`</mark>

* `adam_beta2` is a hyperparameter of the AdamW optimizer that controls the exponential decay rate for the second moment estimates.
* It is typically set to a value close to 1 (e.g., 0.999) to give more importance to recent gradients.
* Setting `adam_beta2` to 0.95 means that the optimizer will give slightly less emphasis to recent gradients compared to the default value.
* The impact of this change may vary depending on the specific task and dataset.

<mark style="color:yellow;">`adam_epsilon: 0.00001`</mark>

* `adam_epsilon` is a small constant value used in the denominator of the AdamW optimizer's update rule to prevent division by zero.
* It helps to stabilize the optimization process and avoid numerical instability.
* The default value of `adam_epsilon` is usually 1e-8 or 1e-7.
* Setting it to 0.00001 (1e-5) is slightly larger than the typical default value, which may have a minor impact on the optimization process.

<mark style="color:yellow;">`max_grad_norm: 1.0`</mark>

* Gradient clipping is a technique used to prevent exploding gradients in deep neural networks.
* `max_grad_norm` sets the maximum L2 norm of the gradients. If the norm exceeds this value, the gradients are scaled down to meet the constraint.
* Setting `max_grad_norm` to 1.0 means that the gradients will be clipped if their L2 norm exceeds 1.0.
* Gradient clipping helps to stabilize the training process and prevents the gradients from becoming too large, which can lead to unstable or divergent behavior.
* The optimal value of `max_grad_norm` may vary depending on the model architecture and the specific training task.

<mark style="color:yellow;">`lr_scheduler: cosine`</mark>

* The learning rate scheduler determines how the learning rate changes over the course of training.
* The cosine scheduler, also known as cosine annealing, gradually decreases the learning rate following a cosine curve.
* It starts with a relatively high learning rate and slowly decreases it until it reaches a minimum value at the end of training.
* Cosine annealing can help the model converge more smoothly and potentially reach better local optima.
* It is a popular choice for learning rate scheduling and has been shown to be effective in various deep learning tasks.

<mark style="color:yellow;">`learning_rate: 0.000003`</mark>

* The learning rate determines the step size at which the model's weights are updated during training.
* Setting `learning_rate` to 0.000003 (3e-6) is a relatively small value.
* Smaller learning rates generally result in slower convergence but can lead to more stable and precise updates.
* The optimal learning rate depends on factors such as the model architecture, dataset complexity, and optimizer choice.
* It may be necessary to experiment with different learning rates to find the best value for a specific task.

The combination of AdamW optimizer with a cosine learning rate scheduler and gradient clipping can help stabilize the training process and potentially improve convergence.&#x20;

The micro-batch size and number of epochs should be adjusted based on the available hardware resources and the specific requirements of the training task. It's important to monitor the model's performance during training and make adjustments as needed to achieve the desired results.


# Llama3 - Data and Precision

These configurations allow you to control various aspects of the training process, such as data handling, precision settings, and hardware utilisation.&#x20;

```yaml
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
```

Let's go through each configuration and explain its purpose and implications:

#### <mark style="color:blue;">`train_on_inputs: false`</mark>

* This configuration determines whether to include or mask out the human's prompt from the training labels.
* When set to `false`, the model will not train on the human's prompt, meaning that the prompt will be excluded from the training labels.
* In other words, the model will only learn from the desired output or response and not from the input prompt.
* This is useful when you want the model to generate responses based on the given prompts without explicitly learning to reproduce the prompts themselves.
* By masking out the human's prompt, the model can focus on learning the mapping between the prompt and the desired output.

#### <mark style="color:blue;">`group_by_length: false`</mark>

* This configuration controls whether to group similarly sized data together to minimise padding during training.
* When set to `false`, the data will not be grouped by length and will be processed in the order it appears in the dataset.
* Grouping data by length can be beneficial when working with variable-length sequences, as it helps to reduce the amount of padding needed.
* Padding is the process of adding dummy tokens to shorter sequences to match the length of the longest sequence in a batch.
* By grouping similarly sized data together, you can minimize the amount of unnecessary padding, which can lead to more efficient memory usage and faster training.
* However, enabling `group_by_length` may result in slower data loading and preprocessing, as it requires downloading and sorting the entire dataset before training.
* It's also worth noting that when `group_by_length` is enabled, the training loss may exhibit an oscillating pattern due to the reordering of the data.

#### <mark style="color:blue;">`bf16: auto`</mark>

* This configuration relates to the use of BFloat16 (BF16) precision during training.
* BFloat16 is a 16-bit floating-point format that offers a wider dynamic range compared to the more common FP16 (Half-precision) format.
* When set to `auto`, the framework will automatically determine whether to use BF16 based on the available hardware and software support.
* If the hardware (e.g., GPU) and software (e.g., PyTorch version) support BF16, it will be used for training.
* BF16 can provide a good balance between computational efficiency and numeric precision, potentially leading to faster training times while maintaining model accuracy.
* However, the actual performance gains may vary depending on the specific hardware and model architecture.

#### <mark style="color:blue;">`fp16:`</mark>

* This configuration is related to the use of FP16 (Half-precision) during training, but in the provided configuration, it is left empty.
* FP16 is a 16-bit floating-point format that offers reduced precision compared to the standard FP32 (Single-precision) format.
* Using FP16 can help to reduce memory usage and accelerate training on certain hardware (e.g., NVIDIA GPUs with Tensor Cores).
* However, the empty value suggests that FP16 is not being explicitly enabled or configured in this case.

#### <mark style="color:blue;">`tf32: true`</mark>

* This configuration is specific to NVIDIA GPUs and relates to the use of TensorFloat-32 (TF32) precision.
* TF32 is a 19-bit floating-point format that is used by default on NVIDIA Ampere architecture GPUs (e.g., NVIDIA A100) for certain operations, such as matrix multiplications and convolutions.
* When set to `true`, TF32 will be used for supported operations on compatible hardware.
* TF32 offers a balance between performance and precision, providing faster computation compared to FP32 while maintaining similar accuracy.
* Enabling TF32 can lead to improved training speeds on NVIDIA Ampere GPUs without significant impact on model quality.


# Llama3 - Optimisations

These configuration options offer various techniques and optimizations to improve the training process&#x20;

Gradient checkpointing and Flash Attention focus on memory efficiency and computational speed, while early stopping and resuming from checkpoints are useful for preventing overfitting and managing the training workflow.&#x20;

Distributed training with `local_rank` enables parallel processing across multiple devices, and adjusting the logging frequency with `logging_steps` helps in monitoring the training progress.

We will use the generic configurations provided by Axolotl.

```yaml
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention: 
```

### <mark style="color:green;">Below is an analysis of each of the configurations</mark>

<mark style="color:yellow;">`gradient_checkpointing: true`</mark>

* Gradient checkpointing is a technique used to reduce the memory usage during training by trading off computation time.
* Instead of storing all the intermediate activations for the backward pass, gradient checkpointing selectively stores a subset of activations and recomputes the others when needed.
* Setting `gradient_checkpointing` to `true` enables this feature, which can be particularly beneficial when training large models with limited memory.
* By enabling gradient checkpointing, you can potentially train larger models or use larger batch sizes without running out of memory.
* However, it's important to note that gradient checkpointing introduces additional computational overhead, as the model needs to recompute the activations during the backward pass.
* The decision to use gradient checkpointing depends on the available memory and the trade-off between memory usage and training speed.

<mark style="color:yellow;">`gradient_checkpointing_kwargs:`</mark>

* This configuration allows you to specify additional keyword arguments for gradient checkpointing.
* In the provided example, `use_reentrant: True` is specified as a keyword argument.
* The `use_reentrant` flag is related to the implementation of gradient checkpointing in PyTorch.
* When set to `True`, it enables the use of reentrant autograd functions, which can provide additional memory savings during gradient checkpointing.
* However, the specific behavior and impact of this flag may depend on the PyTorch version and the model architecture.

<mark style="color:yellow;">`early_stopping_patience:`</mark>

* Early stopping is a technique used to prevent overfitting and improve generalization performance.
* It monitors a validation metric (e.g., validation loss or accuracy) during training and stops the training process if the metric does not improve for a specified number of iterations (patience).
* The `early_stopping_patience` configuration allows you to set the number of iterations to wait before early stopping is triggered.
* For example, if `early_stopping_patience` is set to 3, training will stop if the validation metric does not improve for 3 consecutive iterations.
* Early stopping helps to avoid wasting computational resources on training iterations that do not lead to further improvements and can help prevent the model from overfitting to the training data.

<mark style="color:yellow;">`resume_from_checkpoint:`</mark>

* This configuration allows you to resume training from a specific checkpoint.
* By specifying a checkpoint directory or file path, you can load the model state, optimizer state, and other necessary information to continue training from where it left off.
* Resuming from a checkpoint can be useful in various scenarios, such as when training is interrupted due to system failures, when you want to fine-tune a pre-trained model, or when you want to experiment with different hyperparameters starting from a previously trained model.
* It saves time and resources by avoiding the need to start training from scratch.

<mark style="color:yellow;">`local_rank:`</mark>

* The `local_rank` configuration is related to distributed training, specifically when using techniques like Data Parallel or Distributed Data Parallel.
* In distributed training, multiple GPUs or machines are used to parallelise the training process and speed up computations.
* The `local_rank` represents the unique identifier of a process within a distributed training setup.
* It is typically used to determine the device placement and communication patterns among the processes.
* When using distributed training frameworks like PyTorch's `DistributedDataParallel`, the `local_rank` is automatically set by the framework.

<mark style="color:yellow;">`logging_steps: 1`</mark>

* The `logging_steps` configuration determines the frequency at which training logs and metrics are recorded.
* In this case, setting `logging_steps` to 1 means that logs will be generated after every training step.
* Logging can include information such as the current training loss, learning rate, elapsed time, and other relevant metrics.
* More frequent logging can be useful for monitoring the training progress and identifying any potential issues early on.
* However, generating logs after every step can also introduce overhead and slow down the training process, especially for large datasets or long training runs.
* The logging frequency should be adjusted based on the specific needs and the scale of the training task.

<mark style="color:yellow;">`xformers_attention`</mark>

* The `xformers_attention` configuration is related to the use of the XFormers library, which provides optimized attention implementations for transformers.
* XFormers offers various attention mechanisms, such as memory-efficient attention, that can speed up training and reduce memory usage compared to the standard attention implementation in PyTorch.
* Setting `xformers_attention` to a specific value (not provided in the given configuration) would enable the use of XFormers attention in the model.
* The specific attention mechanism and its parameters would depend on the value provided for `xformers_attention`.
* Using XFormers attention can be beneficial for training large models or when dealing with long sequences, as it can provide computational and memory efficiency improvements.

<mark style="color:yellow;">`flash_attention: true`</mark>

* Flash Attention is a highly optimised attention implementation that can significantly speed up the training of transformers.
* It is designed to be memory-efficient and can handle large sequence lengths and batch sizes.
* Setting `flash_attention` to `true` enables the use of Flash Attention in the model.
* Flash Attention can provide substantial performance improvements, especially for models with a large number of attention heads and long sequences.
* It achieves this by using techniques like kernel fusion, memory optimization, and efficient parallelization.
* Enabling Flash Attention can help reduce training time and allow for training larger models or using larger batch sizes.


# Llama3 - Extra Hyperparameters

This is the balance of the configuration file for the training of Phi 2.0.   We provide a full explanation of each of the configurations below.

```yaml
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
   pad_token: <|end_of_text|>
```

### <mark style="color:blue;">**Training Hyperparameters**</mark>

<mark style="color:yellow;">`warmup_steps: 100`</mark>

Warm-up steps are a crucial part of the learning rate scheduling.&#x20;

Over the first 100 training steps, the learning rate incrementally increases to its target value. This gradual increase helps in stabilizing the training early on, preventing the model from making too large updates too quickly.

<mark style="color:yellow;">`evals_per_epoch: 4`</mark>

This setting determines the frequency of evaluations within each training epoch. With a value of 4, the model will be evaluated four times per epoch, providing regular feedback on its performance. Frequent evaluations help in monitoring the model's progress and ensuring it is learning as expected.

<mark style="color:yellow;">`saves_per_epoch: 1`</mark>

To safeguard your training progress, the model's state is saved once every epoch. This checkpointing allows you to resume training from the last saved state in case of interruptions and also provides opportunities for model fine-tuning.

### <mark style="color:blue;">**Deepspeed and FSDP**</mark>

<mark style="color:yellow;">`deepspeed:`</mark>

DeepSpeed integration offers advanced optimisations for accelerating training and reducing memory consumption. Configuring DeepSpeed enhances training efficiency, particularly for large-scale models, by optimizing computational resources and parallelizing the workload.

<mark style="color:yellow;">`fsdp_config:`</mark>

Fully Sharded Data Parallel (FSDP) is a technique to reduce memory consumption and increase the scale of distributed training. The `fsdp_config` allows you to customize FSDP's behavior, optimizing memory usage and computational efficiency across multiple devices.

<mark style="color:yellow;">`weight decay: 1`</mark>

Weight decay is a regularization technique to prevent overfitting by penalizing large weights. A weight decay factor of 0.1 helps in moderating the update of weights, encouraging the model to learn more general features rather than overly fitting to the training data.

### <mark style="color:blue;">**Special Tokens and Token Embeddings**</mark>

<mark style="color:yellow;">`resize_token_embeddings_to_32x:`</mark>

This configuration enables the resizing of token embeddings, a feature particularly useful for adapting the model to different vocabulary sizes efficiently. Resizing to 32 times the original size allows for more flexible and scalable model architecture.

<mark style="color:yellow;">`pad_token`</mark>

Special tokens play a pivotal role in how the model processes and understands text. The padding token (`pad_token`) is used to fill out sequences to a uniform length, ensuring consistent input size for the model.


# Llama3- All Configurations

This is a summary of the entire configuration file for the fine tuning run of Llama3 usijng 8bit Lora PEFT.

```yaml
base_model: meta-llama/Meta-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: true
load_in_4bit: false
strict: false

datasets:
  - path: datasets/alpagasus/data/train-00000-of-00001-0c59455170918204.parquet
    type: alpaca
    ds_type: parquet
    data_files:
  - train-00000-of-00001-0c59455170918204.parquet
dataset_prepared_path:
val_set_size: 0.10
output_dir: ./llama3-out

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project: llama3-alpagasus
wandb_entity: continuum-labs
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention:

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
   pad_token: <|end_of_text|>
```

The training configuration file looks mostly fine, but there are a few things that could be adjusted or clarified:

1. The `dataset_prepared_path` is empty. If you want to use a prepared dataset, you should provide a path here.
2. The `lora_target_modules` parameter is missing. You might want to specify which modules to apply LoRA to, such as `q_proj`, `v_proj`, `k_proj`, `o_proj`, etc.
3. The `wandb_watch`, `wandb_name`, and `wandb_log_model` parameters are empty. If you want to use Weights & Biases for logging and tracking, you should provide appropriate values for these parameters.
4. The `bf16` parameter is set to `auto`. Make sure your hardware supports bfloat16 if you want to use it. Otherwise, you can set it to `false`.
5. The `early_stopping_patience`, `resume_from_checkpoint`, and `local_rank` parameters are empty. If you want to use early stopping, resume from a checkpoint, or perform distributed training, you should provide appropriate values for these parameters.
6. The `eval_table_size` and `eval_max_new_tokens` parameters are present but not typically used in the training configuration. These parameters are usually used during evaluation or inference.
7. The `fsdp` and `fsdp_config` parameters are empty. If you want to use FullyShardedDataParallel for distributed training, you should provide appropriate configurations.
8. The `special_tokens` parameter only specifies the `pad_token`. If your model requires additional special tokens, such as `bos_token`, `eos_token`, or custom tokens, you should add them here.

Other than these points, the configuration looks valid. Make sure to double-check the paths to your dataset and output directory, and ensure that the hyperparameters are suitable for your specific training task and hardware setup.

<br>


# Llama3 - Preprocessing

To execute the training run, we will first <mark style="color:yellow;">preprocess the dataset.</mark>&#x20;

Axolotl allows us to optionally <mark style="color:blue;">**pre-tokenize dataset**</mark> before finetuning. This is recommended for large datasets.&#x20;

### <mark style="color:blue;">Populate the config.yaml file</mark>

To execute the preprocessing, we have to make sure the <mark style="color:yellow;">datasets component</mark> of the config.yaml is correctly populated. &#x20;

You will need to <mark style="color:yellow;">**direct Axolotl to where your dataset is**</mark>.

To do this, find the path to your dataset in VS Code by right clicking on the dataset and asking for the <mark style="color:yellow;">relative path.</mark>  Then enter it into the config YAML file below which is located at:

### *your directory*/<mark style="color:blue;">axolotl</mark>/<mark style="color:green;">examples</mark>/<mark style="color:purple;">llama-3</mark>/<mark style="color:yellow;">lora-8b.yml</mark>

```bash
your directory/axolotl/examples/llama-3/lora-8b.yml
```

When you have located the file, then <mark style="color:yellow;">populate the configuration file</mark> below.

To do this, find the <mark style="color:yellow;">path to your dataset</mark> in VS Code by right clicking on the dataset and asking for the relative URL.  Use this in the command below.  <mark style="color:yellow;">Remember to include the file name.</mark>

```yaml
datasets:
  - path: datasets/alpagasus/data/train-00000-of-00001-0c59455170918204.parquet
    type: alpaca
    ds_type: parquet
    data_files:
  - train-00000-of-00001-0c59455170918204.parquet
dataset_prepared_path:
val_set_size: 0.10
output_dir: ./llama3-out
```

The data set type (ds\_type) is Parquet, and we have arbitrarily <mark style="color:yellow;">set the output directory as ./llama3-out</mark>

For a refresher of what the Alpgasus data set contains

{% embed url="<https://github.com/gpt4life/alpagasus/tree/main>" %}

Once you have configured the dataset component of the YAML file, then execute the following command which uses the <mark style="color:yellow;">preprocess.py script</mark> within the Axolotl library. &#x20;

{% code fullWidth="false" %}

```bash
python -m axolotl.cli.preprocess examples/llama-3/lora-8b.yml
```

{% endcode %}

The output should indicate success:

```bash
Success! Preprocessed data path: `dataset_prepared_path: ./llama-data`
```

If you have not set a dataset\_prepared\_path in the configuration file, it will default to:

"preprocess CLI called without dataset\_prepared\_path set, using default path: <mark style="color:yellow;">last\_run\_prepared"</mark>

For an <mark style="color:yellow;">analysis of the output</mark> relating to the dataset preprocessing, see below:

<details>

<summary><mark style="color:green;">Analysis of Output</mark></summary>

The numbers:

1. **PID (Process ID)**: `6779`
   * This is the ID of the process running the script.
2. **Token IDs**:
   * End Of Sentence (EOS) Token ID: `2`
   * Beginning Of Sentence (BOS) Token ID: `1`
   * Padding (PAD) Token ID: `0`
   * Unknown (UNK) Token ID: `0`
3. **Data Downloading and Processing**:
   * Number of data files downloaded: `1`
   * Download speed: `12557.80 items/second`
   * Number of data files extracted: `1`
   * Extraction speed: `214.70 items/second`
4. **Dataset Generation**:
   * Number of examples in the train split: `51760`
   * Generation speed: `85351.57 examples/second`
5. **Mapping and Filtering**:
   * Number of examples processed in mapping (num\_proc=12): `51760`
   * Processing speed in mapping: `6035.04 examples/second`
   * Number of examples processed in filtering (num\_proc=12): `51760`
   * Filtering speed: `41994.96 examples/second`
   * Number of examples processed in the second mapping (num\_proc=12): `51760`
   * Second mapping speed: `30435.07 examples/second`
6. **Token Counts**:
   * Total number of tokens: `12104896`
   * Total number of supervised tokens: `8475133`
7. **Efficiency Estimates and Data Loading**:
   * Packing efficiency estimate: `1.0`
   * Total number of tokens per device: `12104896`
   * Data loader length: `1461`
   * Sample packing efficiency estimate across ranks: `0.9798729691644562`
   * Sample packing efficiency estimate: `0.98`
   * Total number of steps: `5844`
8. **Time and Date Information**:
   * Date and time of the log entries: `2023-12-06`
   * Times of various log entries: `05:24:22,428`, `05:24:22,688`, `05:24:32,282`, `05:24:36,028`, `05:24:36,393`, `05:24:41,028`, `05:24:41,029`, `05:24:41,037`
   * Total execution time: `26 seconds`
   * Time when the command prompt was ready again: `05:24:42`

</details>

If you would like information on how the preprocess.py script works you can view the code at   <mark style="color:purple;">axolotl</mark>/<mark style="color:green;">src</mark>/<mark style="color:blue;">axolotl</mark>/<mark style="color:yellow;">cli.</mark> &#x20;

Alternatively, take the time to read our analysis of the scripts, and the output it produced:

<details>

<summary><mark style="color:green;">Axolotl preprocess.py -</mark> <mark style="color:yellow;">analysis of script</mark> <mark style="color:green;">and</mark> <mark style="color:yellow;">command output</mark></summary>

The provided <mark style="color:yellow;">`preprocess.py`</mark> script is a <mark style="color:yellow;">command-line interface (CLI) tool</mark> for preprocessing datasets in the context of training models using the Axolotl platform.&#x20;

<mark style="color:green;">**Imports and Logger Setup**</mark>

* The script imports necessary modules such as <mark style="color:yellow;">`logging`</mark><mark style="color:yellow;">,</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`pathlib.Path`</mark> for file path operations, <mark style="color:yellow;">`fire`</mark> for the CLI, and several modules from <mark style="color:yellow;">`transformers`</mark> and <mark style="color:yellow;">`axolotl`</mark>.
* <mark style="color:yellow;">`colorama`</mark> is used for colored console output, enhancing readability.
* A logger `LOG` is set up for logging purposes, using the `logging` module.

<mark style="color:green;">**do\_cli Function**</mark><mark style="color:green;">:</mark>

<mark style="color:green;">**Load Configurations**</mark><mark style="color:green;">:</mark>   The main function <mark style="color:yellow;">`do_cli`</mark> takes a <mark style="color:yellow;">`config`</mark> argument (with a <mark style="color:yellow;">default path of</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`"examples/"`</mark>) and <mark style="color:yellow;">`**kwargs`</mark> for additional arguments. &#x20;

* These configurations include essential details about the dataset, model, and training setup. This step is crucial as it sets up the parameters for how the dataset will be processed and how the model training will proceed.
* The function <mark style="color:yellow;">`load_datasets`</mark> is called with the loaded configuration and parsed command-line arguments. This function is responsible for the heavy lifting of loading the dataset (which could include downloading it if not already present locally) and preprocessing it according to the specifications in the configuration. This step is central to preparing the data for model training.
* **ASCII Art and Configuration Loading**: It starts by printing ASCII art using <mark style="color:yellow;">`print_axolotl_text_art`</mark> for visual appeal. This is more for aesthetic purposes and has no impact on the functional aspects of the scrip
* Then, <mark style="color:yellow;">it loads the configuration file</mark> using <mark style="color:yellow;">`load_cfg`</mark>, which sets up the parameters for dataset preprocessing.  &#x20;
* &#x20;Initially, there's a deprecation warning from the <mark style="color:yellow;">`transformers.deepspeed`</mark> module. This suggests that in future versions of the Transformers library, you'll <mark style="color:yellow;">need to import DeepSpeed modules differently</mark>.&#x20;
* **Accelerator and User Token Checks**: It ensures that the default accelerator configuration is set and validates the user token for authentication or API access.
* **CLI Arguments Parsing**: The script uses <mark style="color:yellow;">`transformers.HfArgumentParser`</mark> to parse additional command-line arguments specific to preprocessing, defined in <mark style="color:yellow;">`PreprocessCliArgs`</mark>.
* **Dataset Preparation Path Check**: It checks if <mark style="color:yellow;">`dataset_prepared_path`</mark> is set in the configuration.  If the <mark style="color:yellow;">`dataset_prepared_path`</mark> is not explicitly set in the configuration, the script issues a warning and *<mark style="color:yellow;">assigns a default pat</mark>*<mark style="color:yellow;">h (</mark><mark style="color:yellow;">`DEFAULT_DATASET_PREPARED_PATH`</mark><mark style="color:yellow;">).</mark> This path is where the script will store the preprocessed dataset, ensuring that there’s always a defined location for this data.
* **Load and Preprocess Datasets**: The <mark style="color:yellow;">`load_datasets`</mark> function is called with the loaded configuration and parsed CLI arguments, handling the dataset loading and preprocessing.
* The script provides debug <mark style="color:yellow;">information about special tokens</mark> (End Of Sentence, Beginning Of Sentence, Padding, and Unknown) used by the tokenizer. This information is crucial for understanding how the model will interpret different types of tokens in the data.

<mark style="color:green;">Logging Success</mark>

* Upon successful completion, the <mark style="color:yellow;">script logs a message indicating the path</mark> where the preprocessed data is stored. This is done using the <mark style="color:yellow;">`LOG.info`</mark> method, and the message is colored green for visibility.

<mark style="color:green;">**Main Block**</mark>

* This block checks if the script is the main program being run <mark style="color:yellow;">(</mark><mark style="color:yellow;">`__name__ == "__main__"`</mark><mark style="color:yellow;">)</mark> and not a module imported in another script. If it is the main program, it uses <mark style="color:yellow;">`fire.Fire(do_cli)`</mark> to enable the script to be run from the command line, where <mark style="color:yellow;">`do_cli`</mark> is the main function being executed.

<mark style="color:blue;">**OUTPUT**</mark>

1. <mark style="color:green;">**Warnings and ASCII Art**</mark><mark style="color:green;">:</mark>
2. <mark style="color:green;">**Token Information**</mark><mark style="color:green;">:</mark> The script provides debug <mark style="color:yellow;">information about special tokens</mark> (End Of Sentence, Beginning Of Sentence, Padding, and Unknown) used by the tokenizer. This information is crucial for understanding how the model will interpret different types of tokens in the data.
3. <mark style="color:green;">**Data File Processing**</mark><mark style="color:green;">:</mark> The script downloads and extracts data files, then generates a training split with 51,760 examples. This is followed by mapping and filtering operations on the dataset, performed in parallel (noted by `num_proc=12`), which is an efficient way to handle large datasets.
4. <mark style="color:green;">**Dataset Merging and Saving**</mark><mark style="color:green;">:</mark> Post-processing, the datasets are merged and saved to disk. This is an essential step to ensure that the processed data is stored in a format ready for model training.
5. <mark style="color:green;">**Token Counts and Sample Packing**</mark><mark style="color:green;">:</mark> The script calculates the total number of tokens and supervised tokens. It also estimates the packing efficiency and total number of steps needed for training, which are critical for understanding how the data will be batched and fed into the model during training.
6. <mark style="color:green;">**Final Information and Path**</mark><mark style="color:green;">:</mark> Finally, the script concludes with a success message and provides the path to the preprocessed data. This is your confirmation that the preprocessing was successful and where the prepared data is stored.

</details>

This script for our Axolotl platform will be:&#x20;

{% hint style="warning" %}
Flash Attention Dependency Issues
{% endhint %}

### <mark style="color:blue;">Analysing arrow files</mark>

<details>

<summary><mark style="color:green;">Script to analyse arrow file</mark></summary>

```python
import pyarrow as pa  # Import the PyArrow library
import pandas as pd  # Import the Pandas library
import matplotlib.pyplot as plt  # Import the Matplotlib library

# Load the Arrow file
arrow_file_path = '/home/paperspace/axolotl/prepared_data/10cde2a06a273512ed560a21f9744220/data-00000-of-00001.arrow'

try:
    # Try using open_stream to read the file
    with pa.ipc.open_stream(arrow_file_path) as stream:  # Open the Arrow file
        table = stream.read_all()  # Read the entire Arrow file
except Exception as e:  # Handle any exceptions
    print(f"Failed to read the Arrow file: {e}")  # Print the error message
    exit(1)  # Exit the program

# Convert to a Pandas DataFrame
df = table.to_pandas()  # Convert the Arrow table to a Pandas DataFrame

# Display the first few rows of the DataFrame
print("First few rows of the DataFrame:")  
print(df.head())  # Display the first few rows of the DataFrame

# General statistics for 'length' column
print("\nStatistics for 'length' column:")     
print(df['length'].describe())  # Display the statistics for the 'length' column  

# Check unique values and counts for 'labels' if applicable
if 'labels' in df.columns:  # Check if the 'labels' column exists in the DataFrame
    print("\nUnique values and counts for 'labels':")             
    label_counts = df['labels'].apply(lambda x: pd.Series(x)).stack().value_counts()  # Count the unique values in the 'labels' column
    print(label_counts)  # Display the unique values and counts for the 'labels' column
else:
    print("\nNo 'labels' column found in the DataFrame.")  # Print a message if the 'labels' column is not found

# Visualize distribution of sequence lengths 
plt.figure(figsize=(10, 6))  # Set the figure size      
plt.hist(df['length'], bins=20, edgecolor='black')  # Create a histogram of the 'length' column
plt.title('Distribution of Sequence Lengths')  # Set the title of the plot
plt.xlabel('Length of Sequences')  # Set the x-axis label
plt.ylabel('Frequency')  # Set the y-axis label
plt.tight_layout()  # Adjust the layout
plt.show()  # Display the plot

print("\nDistribution of sequence lengths:")  # Print a message
print(f"Mean: {df['length'].mean()}")  # Print the mean of the 'length' column
print(f"Median: {df['length'].median()}")   # Print the median of the 'length' column
print(f"Percentiles: {df['length'].describe(percentiles=[0.25, 0.5, 0.75, 0.9, 0.95, 0.99])}")  # Print the percentiles of the 'length' column

# Analyze the token frequency
print("\nToken frequency analysis:")  # Print a message
token_freq = df['input_ids'].apply(pd.Series).stack().value_counts()  # Count the frequency of tokens
print(f"Most common tokens: {token_freq.head(10)}")  # Display the most common tokens
print(f"Least common tokens: {token_freq.tail(10)}")  # Display the least common tokens

# Examine the 'labels' column
print("\nLabel distribution:")  # Print a message
label_counts = df['labels'].apply(lambda x: pd.Series(x)).stack().value_counts()  # Count the frequency of labels
print(label_counts)  # Display the label distribution

# Randomly sample rows for manual inspection
print("\nRandom sample of preprocessed data:")  # Print a message
sample_rows = df.sample(n=50, random_state=42)  # Randomly sample 50 rows
for _, row in sample_rows.iterrows():  # Iterate over the sampled rows
    print(f"Input IDs: {row['input_ids']}")   # Print the 'input_ids' column
    print(f"Labels: {row['labels']}")   # Print the 'labels' column
    print(f"Attention Mask: {row['attention_mask']}")   # Print the 'attention_mask' column
    print(f"Position IDs: {row['position_ids']}")  # Print the 'position_ids' column
    print("---")
```

</details>

To ensure that the data is fit for fine-tuning, you can perform a more comprehensive analysis.&#x20;

Here are some additional steps you can take:

<mark style="color:green;">**Check the distribution of sequence lengths**</mark>

* Calculate the mean, median, and percentiles of the 'length' column.
* Plot a histogram or density plot of the 'length' column to visualize the distribution.
* Identify if there are any outliers or extreme values in the sequence lengths.

<mark style="color:green;">**Analyze the token frequency**</mark>

* Flatten the 'input\_ids' column and create a frequency distribution of the tokens.
* Identify the most common and least common tokens.
* Check if the token distribution aligns with your expectations based on the original dataset.

<mark style="color:green;">**Examine the 'labels' column**</mark>

* Understand the meaning of the -100 label and how it is used in your fine-tuning task.
* Calculate the distribution of unique labels and their counts.
* Check if the label distribution is balanced or imbalanced, and consider if any class balancing techniques are needed.

<mark style="color:green;">**Validate the 'attention\_mask' and 'position\_ids' columns**</mark>

* Ensure that the 'attention\_mask' column correctly corresponds to the non-padding tokens in 'input\_ids'.
* Verify that the 'position\_ids' column correctly represents the position of each token in the sequence.

<mark style="color:green;">**Assess the quality of the preprocessed data**</mark>

* Randomly sample a subset of rows from the DataFrame and manually inspect the tokenized sequences.
* Check if the tokenization aligns with the expected format and content of the original instructions.
* Look for any anomalies, such as truncated sequences, incorrect tokenization, or missing information.

<mark style="color:green;">**Evaluate the data split**</mark>

* If your Arrow file represents the entire dataset, consider splitting it into train, validation, and test subsets.
* Ensure that the data split is representative and stratified based on important characteristics, such as sequence length or label distribution.

<mark style="color:green;">**Consider domain-specific analysis**</mark>

* Depending on the nature of your instructions dataset, perform domain-specific analysis.
* For example, if the instructions involve specific tasks or categories, analyze the distribution of those tasks or categories within the dataset.

<details>

<summary><mark style="color:green;">Analysis of arrow file</mark></summary>

1. DataFrame Structure:
   * The DataFrame has columns: 'input\_ids', 'attention\_mask', 'labels', 'position\_ids', and 'length'.
   * Each row represents a preprocessed data sample.
2. First Few Rows:
   * The first few rows provide a glimpse of the preprocessed data.
   * 'input\_ids' contains the tokenized input sequences.
   * 'attention\_mask' indicates which tokens should be attended to (1) and which should be ignored (0).
   * 'labels' contains the corresponding labels for each token (-100 is used for padding or non-labeled tokens).
   * 'position\_ids' represents the position of each token in the sequence.
   * 'length' indicates the length of each sequence.
3. Statistics for 'length' Column:
   * The DataFrame has 9,229 samples.
   * The average sequence length is 113.66, with a standard deviation of 61.26.
   * The minimum sequence length is 33, and the maximum is 1,021.
   * The 25th, 50th (median), and 75th percentiles of sequence lengths are 71, 103, and 138, respectively.
4. Unique Values and Counts for 'labels':
   * The 'labels' column contains a large number of unique labels (28,522).
   * The most common label is -100, which likely represents padding or non-labeled tokens.
   * Other frequent labels include 11.0, 13.0, 323.0, and 279.0.
   * There are many labels with very low counts, possibly indicating rare or unique tokens.
5. Distribution of Sequence Lengths:
   * The mean sequence length is 113.66, and the median is 103.0.
   * The 25th, 50th, 75th, 90th, 95th, and 99th percentiles provide insights into the distribution of sequence lengths.
   * The majority of sequences have lengths between 71 and 138 (interquartile range).
6. Token Frequency Analysis:
   * The most common tokens include 264.0, 13.0, 279.0, 11.0, and 430.0.
   * The least common tokens have very low frequencies, possibly representing rare or unique tokens.
7. Label Distribution:
   * The label distribution shows the frequency of each label in the dataset.
   * -100.0 is the most common label, likely representing padding or non-labeled tokens.
   * Other frequent labels include 11.0, 13.0, 323.0, and 279.0.
8. Random Sample of Preprocessed Data:
   * The script randomly selects a few samples to provide a more detailed view of the preprocessed data.
   * Each sample shows the 'input\_ids', 'labels', 'attention\_mask', and 'position\_ids'.
   * This allows for manual inspection and validation of the preprocessing steps.

Overall, the analysis suggests that the Arrow file contains preprocessed data suitable for fine-tuning a large language model.&#x20;

The sequences have varying lengths, with an average of around 114 tokens. The 'labels' column has a large number of unique labels, indicating a diverse set of target tokens. The token frequency analysis shows a skewed distribution, with some tokens being very common while others are rare.

To further assess the suitability of the data for fine-tuning, you may want to:

* Examine the specific meanings of the labels and ensure they align with your task requirements.
* Validate that the preprocessing steps, such as tokenization and label assignment, are performed correctly.
* Consider the balance of the label distribution and whether any class imbalance needs to be addressed.
* Evaluate the quality and diversity of the input sequences to ensure they adequately represent the desired task domain.

Remember to adapt the analysis based on your specific use case and task requirements. The provided script offers a solid foundation for data exploration and can be extended as needed.

</details>

### <mark style="color:blue;">Suggestions</mark>

<mark style="color:green;">Examine the specific meanings of the labels</mark>

* The 'labels' column contains a large number of unique labels, with -100 being the most common.
* It's crucial to understand the meaning of these labels in the context of your task.
* The -100 label likely represents padding or non-labeled tokens, while other labels correspond to specific target tokens.
* You should ensure that the labels align with your task requirements and represent the desired output or target sequence.
* If the labels don't match your expectations or aren't suitable for your task, you may need to modify the preprocessing steps or the labeling scheme.

<mark style="color:green;">Validate the preprocessing steps</mark>

* The preprocessing steps involve tokenization and label assignment.
* Check if the tokenization process is performed correctly and matches the expected tokenization scheme for your model.
* Verify that the 'input\_ids' column contains the correct token IDs corresponding to the input sequences.
* Ensure that the 'attention\_mask' column accurately represents the padding and non-padding tokens.
* Validate that the 'labels' column is correctly aligned with the 'input\_ids' and contains the expected target labels.
* If any issues are found in the preprocessing steps, you may need to revisit and modify the preprocessing code.

<mark style="color:green;">Consider the balance of the label distribution</mark>

* The label distribution shows a significant class imbalance, with some labels being much more frequent than others.
* Class imbalance can impact the model's performance, as it may bias towards the majority class.
* Consider whether the class imbalance is expected and acceptable for your task, or if it needs to be addressed.
* Techniques like oversampling minority classes, undersampling majority classes, or using class weights can help mitigate class imbalance.
* Evaluate the impact of class imbalance on your model's performance and consider applying appropriate techniques if necessary.

<mark style="color:green;">Evaluate the quality and diversity of the input sequences</mark>

* Randomly sample a subset of input sequences and manually inspect them.
* Assess the quality of the sequences in terms of coherence, relevance to your task, and any potential noise or artifacts.
* Check if the sequences adequately represent the desired task domain and cover a diverse range of examples.
* Look for any patterns, anomalies, or issues in the input sequences that may affect the model's learning.
* If the input sequences are of poor quality or lack diversity, you may need to gather additional data or refine the data collection process.

Based on your specific use case and task requirements, you should adapt the analysis accordingly. Consider the following:

* Define clear criteria for assessing label suitability and alignment with your task.
* Establish validation steps to ensure the preprocessing pipeline is performing as expected.
* Determine acceptable levels of class imbalance and consider strategies to handle it.
* Set quality standards and diversity requirements for the input sequences based on your task domain.

Remember, the provided script serves as a starting point for data exploration, and you can extend it to include additional analysis or validation steps specific to your use case.

By thoroughly examining the labels, validating the preprocessing steps, addressing class imbalance, and evaluating the quality and diversity of the input sequences, you can gain confidence in the suitability of your data for fine-tuning your large language model.

### <mark style="color:red;">Flash Attention Issues</mark>

We had some issue with Flash Attention dependencies.  For some reason the axolotl environment keep reverting to Pytorch 2.01 - Flash Attention needs Pytorch version 2.10

We executed this command in the axolotl library, and it upgraded Pytorch to 2.10.

```bash
pip install flash_attn -U --force-reinstall
```

<details>

<summary>Flash Attention Debugging</summary>

Flash Attention needs Pytorch 2.10 - not 2.0.  I had to force install Flash Attention to make it work:\
\
The error message you're encountering originates from running a Python script that attempts to preprocess a configuration file for a machine learning model using the <mark style="color:yellow;">`axolotl.cli.preprocess`</mark> module. The error is an <mark style="color:yellow;">`ImportError`</mark> associated with the `flash_attn_2_cuda` library, which is part of the `flash_attn` package. Let's break down the error message line by line:

1. **Command Executed**: <mark style="color:yellow;">`python -m axolotl.cli.preprocess your directory/axolotl/examples/llama-2/lora.yml`</mark>
   * This command attempts to run a Python module (<mark style="color:yellow;">`axolotl.cli.preprocess`</mark>) with a specific configuration file as its argument.
2. **Initial Traceback**:
   * The Python interpreter starts by <mark style="color:yellow;">trying to execute the module but encounters an issue in the process.</mark>
3. **Import Chain**:
   * The error arises in a chain of imports starting from your script's initial import statement. Python attempts to import various modules and packages needed for the script to run.
4. **Specific ImportError**:
   * The final error, <mark style="color:yellow;">`ImportError`</mark>, occurs when Python tries to import <mark style="color:yellow;">`flash_attn_2_cuda`</mark> from the <mark style="color:yellow;">`flash_attn`</mark> package.
   * The error message specifically states `undefined symbol: _ZN3c104cuda9SetDeviceEi`. This indicates a missing or incompatible symbol in the compiled CUDA extension.
5. <https://github.com/Dao-AILab/flash-attention/issues/620>

#### Troubleshooting Steps:

1. **Verify CUDA Installation**:
   * Ensure that CUDA is correctly installed on your system and is compatible with the versions required by `flash_attn`.
   * Check the CUDA version against the version required by <mark style="color:yellow;">`flash_attn`</mark>. If there's a mismatch, consider upgrading or downgrading CUDA.
2. **Check PyTorch and flash\_attn Compatibility**:
   * Verify that the installed PyTorch version is compatible with `flash_attn`. Incompatibilities between CUDA, PyTorch, and `flash_attn` can lead to such errors.
3. **Reinstall flash\_attn**:
   * Try reinstalling the `flash_attn` package. Sometimes, recompiling the package can resolve symbol mismatch errors.
   * Use the command <mark style="color:blue;">`pip install flash-attn --force-reinstall`</mark> to reinstall.
4. **Check Environment Variables**:
   * Ensure that your <mark style="color:yellow;">`LD_LIBRARY_PATH`</mark> environment variable includes the paths to the CUDA libraries.
5. **Inspect Python Environment**:
   * Verify that you are using the correct Python environment where all required dependencies are installed. Sometimes, conflicts in environments can cause such issues.
6. **Check for System Updates**:
   * Occasionally, system updates or changes to the compiler or related libraries can cause incompatibilities. Ensure your system is up to date.
7. **Consult Documentation or Community Forums**:
   * Check the documentation for `flash_attn` and related packages for any known issues or troubleshooting tips.
   * Seek help from relevant community forums or issue trackers for `flash_attn` and related projects.
8. **Test in a Clean Environment**:
   * If possible, try to run the script in a clean Python environment with only necessary packages installed. This can help rule out conflicts with other packages.

</details>

<details>

<summary><mark style="color:green;">Using the Upload File Function</mark></summary>

The <mark style="color:yellow;">`upload_file`</mark> function from the Hugging Face Hub library is used to upload files to a repository on the Hugging Face Hub. This function is quite versatile, supporting various parameters to customize the upload process. Here's a breakdown of the parameters and what they mean:

1. <mark style="color:blue;">**path\_or\_fileobj (str, Path, bytes, or IO)**</mark><mark style="color:blue;">:</mark> This is the source of the file you want to upload. It can be a <mark style="color:yellow;">path to a file on your local machine</mark>, a binary data stream, a file object, or a buffer.
2. <mark style="color:blue;">**path\_in\_repo (str)**</mark><mark style="color:blue;">:</mark> This specifies where in the repository the file should be placed. For example, if you want to place a file in a folder called 'checkpoints', you would use something like `"checkpoints/weights.bin"`.
3. <mark style="color:blue;">**repo\_id (str)**</mark><mark style="color:blue;">:</mark> The identifier for the repository to which you are uploading. This usually follows the format `"username/repository-name"`.
4. <mark style="color:blue;">**token (str, optional)**</mark><mark style="color:blue;">:</mark> Your authentication token for the Hugging Face Hub. If you have already logged in using `HfApi.login`, this will default to the stored token. If not provided, the function will attempt to use a stored token from a previous login.
5. <mark style="color:blue;">**repo\_type (str, optional)**</mark><mark style="color:blue;">:</mark> Indicates the type of repository. It can be `"dataset"`, `"space"`, or `"model"`. If you are uploading to a dataset or a space, specify accordingly. The default is `None`, which is interpreted as a model.
6. <mark style="color:blue;">**revision (str, optional)**</mark><mark style="color:blue;">:</mark> Specifies the git revision (like a branch name or commit hash) from which the commit should be made. The default is the head of the "main" branch.
7. <mark style="color:blue;">**commit\_message (str, optional)**</mark><mark style="color:blue;">:</mark> A short summary or title for the commit you are making. Think of this as the headline of your changes.
8. <mark style="color:blue;">**commit\_description (str, optional)**</mark><mark style="color:blue;">:</mark> A more detailed description of the changes you are committing.
9. <mark style="color:blue;">**create\_pr (boolean, optional)**</mark><mark style="color:blue;">:</mark> Determines whether to create a Pull Request for the commit. Defaults to `False`. If `True`, a PR will be created based on the specified `revision`.
10. <mark style="color:blue;">**parent\_commit (str, optional)**</mark><mark style="color:blue;">:</mark> The hash of the parent commit to which your changes will be added. This is used to ensure that you are committing to the correct version of the repository, especially useful in concurrent environments.
11. <mark style="color:blue;">**run\_as\_future (bool, optional)**</mark><mark style="color:blue;">:</mark> If set to `True`, the upload process will run in the background as a non-blocking action. This returns a `Future` object that can be used to check the status of the upload.

The `path_in_repo` parameter in the `api.upload_file` function from the `huggingface_hub` library specifies the destination path within the repository on the Hugging Face Hub where the file will be uploaded. This is essentially the relative path in the repository where the file will be placed.

In this example command:

```python
api.upload_file(
    path_or_fileobj="/path/to/trained_model_weights.bin",
    path_in_repo="model_weights.bin",
    repo_id="username/my_ml_model",
    repo_type="model"
)
```

* <mark style="color:yellow;">`path_or_fileobj="/path/to/trained_model_weights.bin"`</mark>: This is the path to the file on your local machine. It indicates where the file is currently stored in your local file system.
* <mark style="color:yellow;">`path_in_repo="model_weights.bin"`</mark>: This determines where within your Hugging Face Hub repository the file will be saved. In this case, you are instructing the function to upload your file directly to the root of the repository and <mark style="color:yellow;">name it</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`model_weights.bin`</mark>. If you wanted to place this file inside a folder within your repository, you would specify a path like `"`<mark style="color:yellow;">`folder_name/model_weights.bin`</mark>`"`.
* <mark style="color:yellow;">`repo_id="username/my_ml_model"`</mark>: This identifies the specific repository on the Hugging Face Hub where the file should be uploaded. It's a combination of your username (or organization name) and the repository name.
* <mark style="color:yellow;">`repo_type="model"`</mark><mark style="color:yellow;">:</mark> This indicates the type of repository you are uploading to. In this case, it's a model repository.

</details>


# Llama3 - Training

### <mark style="color:green;">LLama3</mark>

```bash
accelerate launch -m axolotl.cli.train examples/llama-3/lora-8b.yml
```

If you have not already done so, you will be asked to enter your Weights and Biases API Key.&#x20;

&#x20;Enter the key at the command line prompt:

```yaml
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

### <mark style="color:blue;">An analysis of the axolotl.clt.train module</mark>

<details>

<summary><mark style="color:green;">Analysis of train.py script</mark></summary>

The <mark style="color:yellow;">`train.py`</mark> script in the Axolotl platform is a Command Line Interface (CLI) tool designed for training machine learning models.&#x20;

This script is structured to provide a user-friendly interface for configuring and executing model training. Here's a detailed analysis:

#### <mark style="color:blue;">Script Structure and Functionality</mark>

<mark style="color:green;">**Imports and Logger Setup**</mark>

* Essential modules like `logging`, `pathlib.Path`, `fire`, and `transformers` are imported.
* The script sets up a logger `LOG` using the `logging` module for logging various events and statuses during the script's execution.

<mark style="color:green;">**do\_cli Function**</mark>

* <mark style="color:blue;">**Function Definition**</mark><mark style="color:blue;">:</mark> The <mark style="color:yellow;">`do_cli`</mark> function is the main entry point of the script. It accepts a <mark style="color:yellow;">`config`</mark> argument (with a default value pointing to an "examples" directory) and `**kwargs` for additional arguments.
* <mark style="color:blue;">**ASCII Art Display**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">`print_axolotl_text_art()`</mark> is called to display ASCII art, likely for aesthetic purposes.
* <mark style="color:blue;">**Configuration Loading**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">`load_cfg`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">l</mark>oads configuration details from the provided `config` path. These configurations are essential for setting up model training parameters.
* <mark style="color:blue;">**Accelerator and User Token Checks**</mark><mark style="color:blue;">:</mark> The script verifies the default configuration for the accelerator (such as a GPU) and checks the user token. These checks are crucial for ensuring that the hardware is correctly set up and the user is authenticated.
* <mark style="color:blue;">**CLI Arguments Parsing**</mark><mark style="color:blue;">:</mark> It uses <mark style="color:yellow;">`transformers.HfArgumentParser`</mark> to parse additional CLI arguments into data classes (<mark style="color:yellow;">`TrainerCliArgs`</mark>). This step allows for dynamic customization of training parameters via the command line.
* <mark style="color:blue;">**Dataset Loading**</mark><mark style="color:blue;">:</mark> <mark style="color:yellow;">`load_datasets`</mark> is called with the parsed configuration and CLI arguments. This function is responsible for loading the dataset as per the configuration, which is a critical step in the training process.
* <mark style="color:blue;">**Model Training**</mark><mark style="color:blue;">:</mark> The <mark style="color:yellow;">`train`</mark> function is invoked with the loaded configuration, CLI arguments, and dataset metadata. This function likely encompasses the core logic for model training.

<mark style="color:green;">**Main Block**</mark>

* The script checks if it's being run as the main program <mark style="color:yellow;">(</mark><mark style="color:yellow;">`__name__ == "__main__"`</mark><mark style="color:yellow;">)</mark> and not as a module in another script. If it's the main program, it uses <mark style="color:yellow;">`fire.Fire(do_cli)`</mark> to execute the <mark style="color:yellow;">`do_cli`</mark> function, enabling the script to be interacted with from the command line.

</details>

### <mark style="color:red;">Issues that arose (ignore)</mark>

We had a problem with dependencies - what a surprise

<details>

<summary><mark style="color:green;">What are xformers?</mark></summary>

The GitHub repository for xFormers, as described in the provided text, offers a comprehensive overview of the library and its functionalities. Here's an analysis of the key points:

#### Overview of xFormers:

1. **Customizable Building Blocks**:
   * xFormers provides <mark style="color:yellow;">domain-agnostic components for Transformers</mark>, usable in various fields like vision and NLP, without requiring extensive boilerplate code.
2. **Research-Oriented**:
   * The library contains cutting-edge components not yet available in mainstream libraries, indicating its focus on the latest developments in Transformer technology.
3. **Efficiency**:
   * Designed with speed and memory efficiency in mind, xFormers includes custom CUDA kernels and utilizes other libraries when appropriate.

#### Installation:

1. **Stable Versions**:
   * Recommended installation via conda or pip, depending on the environment (Linux or Windows) and CUDA version (11.8 or 12.1).
   * Specific to PyTorch versions 1.13.1, 2.0.1, or <mark style="color:yellow;">2.1.0.</mark>
2. **Development Binaries**:
   * For users who want to access the latest development features.
3. **Source Installation**:
   * An option for compatibility with different or nightly versions of PyTorch.
   * Additional steps like installing `ninja` for faster builds and setting the `TORCH_CUDA_ARCH_LIST` environment variable are suggested.

#### Key Components:

1. **Functional Operators and Components**:
   * The repository is organized into directories like `ops`, `components`, `benchmarks`, and `triton`. Each contains specific functionalities like attention mechanisms, feedforward blocks, positional embeddings, etc.
2. **Attention Mechanisms and More**:
   * xFormers offers a variety of attention mechanisms, feedforward styles, positional embeddings, and other Transformer components.
3. **Optimized Building Blocks**:
   * Includes memory-efficient attention mechanisms and various optimized operations, indicating a focus on performance.

#### Benchmarks and Testing:

* Benchmarks for memory-efficient multi-head attention (MHA) and other components are provided.
* Instructions for testing the installation and verifying available kernels are included.

#### Hackability and Extensibility:

* xFormers is designed to be hackable with composable building blocks, making it adaptable for research and development.
* It uses Triton for some optimized parts, which are explicit and accessible, indicating ease of customization.

#### Install Troubleshooting:

* The document provides troubleshooting tips for installation issues, like ensuring NVCC and CUDA runtime compatibility and setting the correct environment variables.

</details>

The setup.py script deals with issues between Xformers and Pytorch

<details>

<summary><mark style="color:green;">XFormers and Pytorch</mark></summary>

The setup.py script indicates a temporary workaround for a compatibility issue with `xformers`. If `torch==2.1.0` is in the requirements, the <mark style="color:yellow;">standard</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`xformers`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">dependency is removed and replaced with a specific version</mark> installed directly from the GitHub repository. This suggests that the <mark style="color:yellow;">current release of</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">`xformers`</mark> <mark style="color:yellow;"></mark><mark style="color:yellow;">does not support Torch 2.1.0</mark> and requires using the main branch from the GitHub repo.

The special handling of `xformers` in `setup.py` suggests that the version of `xformers` compatible with `torch==2.1.0` <mark style="color:yellow;">is not available in the standard Python package index</mark> (PyPI). Therefore, the script directly installs `xformers` from the main branch of its GitHub repository.

This approach indicates that the Axolotl platform is using features or fixes from `xformers` that are only available in the latest code and have not yet been released officially.

Users of Axolotl need to be aware that they are using a potentially unstable or untested version of `xformers`, which might have implications for production use.

</details>

The configuration line `eval_sample_packing: False` within a machine learning training configuration file is specifically relevant to how data is managed during the evaluation phase of the training process. Here's a detailed breakdown of what this means and why it's important:

#### Context and Purpose

* **Sample Packing**: This is a technique used in training deep learning models, especially those with sequence data (like text or time series), to optimize the utilization of computational resources like GPU memory. It involves arranging multiple sequences in a single batch in a compact way to reduce padding, which is often necessary when sequences of variable lengths are processed together.
* **Evaluation Phase**: During model training, there is typically a phase called evaluation or validation where the trained model is tested against a separate dataset that was not used during the actual training. This helps in checking the model's performance and generalizability on new, unseen data. The evaluation phase is crucial for monitoring overfitting, underfitting, and for tuning the model's hyperparameters.

#### Impact of `eval_sample_packing: False`

* **Disabling Sample Packing in Evaluation**: By setting `eval_sample_packing` to `False`, you instruct the training process not to use the sample packing technique during the evaluation phase. This means that the evaluation data will be processed in a straightforward, possibly less memory-efficient manner, where each sequence or data point is treated individually without attempting to optimize the batch structure by tightly packing multiple sequences together.

#### Why Disable Sample Packing for Evaluation?

* **Simplicity and Debugging**: Sample packing can complicate the data handling process, making debugging more difficult if things go wrong. Disabling it for evaluation can simplify the computation and make it easier to trace issues or assess the model's performance straightforwardly.
* **Memory and Compute Trade-offs**: While sample packing can save memory and potentially speed up training by reducing the number of operations on padded data, it may not always provide benefits during evaluation, especially if the evaluation dataset is small or if the overhead of managing packed samples outweighs the benefits.
* **Consistency and Accuracy**: In some cases, packing might introduce subtle bugs or inconsistencies (e.g., incorrect handling of sequence boundaries or masking). Evaluating the model without packing ensures that the performance metrics are obtained in a straightforward and consistent manner, closely representing how the model will operate in production (assuming production use does not involve sample packing).

#### Practical Implications

Setting `eval_sample_packing` to `False` typically leads to a simpler and potentially more reliable evaluation phase, at the possible cost of increased memory usage and longer computational times due to less efficient data handling. This setting helps ensure that the evaluation metrics reflect the true performance of the model under standard operating conditions.\
\
Certainly! Let me explain sample packing in more detail and how it relates to other hyperparameters.

Sample packing is a technique used in natural language processing (NLP) to efficiently utilize the available computational resources, particularly when training large language models. It involves combining multiple shorter sequences into a single batch to maximize the utilization of the GPU memory and computational capacity.

In the context of training a language model like the one you are working with (based on the Meta-Llama model), sample packing helps in the following ways:

1. GPU Memory Utilization: Language models often have a fixed input sequence length (e.g., 4096 tokens in your configuration). However, not all input sequences in a batch may have the same length. Sample packing allows you to pack multiple shorter sequences together to fill up the available sequence length in a batch. This way, you can make the most efficient use of the GPU memory by minimizing padding and ensuring that each batch contains a maximum number of actual tokens.
2. Computational Efficiency: By packing multiple sequences into a single batch, you can process more examples in parallel, leading to faster training times. This is because GPUs are designed to perform well on parallelizable tasks, and processing a larger batch size allows for better utilization of the GPU's computational resources.
3. Training Stability: Sample packing can help stabilize the training process by providing a more consistent batch size. When sequences of varying lengths are processed individually, the effective batch size may fluctuate, which can impact the stability of the gradients and the overall training dynamics. Sample packing helps maintain a more consistent batch size, leading to more stable training.

Now, let's discuss how sample packing relates to other hyperparameters:

* Sequence Length: Sample packing is directly related to the sequence length hyperparameter (`sequence_len` in your configuration). The sequence length determines the maximum number of tokens that can be processed in a single batch. Sample packing tries to fill up this sequence length by combining multiple shorter sequences. If the sequence length is too small, it may limit the effectiveness of sample packing.
* Batch Size: The batch size (`micro_batch_size` in your configuration) determines the number of sequences processed in parallel during training. Sample packing aims to maximize the number of sequences that can fit within a batch while staying within the memory constraints of the GPU. The larger the batch size, the more opportunities there are for sample packing to be effective.
* GPU Memory: The available GPU memory is a crucial factor in determining the feasibility of sample packing. Sample packing allows you to utilize the GPU memory more efficiently by minimizing padding and maximizing the number of actual tokens processed in each batch. However, if the GPU memory is limited, you may need to adjust the batch size or sequence length accordingly.

In your specific case, the error message suggests that the evaluation dataset split is too small for sample packing. This means that the number of sequences in the evaluation dataset is not sufficient to effectively apply sample packing. By setting `eval_sample_packing: false`, you are disabling sample packing for the evaluation dataset, which should resolve the issue.

It's important to note that sample packing is more commonly used during training rather than evaluation. During evaluation, you typically want to process sequences individually to get accurate metrics and predictions for each example.

I hope this explanation clarifies the concept of sample packing and its relationship to other hyperparameters. Let me know if you have any further questions!

C


# Full Fine Tune

This is the coonfiguration file for a full fine tune.

```yaml
base_model: meta-llama/Meta-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: datasets/alpagasus/data/train-00000-of-00001-0c59455170918204.parquet
    type: alpaca
    ds_type: parquet
    data_files: 
      - datasets/alpagasus/data/train-00000-of-00001-0c59455170918204.parquet

dataset_prepared_path: ./prepared_data
val_set_size: 0.10
output_dir: ./llama-fft-out

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_project: llama3-fft
wandb_entity: continuum-labs
wandb_watch:
wandb_name: 
wandb_log_model:

gradient_accumulation_steps: 8  
micro_batch_size: 1  
num_epochs: 1  
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 100
evals_per_epoch: 2
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: <|end_of_text|>
```

### <mark style="color:blue;">Suggestions</mark>

#### <mark style="color:green;">Optimizer and Learning Rate</mark>

* You're currently using <mark style="color:yellow;">**`paged_adamw_8bit`**</mark> optimizer with a learning rate of 2e-5. You could experiment with other optimizers like <mark style="color:yellow;">**`lion_8bit`**</mark>, <mark style="color:yellow;">**`galore_adamw_8bit`**</mark>, or <mark style="color:yellow;">**`adamw_torch_fused`**</mark> to see if they yield better performance.
* Additionally, you can try different learning rates, such as 1e-5 or 3e-5, to find the optimal value for your specific dataset and model.

<mark style="color:green;">Learning Rate Scheduler</mark>

* You're using the <mark style="color:yellow;">**`cosine`**</mark> learning rate scheduler. You could explore other options like <mark style="color:yellow;">**`one_cycle`**</mark> or <mark style="color:yellow;">**`log_sweep`**</mark> to see if they improve the training process.
* If using the cosine scheduler, you can set <mark style="color:yellow;">**`cosine_min_lr_ratio`**</mark> and <mark style="color:yellow;">**`cosine_constant_lr_ratio`**</mark> to control the decay and freezing of the learning rate during training.

<mark style="color:green;">Gradient Accumulation and Batch Size</mark>

* Adjust the <mark style="color:yellow;">**`gradient_accumulation_steps`**</mark> and <mark style="color:yellow;">**`micro_batch_size`**</mark> based on your available GPU memory. Increasing the batch size can lead to faster convergence but may require more memory.
* You can also set <mark style="color:yellow;">**`eval_batch_size`**</mark> to a different value than <mark style="color:yellow;">**`micro_batch_size`**</mark> for evaluation.

<mark style="color:green;">Datasets and Preprocessing</mark>

* Consider using multiple datasets by adding more entries to the <mark style="color:yellow;">**`datasets`**</mark> list. This can help improve the model's generalization and robustness.
* Experiment with different <mark style="color:yellow;">**`sequence_len`**</mark> values to find the optimal sequence length for your specific task and dataset.
* Set <mark style="color:yellow;">**`train_on_inputs`**</mark> to <mark style="color:yellow;">**`true`**</mark> if you want to include the human's prompt in the training labels.

<mark style="color:green;">Evaluation and Checkpointing</mark>

* Adjust <mark style="color:yellow;">**`eval_steps`**</mark> or <mark style="color:yellow;">**`evals_per_epoch`**</mark> to control the frequency of evaluation during training. More frequent evaluation can provide better insights into the model's progress.
* Similarly, modify <mark style="color:yellow;">**`save_steps`**</mark> or <mark style="color:yellow;">**`saves_per_epoch`**</mark> to control the frequency of checkpoint saving.
* Set <mark style="color:yellow;">**`save_total_limit`**</mark> to limit the number of checkpoints saved at a time, preventing disk space issues.

<mark style="color:green;">Advanced Techniques</mark>

* Enable <mark style="color:yellow;">**`neftune_noise_alpha`**</mark> to add noise to embeddings using the NEFT technique, which can improve generalization.
* Set <mark style="color:yellow;">**`s2_attention`**</mark> to <mark style="color:yellow;">**`true`**</mark> to use shifted-sparse attention for LLaMA models, which can reduce memory usage and improve efficiency.
* Experiment with different values for <mark style="color:yellow;">**`lora_r`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`lora_alpha`**</mark>, and <mark style="color:yellow;">**`lora_dropout`**</mark> if using LoRA adaptation.

<mark style="color:green;">Debugging and Monitoring</mark>

* Set <mark style="color:yellow;">**`debug`**</mark> to <mark style="color:yellow;">**`true`**</mark> to enable debug mode for more detailed logging and debugging information.
* Adjust <mark style="color:yellow;">**`logging_steps`**</mark> to control the frequency of logging during training.
* Set <mark style="color:yellow;">**`loss_watchdog_threshold`**</mark> and <mark style="color:yellow;">**`loss_watchdog_patience`**</mark> to monitor and abort training if the loss exceeds a certain threshold for a specified number of steps.

<mark style="color:green;">Tokenizer and Special Tokens</mark>

* Add or modify special tokens in the <mark style="color:yellow;">**`special_tokens`**</mark> section based on your specific requirements.
* If you have added extra tokens to your tokenizer, specify them in the <mark style="color:yellow;">**`tokens`**</mark> list.

Remember to experiment and iterate on these configurations to find the optimal settings for your specific use case. It's also important to monitor the training progress, evaluate the model's performance, and make adjustments accordingly.

### <mark style="color:blue;">Machine Memory</mark>

<mark style="color:green;">Memory Usage</mark>

* With <mark style="color:yellow;">**`load_in_8bit`**</mark> and <mark style="color:yellow;">**`load_in_4bit`**</mark> set to <mark style="color:yellow;">**`false`**</mark>, the model will be loaded in <mark style="color:blue;">**full precision (FP32)**</mark>. This may consume a significant amount of memory, especially for a large model like LLaMA-3-8B.
* To reduce memory usage, you can consider setting <mark style="color:yellow;">**`load_in_8bit`**</mark> to <mark style="color:yellow;">**`true`**</mark>. This will load the model in 8-bit precision, which can significantly reduce memory consumption while maintaining comparable performance.

<mark style="color:green;">Sequence Length</mark>

* You have set <mark style="color:yellow;">**`sequence_len`**</mark> to 8192, which is quite large. Depending on your specific task and dataset, you may not need such a long sequence length.
* Increasing the sequence length will require more memory during training. If you encounter out-of-memory (OOM) issues, consider reducing the <mark style="color:yellow;">**`sequence_len`**</mark> to a smaller value, such as 2048 or 4096.

<mark style="color:green;">Gradient Accumulation and Batch Size</mark>

* With <mark style="color:yellow;">**`gradient_accumulation_steps`**</mark> set to 8 and <mark style="color:yellow;">**`micro_batch_size`**</mark> set to 1, the effective batch size will be 8.
* Depending on your available memory, you may be able to increase the <mark style="color:yellow;">**`micro_batch_size`**</mark> to a larger value, such as 2 or 4, while keeping the <mark style="color:yellow;">**`gradient_accumulation_steps`**</mark> the same. This can help speed up the training process.

<mark style="color:green;">Optimizer and Learning Rate</mark>

* You're using the <mark style="color:yellow;">**`paged_adamw_8bit`**</mark> optimizer, which is a good choice for memory efficiency. However, make sure you have the necessary dependencies installed for this optimizer.
* The learning rate of 2e-5 is a reasonable starting point, but you may need to experiment with different values to find the optimal learning rate for your specific dataset and model.

<mark style="color:green;">Evaluation and Checkpointing</mark>

* You have set <mark style="color:yellow;">**`evals_per_epoch`**</mark> to 2, which means evaluation will be performed twice per epoch. Depending on your dataset size and training duration, you may want to adjust this value.
* Similarly, <mark style="color:yellow;">**`saves_per_epoch`**</mark> is set to 1, meaning checkpoints will be saved once per epoch. Adjust this value based on your desired checkpoint frequency and available storage.

<mark style="color:green;">Debugging and Logging</mark>

* Setting <mark style="color:yellow;">**`debug`**</mark> to <mark style="color:yellow;">**`true`**</mark> will enable debug mode, which can provide more detailed logging and debugging information. However, keep in mind that this may slow down the training process.
* You have set <mark style="color:yellow;">**`logging_steps`**</mark> to 1, which means logging will occur at every step. If you find the logging too verbose, you can increase this value to a larger number to reduce the logging frequency.

Overall, your configuration looks reasonable for fine-tuning LLaMA-3-8B on an A100 with 80GB of memory.&#x20;

Just be mindful of the memory usage, especially with the large sequence length and full precision loading. If you encounter OOM issues, consider adjusting the sequence length, enabling 8-bit loading, or increasing the batch size if possible.

Remember to monitor the training progress, evaluate the model's performance, and make adjustments as needed based on your specific requirements and available resources.


# Special Tokens

It is important to have special tokens like delimiters, end-of-sequence, beginning-of-sequence in your tokenizer’s vocabulary.&#x20;

This will help you avoid tokenization issues and help your model train better.&#x20;

You can do this in axolotl like this:

```bash
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"
tokens: # these are delimiters
  - "<|im_start|>"
  - "<|im_end|>"
```

When you include these tokens in your axolotl config, axolotl adds these tokens to the tokenizer’s vocabulary.

## <mark style="color:blue;">Explanation of Special Tokens</mark>

The JSON structure you provided describes the special tokens used by a tokenizer in a Huggingface model.&#x20;

These special tokens have specific roles in language models and their processing. Let's break down what each part means:

### <mark style="color:blue;">**bos\_token**</mark><mark style="color:blue;">:</mark>

* **content**: `"<s>"` - This is the 'beginning of sequence' token. It's used to indicate the start of a text sequence.
* **lstrip**: `false` - Indicates that spaces to the left (beginning) of this token should not be stripped.
* **normalized**: `false` - This token is not subject to normalization during tokenization.
* **rstrip**: `false` - Indicates that spaces to the right (end) of this token should not be stripped.
* **single\_word**: `false` - This token does not represent a single word.

### <mark style="color:blue;">**eos\_token**</mark><mark style="color:blue;">:</mark>

* **content**: `"</s>"` - This is the 'end of sequence' token, used to mark the end of a text sequence.
* **lstrip**: `false` - Spaces to the left of this token should not be stripped.
* **normalized**: `false` - This token is not normalized.
* **rstrip**: `false` - Spaces to the right of this token should not be stripped.
* **single\_word**: `false` - It's not considered a single word.

### <mark style="color:blue;">**pad\_token**</mark><mark style="color:blue;">:</mark>

* `"</s>"` - This is the padding token, used to fill in the sequence to a uniform length in batch processing. Interestingly, it's the same as the 'end of sequence' token here, which is an unusual but not unheard-of configuration.

### <mark style="color:blue;">**unk\_token**</mark><mark style="color:blue;">:</mark>

* **content**: `"<unk>"` - This is the 'unknown' token, used to represent words or characters not found in the model's vocabulary.
* **lstrip**: `false` - Spaces to the left of this token are not removed.
* **normalized**: `false` - The token is not normalized.
* **rstrip**: `false` - Spaces to the right of this token are not removed.
* **single\_word**: `false` - It's not considered a single word.

This configuration is part of the tokenizer setup and dictates how the tokenizer handles these special tokens during the processing of text.&#x20;

Each special token has a role in helping the model understand and generate text, from marking the start and end of a text sequence to dealing with unknown words and padding sequences for consistent length.


# Prompt Construction for Fine-Tuning Large Language Models

Fine-tuning large language models on custom datasets is a powerful technique to adapt foundation models to specific domains and tasks.

A critical aspect of fine-tuning is constructing effective prompts that present the input data to the model in an optimal format for learning.

This guide covers strategies and best practices for prompt construction when fine-tuning models using tools like Axolotl. It assumes a basic familiarity with language model fine-tuning.

### <mark style="color:blue;">Prompt Templates</mark>&#x20;

A common approach is to use prompt templates that provide a standard structure for the model inputs and outputs. Popular prompt templates include:

#### <mark style="color:green;">Alpaca format</mark>

```json
{
  "instruction": "...",
  "input": "...", 
  "output": "..."
}
```

#### <mark style="color:green;">ShareGPT format</mark>

```json
{
  "conversations": [
    {
      "from": "human",
      "value": "..."
    },
    {
      "from": "assistant", 
      "value": "..."
    }
  ]
}
```

Prompt templates offer several benefits:

* Provide a clear demarcation between inputs and outputs
* Enforce a consistent structure across examples
* Allow specifying roles like "human" and "assistant"

However, there are also some <mark style="color:yellow;">drawbacks to templated prompts</mark>:

* Can add unnecessary boilerplate and special tokens
* May enforce a conversational structure when a direct mapping of inputs to outputs is sufficient
* Limit flexibility to only the roles and format dictated by the template

#### <mark style="color:green;">Template-Free Prompts with input\_output Format</mark>

For situations where a prompt template is overly restrictive, <mark style="color:yellow;">Axolotl supports the input\_output format</mark> for constructing template-free prompts.

With input\_output, you provide a list of text segments, each with a boolean label indicating if the segment should be used as input to the model or treated as the target output.

<mark style="color:green;">input\_output format example</mark>

```json
{
  "segments": [
    {
      "text": "<s>Hello\n",
      "label": true
    },
    {
      "text": "Hi there! ",
      "label": true
    },
    {
      "text": "Goodbye ",
      "label": false
    },
    {
      "text": "Farewell</s>",
      "label": true
    }
  ]
}
```

Configuring dataset for input\_output format:

```yaml
datasets:
  - path: data.jsonl
    type: input_output

train_on_inputs: false   # Ignores the label:false segments
```

### <mark style="color:blue;">Some other prompt construction ideas</mark>

Here are three creative ideas for other formats for fine-tuning a large language model:

<mark style="color:green;">Tree-structured format</mark>

```json
{
  "root": {
    "text": "Once upon a time...",
    "children": [
      {
        "text": "There was a young girl named Lily.",
        "children": [...]
      },
      {
        "text": "She lived in a small village.",
        "children": [...]
      }
    ]
  }
}
```

This format could be useful for fine-tuning on hierarchical data like stories, articles, or dialogue trees. The model would learn to generate text that follows a coherent structure.

<mark style="color:green;">Linked-segments format</mark>

```json
{
  "segments": [
    {
      "id": 1,
      "text": "Paris is the capital of France.",
      "links": [2, 3]
    },
    {
      "id": 2, 
      "text": "It is known for its art, fashion, and cuisine.",
      "links": [4]
    },
    {
      "id": 3,
      "text": "The Eiffel Tower is a famous landmark in Paris.",
      "links": [4]  
    },
    {
      "id": 4,
      "text": "Paris attracts millions of tourists every year."
    }
  ]
}
```

The linked-segments format allows specifying relationships between different parts of the input.&#x20;

The model could learn to generate coherent text that follows the linked structure. This could be useful for tasks involving reasoning or inferring relationships between facts.

<mark style="color:green;">Multi-field format</mark>

```json
{
  "title": "Apple Pie Recipe",
  "ingredients": [
    "3 cups all-purpose flour",
    "1 teaspoon salt",
    "1 cup unsalted butter", 
    "2/3 cup ice water",
    "8 cups sliced apples",
    "2 tablespoons lemon juice",
    "3/4 cup white sugar",
    "1/2 teaspoon ground cinnamon"
  ],
  "instructions": [
    "In a large bowl, mix flour and salt...",
    "Stir in butter until mixture is crumbly...",
    "Knead the dough, adding ice water as needed...",
    "Preheat the oven to 375°F...",
    "Mix apples with lemon juice, sugar and cinnamon...",
    "Place filling in the pie crust...", 
    "Cover with top crust, seal edges, and cut slits to vent...",
    "Bake for 45 minutes until crust is golden brown..."
  ]
}
```

This multi-field format separates different aspects of the input into designated fields. T

he model learns to interpret the fields and generate text based on their roles (title, ingredients, instructions). This could enable fine-tuning for domain-specific applications like generating recipes, product descriptions, or other structured content.

The key aspects are:

* Each segment is a snippet of raw text
* The label:true segments are concatenated to form the model input
* The label:false segments are ignored when training (using a standard technique of setting the label to a special ignore token)
* You are responsible for including any special tokens, whitespace, etc. The segments are simply concatenated as-is.

#### <mark style="color:green;">Validating Prompts</mark>

When using template-free prompts, it's important to validate that the prompts are being constructed as intended.&#x20;

Some tips:

* Use the --debug flag with the preprocessing command to print out the tokenized prompts with labels
* Load the tokenized dataset and spot check decoded examples
* Verify that the correct segments have a label of -100 indicating they will be ignored

#### <mark style="color:green;">Best Practices</mark>&#x20;

Some general best practices for prompt construction:

* Keep prompts concise; avoid extraneous text that may distract the model
* Put key information like instructions or questions towards the beginning
* For most cases, avoid multi-turn conversation unless essential for the task
* Use a clear separator like a newline between input and output
* Include end-of-sequence (\</s>) token to help model recognize completion
* Aim for consistency across examples
* Experiment with different formulations and validate what works best

#### <mark style="color:green;">Conclusion</mark>

Effective prompt construction is essential for fine-tuning performance. Template-free prompts using the input\_output format in Axolotl provide flexibility to optimize prompts for your specific task. Validate prompts carefully, aim for consistency and clarity, and iterate to find the optimal approach.


# Memory-Efficient Fine-Tuning Techniques for Large Language Models

Fine-tuning large language models (LLMs) can be a memory-intensive process, especially when working with huge datasets and model architectures.&#x20;

As model sizes continue to grow, it becomes increasingly important to utilise memory-efficient techniques to make the most of available hardware resources.&#x20;

In this documentation, we will explore several strategies that can significantly reduce the memory footprint during fine-tuning while maintaining or even improving training speed.

### <mark style="color:blue;">Gradient Accumulation</mark>

Gradient accumulation is a powerful technique that allows you to effectively increase the batch size without the need for additional GPU memory.&#x20;

The idea is to break down the batch into smaller mini-batches and perform multiple forward and backward passes, accumulating the gradients in the process. Once the desired number of gradients has been accumulated, the optimizer step is performed.

Here's how you can enable gradient accumulation using the `Trainer` class in the Transformers library:

```python
training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    **default_args
)

trainer = Trainer(model=model, args=training_args, train_dataset=ds)
result = trainer.train()
```

In this example, `per_device_train_batch_size` is set to 1, and `gradient_accumulation_steps` is set to 4.&#x20;

This means that the model will perform 4 forward and backward passes, accumulating the gradients, before updating the weights.&#x20;

The effective batch size becomes 4 (1 \* 4), while the memory usage remains equivalent to processing a batch size of 1.

Gradient accumulation can significantly reduce memory usage at the cost of slightly slower training speed due to the additional forward and backward passes. However, it allows you to train with larger effective batch sizes that would otherwise not fit into GPU memory.

### <mark style="color:blue;">Gradient Checkpointing</mark>

Gradient checkpointing is another technique that helps reduce memory usage during the backward pass.&#x20;

During a typical backward pass, all the activations from the forward pass are stored in memory to compute the gradients. This can lead to a significant memory overhead, especially for deep models.

Gradient checkpointing offers a compromise by strategically saving a subset of activations at checkpoints throughout the computational graph.&#x20;

During the backward pass, the activations are recomputed from these checkpoints as needed, reducing the memory footprint at the cost of some additional computation.

To enable gradient checkpointing in the `Trainer`, you can simply pass the `gradient_checkpointing` flag to the `TrainingArguments`:

```python
training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    **default_args
)

trainer = Trainer(model=model, args=training_args, train_dataset=ds)
result = trainer.train()
```

Gradient checkpointing can provide memory savings of up to 20-30% but may slow down training by around 20% due to the recomputation of activations.&#x20;

It is particularly useful when dealing with large models that have deep computational graphs.

### <mark style="color:blue;">Mixed Precision Training (FP16)</mark>

Mixed precision training, also known as FP16 training, is a technique that leverages the reduced precision of 16-bit floating-point numbers to speed up computations and save memory.&#x20;

By using half-precision (FP16) for storing activations and performing computations, while keeping the model weights in full precision (FP32), mixed precision training can significantly reduce memory usage and improve training speed.

To enable mixed precision training in the `Trainer`, you can set the `fp16` flag to `True`:

```python
training_args = TrainingArguments(
    per_device_train_batch_size=4,
    fp16=True,
    **default_args
)

trainer = Trainer(model=model, args=training_args, train_dataset=ds)
result = trainer.train()
```

Mixed precision training can provide up to 2x speedup compared to full precision training while reducing memory usage. However, it's important to note that not all models and hardware support FP16 training, so it's crucial to check compatibility before enabling this feature.

### <mark style="color:blue;">Optimizer Choices</mark>

The choice of optimizer can also impact memory usage during fine-tuning.&#x20;

The commonly used Adam optimizer stores rolling averages of the gradients, which can add a significant memory footprint, especially for large models with millions of parameters.

#### <mark style="color:green;">Adafactor</mark>

Adafactor is an alternative optimizer that reduces memory usage by storing only aggregated information (row-wise and column-wise sums) of the rolling averages instead of the full matrices.&#x20;

This can lead to substantial memory savings without sacrificing much in terms of convergence speed.

To use Adafactor in the `Trainer`, you can set the `optim` argument to `"adafactor"`:

```python
training_args = TrainingArguments(
    per_device_train_batch_size=4,
    optim="adafactor",
    **default_args
)

trainer = Trainer(model=model, args=training_args, train_dataset=ds)
result = trainer.train()
```

Adafactor can provide memory savings of around 50% compared to Adam, making it a good choice for memory-constrained environments. However, it's worth noting that in some cases, Adafactor may exhibit slower convergence compared to Adam, so experimentation is recommended.

#### <mark style="color:green;">8-bit Adam</mark>

8-bit Adam is another memory-efficient optimizer that quantizes the optimizer states to 8-bit precision, reducing memory usage while maintaining the full optimizer state.&#x20;

It strikes a balance between memory efficiency and convergence speed.

To use 8-bit Adam, you need to install the `bitsandbytes` library and pass a custom optimizer to the `Trainer`:

```python
import bitsandbytes as bnb
from torch import nn
from transformers.trainer_pt_utils import get_parameter_names

decay_parameters = get_parameter_names(model, [nn.LayerNorm])
decay_parameters = [name for name in decay_parameters if "bias" not in name]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if n in decay_parameters],
        "weight_decay": training_args.weight_decay,
    },
    {
        "params": [p for n, p in model.named_parameters() if n not in decay_parameters],
        "weight_decay": 0.0,
    },
]

optimizer_kwargs = {
    "betas": (training_args.adam_beta1, training_args.adam_beta2),
    "eps": training_args.adam_epsilon,
}
optimizer_kwargs["lr"] = training_args.learning_rate
adam_bnb_optim = bnb.optim.Adam8bit(
    optimizer_grouped_parameters,
    betas=(training_args.adam_beta1, training_args.adam_beta2),
    eps=training_args.adam_epsilon,
    lr=training_args.learning_rate,
)

trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
result = trainer.train()
```

8-bit Adam can provide memory savings similar to Adafactor while maintaining convergence speed closer to Adam. It's a good choice when memory is limited, but convergence speed is still a priority.

### <mark style="color:blue;">Combining Techniques</mark>

The real power of these memory-efficient techniques lies in combining them to achieve optimal memory usage and training speed.&#x20;

By using gradient accumulation, gradient checkpointing, mixed precision training, and memory-efficient optimizers together, you can significantly reduce the memory footprint of fine-tuning large language models.

Here's an example that combines all the techniques:

```python
training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    fp16=True,
    **default_args,
)

trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
result = trainer.train()
```

In this setup, we use a per-device batch size of 1 with gradient accumulation steps of 4, enabling gradient checkpointing, mixed precision training, and the 8-bit Adam optimizer.&#x20;

This combination can lead to memory savings of up to 3x compared to the baseline while maintaining or even improving training speed.

### <mark style="color:blue;">Conclusion</mark>

Fine-tuning large language models can be a memory-intensive process, but by leveraging techniques like gradient accumulation, gradient checkpointing, mixed precision training, and memory-efficient optimizers, you can significantly reduce the memory footprint and make the most of your hardware resources.&#x20;

Experiment with different combinations of these techniques to find the optimal balance between memory usage and training speed for your specific use case.

Remember to profile your training process and monitor memory usage to ensure that you are within the limits of your hardware.&#x20;

Additionally, keep in mind that while these techniques can provide substantial memory savings, they may introduce slight performance overheads or impact convergence speed in some cases. It's always a good idea to validate the results and compare them against the baseline to ensure that you are achieving the desired performance.

By applying these memory-efficient techniques, you can push the boundaries of fine-tuning large language models on resource-constrained environments and unlock the potential of LLMs for a wide range of applications.


# Training Ideas around Hyperparameters

To improve the loss rate, you can consider adjusting the following hyperparameters:

`learning_rate`: The learning rate determines the step size at which the model's weights are updated during training. Adjusting the learning rate can significantly impact the model's performance. You can try different values, such as 0.0001, 0.0005, or 0.001, to see if they lead to better convergence and lower loss.

`num_epochs`: Increasing the number of epochs allows the model to train for a longer period and potentially learn better representations. However, training for too many epochs can also lead to overfitting. You can experiment with increasing the number of epochs, such as 6, 8, or 10, and monitor the validation loss to find the optimal number of epochs.

`micro_batch_size`: The micro batch size determines the number of samples processed in each iteration within a batch. Increasing the micro batch size can provide more stable gradients and improve training efficiency. You can try increasing the micro batch size to 4, 8, or 16, depending on your available GPU memory.

`gradient_accumulation_steps`: Gradient accumulation allows you to increase the effective batch size without increasing the memory usage. By accumulating gradients over multiple steps, you can train with larger batch sizes. Increasing the gradient accumulation steps can help stabilize training and improve convergence. You can try values like 8, 16, or 32.

`lora_r` and `lora_alpha`: These parameters control the rank and scaling factor of the LoRA adaptation. Adjusting these values can impact the capacity and expressiveness of the LoRA adaptation. You can experiment with different values of `lora_r`, such as 64 or 128, and `lora_alpha`, such as 32 or 64, to find the optimal configuration for your task.

`warmup_steps`: Warmup steps gradually increase the learning rate from a low value to the specified learning rate over a certain number of steps. Increasing the number of warmup steps can help stabilize training in the early stages. You can try increasing the warmup steps to 100, 500, or 1000, depending on the total number of training steps.

`weight_decay`: Weight decay is a regularization technique that adds a penalty term to the loss function, discouraging large weights. Increasing the weight decay can help prevent overfitting. You can try values like 0.01, 0.05, or 0.1 to see if they improve generalization.

Remember to monitor the validation loss and other evaluation metrics while adjusting these hyperparameters. It's important to strike a balance and find the combination of hyperparameters that works best for your specific task and dataset.

Additionally, make sure to track your experiments and results using tools like Weights and Biases (wandb) to compare different hyperparameter configurations and identify the most effective settings.

To improve the loss curve in your training configuration, several hyperparameters could be adjusted to optimize the learning process and potentially achieve better performance. Here's a breakdown of possible changes:

Here are some ideas on how you can change the hyperparameters for fine-tuning LLaMA-2 on the Alpagasus dataset:

<mark style="color:green;">Learning Rate</mark>

* You can experiment with different learning rates to see how they impact the model's performance. Try values like 0.0001, 0.0005, or 0.001 and compare the results.
* You can also consider using a learning rate scheduler, such as the cosine scheduler, which gradually decreases the learning rate over the course of training.

<mark style="color:green;">Batch Size</mark>

* Increase the `micro_batch_size` to a larger value, like 8 or 16, to process more samples in parallel and potentially speed up training.
* Adjust the `gradient_accumulation_steps` accordingly to maintain the effective batch size. For example, if you double the `micro_batch_size`, you can halve the `gradient_accumulation_steps`.

<mark style="color:green;">Number of Epochs</mark>

* Increase the `num_epochs` to allow the model to train for a longer period. Try values like 8, 10, or 12 epochs and monitor the validation performance to find the optimal number of epochs.

<mark style="color:green;">LoRA Parameters</mark>

* Experiment with different values for `lora_r` and `lora_alpha` to control the rank and scaling of the LoRA adaptation. Try increasing `lora_r` to 64 or 128 and `lora_alpha` to 32 or 64 to see if it improves the model's performance.
* You can also try different values for `lora_dropout`, such as 0.1 or 0.2, to introduce more regularization during training.

<mark style="color:green;">Sequence Length</mark>

* Consider reducing the `sequence_len` to a smaller value, like 2048 or 1024, to process shorter sequences. This can help reduce memory usage and potentially allow for larger batch sizes.
* Adjust the `pad_to_sequence_len` accordingly to match the new sequence length.

<mark style="color:green;">Optimizer</mark>

* Experiment with different optimizers, such as AdamW or AdaFactor, to see if they improve the model's convergence and performance.
* You can also try adjusting the `weight_decay` value to control the regularization strength. Try values like 0.05 or 0.1.

<mark style="color:green;">Evaluation</mark>

* Increase the `evals_per_epoch` to perform more frequent evaluations during training. This can help you monitor the model's progress and detect overfitting or underfitting.
* Adjust the `eval_max_new_tokens` to control the maximum number of tokens generated during evaluation. Try values like 256 or 512 to generate longer responses.

<mark style="color:green;">Mixed Precision</mark>

* Enable mixed precision training by setting `bf16` to `true` if your hardware supports it. This can help reduce memory usage and speed up training.

<mark style="color:green;">Gradient Checkpointing</mark>

* Set `gradient_checkpointing` to `true` to enable gradient checkpointing, which can help reduce memory usage during training by recomputing activations during the backward pass.

<mark style="color:green;">Early Stopping</mark>

* Set `early_stopping_patience` to a value like 2 or 3 to enable early stopping based on the validation performance. This can help prevent overfitting and save training time.

Remember to experiment with different combinations of hyperparameters and evaluate the model's performance on a validation set to find the optimal configuration for your specific task and dataset.&#x20;

It's also important to monitor the training progress, loss curves, and validation metrics to ensure the model is learning effectively.

Keep in mind that changing hyperparameters can have a significant impact on the model's performance and training time, so it's recommended to start with small changes and gradually fine-tune the hyperparameters based on the observed results.

#### <mark style="color:green;">Learning Rate and Scheduler</mark>

* **`learning_rate`**: Consider experimenting with different learning rates. A slightly lower or higher rate might be more optimal depending on how quickly or slowly your model is currently learning.
* **`lr_scheduler`**: You are currently using a cosine scheduler. If the loss seems to plateau early or if the learning rate adjustments don't seem to match well with performance improvements, consider experimenting with other schedulers like step decay or exponential decay that might provide more control at different phases of training.

#### <mark style="color:green;">Regularization and Dropout</mark>

* **`lora_dropout`**: Adjusting the dropout rate can help prevent overfitting, especially if your model is complex and trained on a large dataset. If your current dropout setting leads to too much regularization, reducing it slightly might help the model learn more effectively.
* **`weight_decay`**: Although set to 0.0, introducing a small amount of weight decay can help in regularizing the model further and avoid overfitting.

#### <mark style="color:green;">Batch and Micro-Batch Sizes</mark>

* **`micro_batch_size`**: Increasing the micro-batch size, if hardware permits, can lead to more stable gradient estimates, which might improve the model's training efficiency.
* **`gradient_accumulation_steps`**: Adjusting this can help simulate larger batch sizes without increasing memory requirements by accumulating gradients over several forward passes before performing a backward update.

#### <mark style="color:green;">Gradient Handling</mark>

* **`gradient_checkpointing`**: While you have this enabled, which is good for memory management with large models, ensure that it doesn't interfere with training dynamics. Sometimes, turning it off (if memory allows) can provide a minor boost in training performance by reducing computation overhead.
* **`bf16`**: If using mixed precision training (`bf16`), make sure that it’s well supported and optimized on your hardware. Sometimes adjusting the precision can impact both performance and the resulting model accuracy/loss.

#### <mark style="color:green;">Optimizer Configuration</mark>

* **`optimizer`**: You are using `adamw_bnb_8bit`. While bit-based optimization can reduce memory usage and potentially speed up training, it might impact the convergence characteristics. Consider trying the standard AdamW or another robust optimizer like SGD with momentum for comparison.

#### <mark style="color:green;">Epochs and Early Stopping</mark>

* **`num_epochs`**: If the loss is not satisfactory by the end of the current epochs, consider increasing the number of epochs.
* **`early_stopping_patience`**: Implement or adjust this parameter to stop training early if the validation loss does not improve for a given number of epochs. This prevents overfitting and wastage of computational resources.

#### <mark style="color:green;">Additional Adjustments</mark>

* **`group_by_length`**: Enabling this could lead to more efficient batching by reducing the number of padding tokens, which might help in improving the training speed and effectiveness.
* **`warmup_steps`**: Adjusting the number of warmup steps for the learning rate scheduler can help in stabilizing the training initially.

These adjustments should be tested systematically. It’s generally good practice to change one or two hyperparameters at a time and monitor the impact before proceeding with other changes to understand which adjustments are most effective.


# Hugging Face documentation on loading PEFT

### <mark style="color:blue;">Summary</mark>

* PEFT (Parameter-Efficient Fine Tuning) methods freeze pretrained model parameters and add a small number of trainable adapter parameters, allowing for memory-efficient fine-tuning.
* Adapters are much smaller than full models, making them easier to share and store. Examples: OPT adapter is 6MB vs 700MB for full model.
* Transformers natively supports Low Rank Adapters, IA3, and AdaLoRA PEFT methods. Other methods require using the PEFT library.
* To load a PEFT adapter, the Hub repo or local directory needs an <mark style="color:yellow;">**adapter\_config.json**</mark> and the <mark style="color:yellow;">**adapter weights**</mark>. Use AutoModelFor\* class or model.load\_adapter().
* bitsandbytes integration allows loading in 8-bit or 4-bit precision to save memory.
* Multiple adapters of the same type can be added to a model. Use model.set\_adapter() to switch between them.
* Adapters can be enabled/disabled with model.enable\_adapters() and model.disable\_adapters() after being added.
* The Trainer class supports training PEFT adapters with minor code additions. Define adapter config, add to model, pass model to Trainer.
* Additional layers like the language model head can be fine-tuned on top of a PEFT adapter by specifying modules\_to\_save in the config.

### <mark style="color:blue;">Tutorials</mark>

#### <mark style="color:green;">Choosing the Right PEFT Method</mark>

Example: If you have a large model and limited GPU memory, consider using LoRA or AdaLoRA for parameter-efficient fine-tuning

```python
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b")
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
```

* Tip: Experiment with different PEFT methods to find the one that works best for your specific task and dataset.
* Best Practice: Consider the trade-offs between memory efficiency and performance when selecting a PEFT method.
* Potential Error: Using a PEFT method that is not compatible with your model architecture or task type.

#### <mark style="color:green;">Optimising Adapter Hyperparameters</mark>

Example: When configuring a LoRA adapter, experiment with different values for `lora_alpha`, `lora_dropout`, and `r` to find the optimal balance between performance and efficiency.

```python
lora_config = LoraConfig(
    r=16,  # Experiment with different values of r
    lora_alpha=32,  # Experiment with different values of lora_alpha
    lora_dropout=0.1,  # Experiment with different values of lora_dropout
    target_modules=["q_proj", "v_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)
```

* Tip: Start with the default hyperparameters and gradually tune them based on your task and dataset.
* Best Practice: Use a validation set to evaluate the performance of different hyperparameter configurations.
* Potential Error: Setting the hyperparameters to extreme values that lead to poor performance or unstable training.

#### <mark style="color:green;">Efficient Storage and Sharing of Adapters</mark>

Example: When saving a trained adapter, use a descriptive name that includes the model architecture, PEFT method, and task information.

```python
model.save_pretrained("output/opt-1.3b-lora-custom-task")
```

* Tip: Store adapters separately from the base model to facilitate reuse across different projects.
* Best Practice: Use a version control system like Git to track changes to your adapter configurations and training scripts.
* Potential Error: Overwriting an existing adapter by mistake when saving a new one.

#### <mark style="color:green;">Combining Multiple Adapters</mark>

Example: If you have multiple adapters trained on different tasks or datasets, you can combine them using `model.set_adapter()` to leverage their combined knowledge.

```python
model.load_adapter("adapter1")
model.load_adapter("adapter2")
model.set_active_adapters(["adapter1", "adapter2"])
```

* Tip: Experiment with different adapter combinations to find the ones that yield the best performance for your specific use case.
* Best Practice: Ensure that the adapters you combine are compatible in terms of their PEFT method and model architecture.
* Potential Error: Combining adapters that have conflicting or incompatible weights, leading to poor performance or unexpected behavior.

#### <mark style="color:green;">Fine-Tuning Additional Layers with PEFT Adapters</mark>

Example: If your task requires fine-tuning the language model head in addition to the adapter, specify `modules_to_save=["lm_head"]` in your PEFT configuration.

```python
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    modules_to_save=["lm_head"],  # Fine-tune the language model head
)
```

* Tip: Be cautious when fine-tuning additional layers, as it may increase the risk of overfitting, especially if you have a small dataset.
* Best Practice: Start by fine-tuning only the adapter and gradually add additional layers if needed based on performance evaluation.
* Potential Error: Fine-tuning too many additional layers, leading to overfitting and poor generalization.

#### <mark style="color:green;">Monitoring Adapter Training</mark>

Example: Use a logging library like Weights and Biases (wandb) to track the training progress, including loss curves, evaluation metrics, and hardware utilization.

```python
import wandb

wandb.init(project="peft-adapter-training")

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],  # Set up early stopping
)
trainer.train()
```

* Tip: Regularly check the training logs to identify any anomalies or convergence issues.
* Best Practice: Set up early stopping criteria to prevent overfitting and save computational resources.
* Potential Error: Neglecting to monitor the training progress, leading to suboptimal results or wasted resources.

Remember to refer to the official Hugging Face PEFT documentation for the most up-to-date information and API references.&#x20;

These code examples are meant to provide a starting point and may need to be adapted to your specific use case.


# After fine tuning LLama3

The files you have obtained after fine-tuning the LLama3 model are the essential components needed to run inference with your fine-tuned model.&#x20;

Here's what each file represents and how you can use them to run your fine-tuned LLM:

<mark style="color:yellow;">**`adapter_config.json`**</mark><mark style="color:yellow;">**:**</mark> This file contains the configuration settings for the adapter (LoRA) used during fine-tuning. It includes information such as the base model path, LoRA hyperparameters, target modules, and more.

<mark style="color:yellow;">**`adapter_model.bin`**</mark><mark style="color:yellow;">**:**</mark> This file contains the trained LoRA weights. It represents the learned adaptations to the base model during fine-tuning.

<mark style="color:yellow;">**`checkpoint-*`**</mark><mark style="color:yellow;">**:**</mark> These files (e.g., `checkpoint-112`, `checkpoint-28`, etc.) represent the saved model checkpoints at different stages of the training process. They contain the model's state at specific iterations or epochs.

<mark style="color:yellow;">**`config.json`**</mark><mark style="color:yellow;">**:**</mark> This file contains the configuration settings for the base model, such as the model architecture, hidden size, number of layers, and other hyperparameters.

<mark style="color:yellow;">**`README.md`**</mark><mark style="color:yellow;">**:**</mark> This file provides information about the fine-tuned model, including the training details, evaluation results, and any additional notes.

<mark style="color:yellow;">**`special_tokens_map.json`**</mark><mark style="color:yellow;">**:**</mark> This file maps special token names to their corresponding token IDs in the tokenizer.

<mark style="color:yellow;">**`tokenizer_config.json`**</mark> and <mark style="color:yellow;">**`tokenizer.json`**</mark>: These files contain the configuration and trained weights of the tokenizer used by the model.

To run inference with your fine-tuned LLM using these files, you can follow these steps:

#### <mark style="color:green;">Load the base model (LLama3) using the</mark> <mark style="color:green;"></mark><mark style="color:green;">`config.json`</mark> <mark style="color:green;"></mark><mark style="color:green;">file</mark>

```python
from transformers import LlamaForCausalLM, AutoConfig

config = AutoConfig.from_pretrained("path/to/config.json")
model = LlamaForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B", config=config)
```

#### <mark style="color:green;">Load the LoRA adapter using the</mark> <mark style="color:green;"></mark><mark style="color:green;">`adapter_config.json`</mark> <mark style="color:green;"></mark><mark style="color:green;">and</mark> <mark style="color:green;"></mark><mark style="color:green;">`adapter_model.bin`</mark> <mark style="color:green;"></mark><mark style="color:green;">files:</mark>

```python
from peft import PeftModel

model = PeftModel.from_pretrained(model, "path/to/adapter_model.bin")
```

#### <mark style="color:green;">Load the tokenizer using the</mark> <mark style="color:green;"></mark><mark style="color:green;">`tokenizer_config.json`</mark> <mark style="color:green;"></mark><mark style="color:green;">and</mark> <mark style="color:green;"></mark><mark style="color:green;">`tokenizer.json`</mark> <mark style="color:green;"></mark><mark style="color:green;">files:</mark>

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("path/to/tokenizer_config.json")
```

#### <mark style="color:green;">Use the loaded model and tokenizer to run inference on your input text:</mark>

```python
input_text = "Your input text here"
input_ids = tokenizer.encode(input_text, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(input_ids, max_length=100)

generated_text = tokenizer.decode(outputs[0])
print(generated_text)
```

Make sure to replace `"path/to/..."` with the actual paths to your files.

By following these steps, you can load your fine-tuned LLM and run inference on new input text using the trained LoRA adapter and tokenizer.

Remember to have the necessary dependencies installed, such as the `transformers` and `peft` libraries, and ensure that you have the required hardware (GPU) and sufficient memory to run the model.

You can refer to the `README.md` file for any additional instructions or notes specific to your fine-tuned model.


# Merging Model Weights

To merge your fine-tuned LoRA adapter with the base model and create a single model that can be used for inference, you can use the `peft` library's `merge_and_unload()` function.&#x20;

Here's how you can do it:

<mark style="color:green;">Load the base model and the LoRA adapter</mark>

```python
from transformers import LlamaForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_path = "/home/paperspace/axolotl/models/Meta-Llama-3-8B"
lora_model_path = "path/to/llama3-out"

tokenizer = AutoTokenizer.from_pretrained(base_model_path)
base_model = LlamaForCausalLM.from_pretrained(base_model_path)
lora_model = PeftModel.from_pretrained(base_model, lora_model_path)
```

<mark style="color:green;">Merge the LoRA adapter with the base model</mark>

```python
merged_model = lora_model.merge_and_unload()
```

<mark style="color:green;">Save the merged model in the desired format (e.g., SafeTensors)</mark>

```python
merged_model.save_pretrained("path/to/merged_model", safe_serialization=True)
```

This will save the merged model in the SafeTensors format, which is compatible with TensorRT-LLM.

<mark style="color:green;">Copy the necessary files from the base model directory to the merged model directory</mark>

* `config.json`
* `generation_config.json`
* `special_tokens_map.json`
* `tokenizer_config.json`
* `tokenizer.json`

You can use the following commands to copy these files:

```bash
cp /home/paperspace/axolotl/models/Meta-Llama-3-8B/config.json path/to/merged_model/
cp /home/paperspace/axolotl/models/Meta-Llama-3-8B/generation_config.json path/to/merged_model/
cp /home/paperspace/axolotl/models/Meta-Llama-3-8B/special_tokens_map.json path/to/merged_model/
cp /home/paperspace/axolotl/models/Meta-Llama-3-8B/tokenizer_config.json path/to/merged_model/
cp /home/paperspace/axolotl/models/Meta-Llama-3-8B/tokenizer.json path/to/merged_model/
```

Make sure to replace `"path/to/merged_model"` with the actual path where you saved the merged model.

After following these steps, you should have a merged model directory that contains the necessary files for TensorRT-LLM optimization:

* `config.json`
* `generation_config.json`
* `model-00001-of-00001.safetensors` (assuming a single SafeTensors file)
* `special_tokens_map.json`
* `tokenizer_config.json`
* `tokenizer.json`

You can now use this merged model directory as input to TensorRT-LLM for optimization and deployment.

Note: The exact number of SafeTensors files generated during the saving process may vary depending on the model size and configuration. Make sure to include all the generated SafeTensors files in the merged model directory.


# Merge Lora Instructions

merge\_lora

### <mark style="color:blue;">Training LoRA (Low-Rank Adapter)</mark>

* Axolotl allows you to fine-tune a base model using LoRA, which is a parameter-efficient fine-tuning method.
* You can train a LoRA adapter on top of the base model using a configuration file that specifies the training details, such as the dataset, hyperparameters, and LoRA-specific settings.

#### <mark style="color:green;">Example configuration snippet</mark>

```yaml
load_in_8bit: true
load_in_4bit: false
strict: false
sequence_len: 4096
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
```

#### <mark style="color:green;">Merging LoRA with the Base Model</mark>

* After training the LoRA adapter, you need to merge it with the base model to create a single, fine-tuned model.
* Axolotl provides a command to merge the LoRA adapter using the <mark style="color:yellow;">**`axolotl.cli.merge_lora`**</mark> command.
* Typical command to merge a local LoRA:

{% code overflow="wrap" %}

```
python3 -m axolotl.cli.merge_lora examples/llama-3/lora-8b.yml --lora_model_dir="llama4-out" 
```

{% endcode %}

* If the LoRA model is not stored locally, you may need to download it first and specify the local directory using the <mark style="color:yellow;">**`--lora_model_dir`**</mark> argument.
* If you encounter CUDA memory issues during merging, you can try merging in system RAM by setting <mark style="color:yellow;">**`CUDA_VISIBLE_DEVICES=""`**</mark> before the merge command.

<details>

<summary><mark style="color:green;"><strong>merge_lora script analysis</strong></mark></summary>

The script is a command-line interface (CLI) tool to merge a trained LoRA (Low-Rank Adaptation) model into a base model.

### <mark style="color:blue;">It imports the following external classes and modules</mark>

* <mark style="color:yellow;">**`Path`**</mark> from the <mark style="color:yellow;">**`pathlib`**</mark> module: This class provides an object-oriented interface for working with file and directory paths.
* <mark style="color:yellow;">**`fire`**</mark> module: This is a library for automatically generating command-line interfaces from Python functions and classes.
* <mark style="color:yellow;">**`transformers`**</mark> module: This is the Hugging Face Transformers library, which provides state-of-the-art pre-trained models for natural language processing tasks.
* <mark style="color:yellow;">**`do_merge_lora`**</mark>, <mark style="color:yellow;">**`load_cfg`**</mark>, and <mark style="color:yellow;">**`print_axolotl_text_art`**</mark> from the <mark style="color:yellow;">**`axolotl.cli`**</mark> module: These are custom functions specific to the Axolotl project, likely related to merging LoRA models and loading configuration files.
* <mark style="color:yellow;">**`TrainerCliArgs`**</mark> from the <mark style="color:yellow;">**`axolotl.common.cli`**</mark> module: This is likely a custom class defining the command-line arguments for the trainer.

1. The script defines a <mark style="color:yellow;">**`do_cli`**</mark> function that takes a <mark style="color:yellow;">**`config`**</mark> parameter (default is `Path("examples/")`) and any additional keyword arguments (<mark style="color:yellow;">**`**kwargs`**</mark>).&#x20;

#### <mark style="color:green;">This function is the main entry point for the CLI.</mark>

Inside the <mark style="color:yellow;">**`do_cli`**</mark> function:

* It prints the Axolotl text art using the <mark style="color:yellow;">**`print_axolotl_text_art`**</mark> function.
* It creates a <mark style="color:yellow;">**`transformers.HfArgumentParser`**</mark> instance with <mark style="color:yellow;">**`TrainerCliArgs`**</mark> to parse the command-line arguments.
* It parses the command-line arguments into <mark style="color:yellow;">**`parsed_cli_args`**</mark> using the <mark style="color:yellow;">**`parse_args_into_dataclasses`**</mark> method.
* It sets <mark style="color:yellow;">**`parsed_cli_args.merge_lora`**</mark> to <mark style="color:yellow;">**`True`**</mark>**.**
* It loads the configuration using the <mark style="color:yellow;">**`load_cfg`**</mark> function with the provided <mark style="color:yellow;">**`config`**</mark> path and additional keyword arguments.
* It performs some validation and sets default values for the <mark style="color:yellow;">**`lora_model_dir`**</mark> and <mark style="color:yellow;">**`output_dir`**</mark> based on the loaded configuration.
* It sets <mark style="color:yellow;">**`load_in_4bit`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`load_in_8bit`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`flash_attention`**</mark><mark style="color:yellow;">**,**</mark><mark style="color:yellow;">**&#x20;**</mark><mark style="color:yellow;">**`deepspeed`**</mark>, and <mark style="color:yellow;">**`fsdp`**</mark> to <mark style="color:yellow;">**`False`**</mark> or <mark style="color:yellow;">**`None`**</mark>.
* It calls the <mark style="color:yellow;">**`do_merge_lora`**</mark> function with the loaded configuration (<mark style="color:yellow;">**`parsed_cfg`**</mark>) and parsed command-line arguments (<mark style="color:yellow;">**`parsed_cli_args`**</mark>).

Finally, if the script is run as the main module (<mark style="color:yellow;">**`__name__ == "__main__"`**</mark>), it uses the <mark style="color:yellow;">**`fire.Fire`**</mark> function to automatically generate a command-line interface for the <mark style="color:yellow;">**`do_cli`**</mark> function.

To learn more about the external classes and modules used in this script:

* For the <mark style="color:yellow;">**`fire`**</mark> module, refer to the Fire documentation: <https://github.com/google/python-fire>
* For the <mark style="color:yellow;">**`transformers`**</mark> module, refer to the Hugging Face Transformers documentation: <https://huggingface.co/docs/transformers/>
* For `axolotl.cli` and `axolotl.common.cli`, these are likely custom modules specific to the Axolotl project. You should refer to the project's documentation or codebase for more information.

</details>

<mark style="color:blue;">python3</mark> -m <mark style="color:purple;">axolotl.cli.merge\_lora</mark> <mark style="color:yellow;">examples/llama-3/lora-8b.yml</mark> --lora\_model\_dir="llama4-out"&#x20;

* If you trained a QLoRA (Quantized LoRA) model that can only fit into GPU memory at 4-bit quantization, merging can be challenging due to memory constraints.
* To merge a QLoRA model, you need to ensure that the model remains quantized throughout the merging process.
* Modify the merge script to load the model with the appropriate quantization configuration, such as using the <mark style="color:yellow;">**`bitsandbytes`**</mark> library for 4-bit quantization.
* Use libraries like <mark style="color:yellow;">**`accelerate`**</mark> for managing device memory and offloading parts of the model to CPU or disk if necessary.

#### <mark style="color:green;">Warnings and Considerations</mark>

* Axolotl may raise warnings related to sample packing without flash attention or SDP attention, indicating that it does not handle cross-attention in those cases.
* It is recommended to set <mark style="color:yellow;">**`load_in_8bit: true`**</mark> for LoRA fine-tuning, even if the warning suggests otherwise.
* Merging quantized models, especially with parameter-efficient fine-tuning methods like QLoRA, can be complex and may require adjustments to the standard merging scripts.

#### <mark style="color:green;">Merging Duration</mark>

* The time taken to merge a LoRA adapter back to the base model depends on the model size and hardware.
* For a 70B parameter model fine-tuned on 4 A100 GPUs, the merging process can take a significant amount of time (over an hour or more).

These are the key ideas and considerations when using model merging with Axolotl. It's important to carefully configure the training and merging processes, handle quantization appropriately, and be aware of potential memory constraints and warnings.&#x20;

Consulting the official Axolotl documentation and seeking guidance from the Axolotl community can provide further assistance in navigating the model merging process.

### <mark style="color:blue;">Tips and Tricks</mark>

Ensure that you have generated the LoRA model before attempting to merge it.&#x20;

The typical workflow is to first train the LoRA model and then merge it in a separate command.

Use the <mark style="color:yellow;">**`--merge_lora`**</mark> flag along with the <mark style="color:yellow;">**`--lora_model_dir`**</mark> flag to specify the directory containing the trained LoRA model.&#x20;

For example:

{% code overflow="wrap" %}

```bash
accelerate launch scripts/finetune.py examples/openllama-3b/qlora.yml --merge_lora --lora_model_dir="./qlora-out"
```

{% endcode %}

Set <mark style="color:yellow;">**`--load_in_8bit=False`**</mark> and <mark style="color:yellow;">**`--load_in_4bit=False`**</mark> when merging the LoRA model to avoid compatibility issues. The 4-bit and 8-bit loading options are not supported for merging.

If you encounter the error "ValueError: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is", try using the old command for merging:

{% code overflow="wrap" %}

```bash
python3 scripts/finetune.py examples/code-llama/7b/qlora.yml --merge_lora --lora_model_dir="./qlora-out" --load_in_8bit=False --load_in_4bit=False
```

{% endcode %}

Make sure you have the latest version of Axolotl installed, as some issues might have been resolved in newer versions.

If you encounter CUDA-related errors, try setting the <mark style="color:yellow;">**`CUDA_VISIBLE_DEVICES`**</mark> environment variable to specify the desired GPU device. For example:

{% code overflow="wrap" fullWidth="false" %}

```bash
CUDA_VISIBLE_DEVICES="0" accelerate launch scripts/finetune.py examples/openllama-3b/qlora.yml --merge_lora --lora_model_dir="./qlora-out" --load_in_8bit=False --load_in_4bit=False
```

{% endcode %}

After merging the LoRA model, only the <mark style="color:yellow;">**`pytorch_model.bin`**</mark> file will work.&#x20;

Attempting to quantize it directly may fail.&#x20;

To quantize the merged model, you may need to copy additional files (e.g., <mark style="color:yellow;">**`tokenizer.model`**</mark>) from the original model and use external tools like <mark style="color:yellow;">**`llama.cpp`**</mark> for quantization.

If you encounter issues with missing files or directories, double-check the paths specified in the command and ensure that the necessary files and directories exist.

Remember to refer to the Axolotl documentation and the GitHub issues for the most up-to-date information and troubleshooting steps.


# Axolotl Configuration Files

Within Axolotl the training file is a <mark style="color:yellow;">YAML file</mark>. &#x20;

There are an array of <mark style="color:yellow;">example training scripts</mark> spread across *<mark style="color:yellow;">**24 different types of models.**</mark>* &#x20;

### <mark style="color:blue;">Below are the primary categories for the</mark> <mark style="color:yellow;">YAML Training Configuration File</mark>

| <mark style="color:green;">Category</mark> |
| ------------------------------------------ |
| Model Configuration                        |
| Data Loading and Processing                |
| Sequence Configuration                     |
| Adapter Configuration                      |
| Logging and Monitoring                     |
| Training Configuration                     |
| Training Options                           |
| Attention Mechanisms                       |
| Training Schedules and Evaluation          |
| Debugging and Optimization                 |
| Special Tokens                             |

### **The balance of this module is describing each of these configuration areas.**


# Configuration Options

Sourced from the Axolotl documentation

### <mark style="color:blue;">A complete list of all configuration options from the Axolotl github repository</mark>

{% code fullWidth="false" %}

```yaml
# This is the huggingface model that contains *.pt, *.safetensors, or *.bin files
# This can also be a relative path to a model on disk
base_model: ./llama-7b-hf
# You can specify an ignore pattern if the model repo contains more than 1 model type (*.pt, etc)
base_model_ignore_patterns:
# If the base_model repo on hf hub doesn't include configuration .json files,
# You can set that here, or leave this empty to default to base_model
base_model_config: ./llama-7b-hf
# You can specify to choose a specific model revision from huggingface hub
revision_of_model:
# Optional tokenizer configuration path in case you want to use a different tokenizer
# than the one defined in the base model
tokenizer_config:
# If you want to specify the type of model to load, AutoModelForCausalLM is a good choice too
model_type: AutoModelForCausalLM
# Corresponding tokenizer for the model AutoTokenizer is a good choice
tokenizer_type: AutoTokenizer
# Trust remote code for untrusted source
trust_remote_code:
# use_fast option for tokenizer loading from_pretrained, default to True
tokenizer_use_fast:
# Whether to use the legacy tokenizer setting, defaults to True
tokenizer_legacy:
# Resize the model embeddings when new tokens are added to multiples of 32
# This is reported to improve training speed on some models
resize_token_embeddings_to_32x:

# (Internal use only)
# Used to identify which the model is based on
is_falcon_derived_model:
is_llama_derived_model:
is_qwen_derived_model:
# Please note that if you set this to true, `padding_side` will be set to "left" by default
is_mistral_derived_model:

# optional overrides to the base model configuration
overrides_of_model_config:
  # RoPE Scaling https://github.com/huggingface/transformers/pull/24653
  rope_scaling:
    type: # linear | dynamic
    factor: # float

# optional overrides to the bnb 4bit quantization configuration
# https://huggingface.co/docs/transformers/main/main_classes/quantization#transformers.BitsAndBytesConfig
bnb_config_kwargs:
  # These are default values
  llm_int8_has_fp16_weight: false
  bnb_4bit_quant_type: nf4
  bnb_4bit_use_double_quant: true


# Whether you are training a 4-bit GPTQ quantized model
gptq: true

# This will attempt to quantize the model down to 8 bits and use adam 8 bit optimizer
load_in_8bit: true
# Use bitsandbytes 4 bit
load_in_4bit:

# Use CUDA bf16
bf16: true # bool or 'full' for `bf16_full_eval`. require >=ampere
# Use CUDA fp16
fp16: true
# Use CUDA tf32
tf32: true # require >=ampere

# No AMP (automatic mixed precision)
bfloat16: true # require >=ampere
float16: true

# Limit the memory for all available GPUs to this amount (if an integer, expressed in gigabytes); default: unset
gpu_memory_limit: 20GiB
# Do the LoRA/PEFT loading on CPU -- this is required if the base model is so large it takes up most or all of the available GPU VRAM, e.g. during a model and LoRA merge
lora_on_cpu: true

# A list of one or more datasets to finetune the model with
datasets:
  # HuggingFace dataset repo | s3://,gs:// path | "json" for local dataset, make sure to fill data_files
  - path: vicgalle/alpaca-gpt4
  # The type of prompt to use for training. [alpaca, sharegpt, gpteacher, oasst, reflection]
    type: alpaca # format | format:<prompt_style> (chat/instruct) | <prompt_strategies>.load_<load_fn>
    ds_type: # Optional[str] (json|arrow|parquet|text|csv) defines the datatype when path is a file
    data_files: # Optional[str] path to source data files
    shards: # Optional[int] number of shards to split data into
    name: # Optional[str] name of dataset configuration to load
    train_on_split: train # Optional[str] name of dataset split to load from

    # Optional[str] fastchat conversation type, only used with type: sharegpt
    conversation: # Options (see Conversation 'name'): https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py
    field_human: # Optional[str]. Human key to use for conversation.
    field_model: # Optional[str]. Assistant key to use for conversation.
    # Add additional keys from your dataset as input or output roles
    roles:
      input: # Optional[List[str]]. These will be masked based on train_on_input
      output: # Optional[List[str]].

  # Custom user instruction prompt
  - path: repo
    type:
      # The below are defaults. only set what's needed if you use a different column name.
      system_prompt: ""
      system_format: "{system}"
      field_system: system
      field_instruction: instruction
      field_input: input
      field_output: output

      # Customizable to be single line or multi-line
      # Use {instruction}/{input} as key to be replaced
      # 'format' can include {input}
      format: |-
        User: {instruction} {input}
        Assistant:
      # 'no_input_format' cannot include {input}
      no_input_format: "{instruction} "

      # For `completion` datsets only, uses the provided field instead of `text` column
      field:

# If false, the datasets will not be shuffled and will keep their original order in `datasets`.
# The same applies to the `test_datasets` option and the `pretraining_dataset` option. Default is true.
shuffle_merged_datasets: true

# A list of one or more datasets to eval the model with.
# You can use either test_datasets, or val_set_size, but not both.
test_datasets:
  - path: /workspace/data/eval.jsonl
    ds_type: json
    # You need to specify a split. For "json" datasets the default split is called "train".
    split: train
    type: completion
    data_files:
      - /workspace/data/eval.jsonl

# use RL training: 'dpo', 'ipo', 'kto_pair'
rl:

# Saves the desired chat template to the tokenizer_config.json for easier inferencing
# Currently supports chatml and inst (mistral/mixtral)
chat_template: chatml
# Changes the default system message
default_system_message: You are a helpful assistant. Please give a long and detailed answer. # Currently only supports chatml.
# Axolotl attempts to save the dataset as an arrow after packing the data together so
# subsequent training attempts load faster, relative path
dataset_prepared_path: data/last_run_prepared
# Push prepared dataset to hub
push_dataset_to_hub: # repo path
# The maximum number of processes to use while preprocessing your input dataset. This defaults to `os.cpu_count()`
# if not set.
dataset_processes: # defaults to os.cpu_count() if not set
# Keep dataset in memory while preprocessing
# Only needed if cached dataset is taking too much storage
dataset_keep_in_memory:
# push checkpoints to hub
hub_model_id: # private repo path to push finetuned model
# how to push checkpoints to hub
# https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.hub_strategy
hub_strategy:
# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets
# Required to be true when used in combination with `push_dataset_to_hub`
hf_use_auth_token: # boolean
# How much of the dataset to set aside as evaluation. 1 = 100%, 0.50 = 50%, etc. 0 for no eval.
val_set_size: 0.04
# Num shards for whole dataset
dataset_shard_num:
# Index of shard to use for whole dataset
dataset_shard_idx:

# The maximum length of an input to train with, this should typically be less than 2048
# as most models have a token/context limit of 2048
sequence_len: 2048
# Pad inputs so each step uses constant sized buffers
# This will reduce memory fragmentation and may prevent OOMs, by re-using memory more efficiently
pad_to_sequence_len:
# Use efficient multi-packing with block diagonal attention and per sequence position_ids. Recommend set to 'true'
sample_packing:
# Set to 'false' if getting errors during eval with sample_packing on.
eval_sample_packing:
# You can set these packing optimizations AFTER starting a training at least once.
# The trainer will provide recommended values for these values.
sample_packing_eff_est:
total_num_tokens:

# Passed through to transformers when loading the model when launched without accelerate
# Use `sequential` when training w/ model parallelism to limit memory
device_map:
# Defines the max memory usage per gpu on the system. Passed through to transformers when loading the model.
max_memory:

# If you want to use 'lora' or 'qlora' or leave blank to train all parameters in original model
adapter: lora
# If you already have a lora model trained that you want to load, put that here.
# This means after training, if you want to test the model, you should set this to the value of `output_dir`.
# Note that if you merge an adapter to the base model, a new subdirectory `merged` will be created under the `output_dir`.
lora_model_dir:

# LoRA hyperparameters
# For more details about the following options, see:
# https://www.anyscale.com/blog/fine-tuning-llms-lora-or-full-parameter-an-in-depth-analysis-with-llama-2
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - v_proj
#  - k_proj
#  - o_proj
#  - gate_proj
#  - down_proj
#  - up_proj
lora_target_linear: # If true, will target all linear modules
peft_layers_to_transform: # The layer indices to transform, otherwise, apply to all layers

# If you added new tokens to the tokenizer, you may need to save some LoRA modules because they need to know the new tokens.
# For LLaMA and Mistral, you need to save `embed_tokens` and `lm_head`. It may vary for other models.
# `embed_tokens` converts tokens to embeddings, and `lm_head` converts embeddings to token probabilities.
# https://github.com/huggingface/peft/issues/334#issuecomment-1561727994
lora_modules_to_save:
#  - embed_tokens
#  - lm_head

lora_fan_in_fan_out: false

peft:
  # Configuration options for loftq initialization for LoRA
  # https://huggingface.co/docs/peft/developer_guides/quantization#loftq-initialization
  loftq_config:
    loftq_bits:  # typically 4 bits

# ReLoRA configuration
# Must use either 'lora' or 'qlora' adapter, and does not support fsdp or deepspeed
relora_steps: # Number of steps per ReLoRA restart
relora_warmup_steps: # Number of per-restart warmup steps
relora_anneal_steps: # Number of anneal steps for each relora cycle
relora_prune_ratio: # threshold for optimizer magnitude when pruning
relora_cpu_offload: # True to perform lora weight merges on cpu during restarts, for modest gpu memory savings

# wandb configuration if you're using it
# Make sure your `WANDB_API_KEY` environment variable is set (recommended) or you login to wandb with `wandb login`.
wandb_mode: # "offline" to save run metadata locally and not sync to the server, "disabled" to turn off wandb
wandb_project: # Your wandb project name
wandb_entity: # A wandb Team name if using a Team
wandb_watch:
wandb_name: # Set the name of your wandb run
wandb_run_id: # Set the ID of your wandb run
wandb_log_model: # "checkpoint" to log model to wandb Artifacts every `save_steps` or "end" to log only at the end of training

# mlflow configuration if you're using it
mlflow_tracking_uri: # URI to mlflow
mlflow_experiment_name: # Your experiment name
hf_mlflow_log_artifacts:  # set to true to copy each saved checkpoint on each save to mlflow artifact registry

# Where to save the full-finetuned model to
output_dir: ./completed-model

# Whether to use torch.compile and which backend to use
torch_compile:  # bool
torch_compile_backend:  # Optional[str]

# Training hyperparameters

# If greater than 1, backpropagation will be skipped and the gradients will be accumulated for the given number of steps.
gradient_accumulation_steps: 1
# The number of samples to include in each batch. This is the number of samples sent to each GPU.
micro_batch_size: 2
eval_batch_size:
num_epochs: 4
warmup_steps: 100  # cannot use with warmup_ratio
warmup_ratio: 0.05  # cannot use with warmup_steps
learning_rate: 0.00003
lr_quadratic_warmup:
logging_steps:
eval_steps: # Leave empty to eval at each epoch, integers for every N steps. decimal for fraction of total steps
evals_per_epoch: # number of times per epoch to run evals, mutually exclusive with eval_steps
save_strategy: # Set to `no` to skip checkpoint saves
save_steps: # Leave empty to save at each epoch
saves_per_epoch: # number of times per epoch to save a checkpoint, mutually exclusive with save_steps
save_total_limit: # Checkpoints saved at a time
# Maximum number of iterations to train for. It precedes num_epochs which means that
# if both are set, num_epochs will not be guaranteed.
# e.g., when 1 epoch is 1000 steps => `num_epochs: 2` and `max_steps: 100` will train for 100 steps
max_steps:

eval_table_size: # Approximate number of predictions sent to wandb depending on batch size. Enabled above 0. Default is 0
eval_max_new_tokens: # Total number of tokens generated for predictions sent to wandb. Default is 128
eval_causal_lm_metrics: # HF evaluate metrics used during evaluation. Default is ["sacrebleu", "comet", "ter", chrf]

loss_watchdog_threshold: # High loss value, indicating the learning has broken down (a good estimate is ~2 times the loss at the start of training)
loss_watchdog_patience: # Number of high-loss steps in a row before the trainer aborts (default: 3)

# Save model as safetensors (require safetensors package)
save_safetensors:

# Whether to mask out or include the human's prompt from the training labels
train_on_inputs: false
# Group similarly sized data to minimize padding.
# May be slower to start, as it must download and sort the entire dataset.
# Note that training loss may have an oscillating pattern with this enabled.
group_by_length: false

# Whether to use gradient checkpointing https://huggingface.co/docs/transformers/v4.18.0/en/performance#gradient-checkpointing
gradient_checkpointing: false
# additional kwargs to pass to the trainer for gradient checkpointing
# gradient_checkpointing_kwargs:
#   use_reentrant: true

# Stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3

# Specify a scheduler and kwargs to use with the optimizer
lr_scheduler: # 'one_cycle' | 'log_sweep' | empty for cosine
lr_scheduler_kwargs:
cosine_min_lr_ratio: # decay lr to some percentage of the peak lr, e.g. cosine_min_lr_ratio=0.1 for 10% of peak lr
cosine_constant_lr_ratio: # freeze lr at some percentage of the step, e.g. cosine_constant_lr_ratio=0.8 means start cosine_min_lr at 80% of training step (https://arxiv.org/pdf/2308.04014.pdf)

# For one_cycle optim
lr_div_factor: # Learning rate div factor

# Specify optimizer
# Valid values are driven by the Transformers OptimizerNames class, see:
# https://github.com/huggingface/transformers/blob/95b374952dc27d8511541d6f5a4e22c9ec11fb24/src/transformers/training_args.py#L134
#
# Note that not all optimizers may be available in your environment, ex: 'adamw_anyprecision' is part of
# torchdistx, 'adamw_bnb_8bit' is part of bnb.optim.Adam8bit, etc. When in doubt, it is recommended to start with the optimizer used
# in the examples/ for your model and fine-tuning use case.
#
# Valid values for 'optimizer' include:
# - adamw_hf
# - adamw_torch
# - adamw_torch_fused
# - adamw_torch_xla
# - adamw_apex_fused
# - adafactor
# - adamw_anyprecision
# - sgd
# - adagrad
# - adamw_bnb_8bit
# - lion_8bit
# - lion_32bit
# - paged_adamw_32bit
# - paged_adamw_8bit
# - paged_lion_32bit
# - paged_lion_8bit
# - galore_adamw
# - galore_adamw_8bit
# - galore_adafactor
# - galore_adamw_layerwise
# - galore_adamw_8bit_layerwise
# - galore_adafactor_layerwise
optimizer:
# Dictionary of arguments to pass to the optimizer
optim_args:
# For Galore Optimizers the following optim_args are available
# rank:  # type: int
# update_proj_gap  # type: int
# scale  # type: float
# proj_type:  # type: str, default = std

# The target modules to optimize, i.e. the module names that you would like to train, right now this is used only for GaLore algorithm
optim_target_modules:
# - self_attn  # for llama
# - mlp

# Specify weight decay
weight_decay:
# adamw hyperparams
adam_beta1:
adam_beta2:
adam_epsilon:
# Gradient clipping max norm
max_grad_norm:

# Augmentation techniques
# NEFT https://arxiv.org/abs/2310.05914, set this to a number (paper default is 5) to add noise to embeddings
# currently only supported on Llama and Mistral
neftune_noise_alpha:

# Whether to bettertransformers
flash_optimum:
# Whether to use xformers attention patch https://github.com/facebookresearch/xformers:
xformers_attention:
# Whether to use flash attention patch https://github.com/Dao-AILab/flash-attention:
flash_attention:
flash_attn_cross_entropy:  # Whether to use flash-attention cross entropy implementation - advanced use only
flash_attn_rms_norm:  # Whether to use flash-attention rms norm implementation - advanced use only
flash_attn_fuse_qkv: # Whether to fuse QKV into a single operation
flash_attn_fuse_mlp: # Whether to fuse part of the MLP into a single operation
# Whether to use scaled-dot-product attention
# https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html
sdp_attention:
# Shifted-sparse attention (only llama) - https://arxiv.org/pdf/2309.12307.pdf
s2_attention:
# Resume from a specific checkpoint dir
resume_from_checkpoint:
# If resume_from_checkpoint isn't set and you simply want it to start where it left off.
# Be careful with this being turned on between different models.
auto_resume_from_checkpoints: false

# Don't mess with this, it's here for accelerate and torchrun
local_rank:

# Add or change special tokens.
# If you add tokens here, you don't need to add them to the `tokens` list.
special_tokens:
  # bos_token: "<s>"
  # eos_token: "</s>"
  # unk_token: "<unk>"

# Add extra tokens.
tokens:

# FSDP
fsdp:
fsdp_config:

# Deepspeed config path. e.g., deepspeed_configs/zero3.json
deepspeed:

# Advanced DDP Arguments
ddp_timeout:
ddp_bucket_cap_mb:
ddp_broadcast_buffers:

# Path to torch distx for optim 'adamw_anyprecision'
torchdistx_path:

# Set to HF dataset for type: 'completion' for streaming instead of pre-tokenize
pretraining_dataset:

# Debug mode
debug:

# Seed
seed:

# Allow overwrite yml config using from cli
strict:
```

{% endcode %}


# Model Configuration

Model Configuration

| Field Name                       | Description                                                                                                |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `base_model`                     | Specifies the Hugging Face model or its path containing model files (`*.pt`, `*.safetensors`, or `*.bin`). |
| `base_model_ignore_patterns`     | Allows specifying an ignore pattern for model files in the repository.                                     |
| `base_model_config`              | Location of the configuration `.json` file if not included in the base model repository.                   |
| `model_revision`                 | Specifies a specific model revision from Hugging Face's hub.                                               |
| `tokenizer_config`               | Optional override for the tokenizer configuration, different from the base model.                          |
| `model_type`                     | Defines the type of model to load, e.g., `AutoModelForCausalLM`.                                           |
| `tokenizer_type`                 | Corresponding tokenizer type, e.g., `AutoTokenizer`.                                                       |
| `trust_remote_code`              | Allows trusting remote code for untrusted sources.                                                         |
| `tokenizer_use_fast`             | Indicates whether to use the `use_fast` option for tokenizer loading.                                      |
| `tokenizer_legacy`               | Specifies whether to use the legacy tokenizer setting.                                                     |
| `resize_token_embeddings_to_32x` | Resizes model embeddings to multiples of 32 when new tokens are added.                                     |
| `is_falcon_derived_model`        | Boolean flag indicating if the model is derived from Falcon.                                               |
| `is_llama_derived_model`         | Boolean flag indicating if the model is derived from Llama.                                                |
| `is_mistral_derived_model`       | Boolean flag indicating if the model is derived from Mistral.                                              |
| `model_config`                   | Optional overrides for the base model configuration, including RoPE Scaling settings.                      |


# Data Loading and Processing

<table data-full-width="false"><thead><tr><th width="288">Field Name</th><th>Explanation</th></tr></thead><tbody><tr><td>datasets</td><td>Datasets provide the training data for the model.</td></tr><tr><td><mark style="color:blue;">path</mark></td><td><mark style="color:blue;">The</mark> <mark style="color:yellow;"><code>path</code></mark> <mark style="color:blue;">field specifies the location of the dataset. It can be a</mark> <mark style="color:yellow;">HuggingFace dataset repository</mark> <mark style="color:blue;">path, a</mark> <mark style="color:yellow;">cloud storage path</mark> <mark style="color:blue;">(e.g., s3:// or gs://), or a</mark> <mark style="color:yellow;">local "json" file.</mark></td></tr><tr><td>type</td><td>The <code>type</code> field defines the <mark style="color:blue;">type of prompt strategy used for training the model</mark>. For example, it can be set to <mark style="color:yellow;">"alpaca," "sharegpt," "gpteacher," "oasst," or "reflection."</mark> Each type may have specific characteristics tailored to different training approaches.</td></tr><tr><td>ds_type</td><td>The optional <code>ds_type</code> field <mark style="color:blue;">specifies the datatype when the <code>path</code> points to a local file.</mark> It can be set to <mark style="color:yellow;">"json," "arrow," "parquet," "text," or "csv,"</mark> depending on the format of the dataset file.</td></tr><tr><td>data_files</td><td>If necessary, the <mark style="color:yellow;"><code>data_files</code></mark> field can be used to specify the path to the source data files associated with the dataset. This field helps the training process locate the relevant data.</td></tr><tr><td>shards</td><td>The optional <mark style="color:yellow;"><code>shards</code></mark> field allows you to specify the number of <mark style="color:blue;">data shards into which the dataset should be divided</mark>. Sharding can help distribute the data efficiently for parallel processing during training.</td></tr><tr><td>name</td><td>You can provide an optional <mark style="color:yellow;"><code>name</code></mark> for the dataset configuration. This name can be useful for reference when working with multiple datasets during fine-tuning.</td></tr><tr><td>train_on_split</td><td>The optional <mark style="color:yellow;"><code>train_on_split</code></mark> field lets you <mark style="color:blue;">specify the name of the dataset split to load from</mark>. For instance, you might use "train" to load the training split of the dataset.</td></tr><tr><td>conversation</td><td>For specific types of prompts like "sharegpt," this optional field defines the fastchat conversation type. It's typically used in conjunction with the "sharegpt" type and allows customization of conversation style.</td></tr></tbody></table>

<table data-full-width="false"><thead><tr><th width="306">Custom User Prompt</th><th>Explanation</th></tr></thead><tbody><tr><td>system_prompt</td><td>The <mark style="color:yellow;"><code>system_prompt</code></mark> field is part of the <mark style="color:blue;">custom user prompt configuration.</mark> It defines the prompt provided to the system or assistant as part of the interaction with the model.</td></tr><tr><td>system_format</td><td><mark style="color:yellow;"><code>system_format</code></mark> specifies the format in which the system prompt is presented. In this case, it uses a placeholder "{system}" to represent the system's response.</td></tr><tr><td>field_system</td><td><mark style="color:yellow;"><code>field_system</code></mark> specifies the <mark style="color:blue;">name of the field where the system prompt is located in the dataset</mark>. It helps the training process identify the system's responses in the dataset.</td></tr><tr><td>field_instruction</td><td><mark style="color:yellow;"><code>field_instruction</code></mark> specifies the field name for the instruction or query provided to the model. It's used to extract user instructions or queries from the dataset.</td></tr><tr><td>field_input</td><td><mark style="color:yellow;"><code>field_input</code></mark> defines the field name where <mark style="color:blue;">user inputs are stored in the dataset</mark>. It's essential for the model to understand and respond to user inputs effectively.</td></tr><tr><td>field_output</td><td><mark style="color:yellow;"><code>field_output</code></mark> represents the field name where the <mark style="color:blue;">assistant's outputs or responses are stored in the dataset.</mark> It helps in training the model to generate appropriate responses.</td></tr><tr><td>format</td><td>The <mark style="color:yellow;"><code>format</code></mark> field allows customization of the conversation format. It can be configured to be single-line or multi-line and includes placeholders for instruction and input, making it flexible for various dialogue styles.</td></tr><tr><td>no_input_format</td><td><mark style="color:yellow;"><code>no_input_format</code></mark> defines the format of the conversation when there is no input (e.g., for system prompts). It's important for consistency and readability of the generated interactions.</td></tr><tr><td>field</td><td>For "completion" datasets, this field can be used to specify a custom field in the dataset to be used instead of the default "text" column. This customization can be beneficial for specific use cases.</td></tr></tbody></table>

<table data-full-width="false"><thead><tr><th width="307">Dataset Prepared Path</th><th>Explanation</th></tr></thead><tbody><tr><td>dataset_prepared_path</td><td>The <mark style="color:yellow;"><code>dataset_prepared_path</code></mark> specifies the relative path where the prepared dataset is saved as an Arrow file. This prepared dataset is packed together for more efficient loading during subsequent training attempts, enhancing training performance.</td></tr></tbody></table>

<table data-full-width="false"><thead><tr><th width="321">Push Dataset to Hub</th><th>Explanation</th></tr></thead><tbody><tr><td>push_dataset_to_hub</td><td>The <mark style="color:yellow;"><code>push_dataset_to_hub</code></mark> field specifies the <mark style="color:blue;">repository path to which the prepared dataset should be pushed</mark>. This feature is useful for sharing datasets with others, making it accessible through the HuggingFace dataset hub.</td></tr></tbody></table>

<table data-full-width="false"><thead><tr><th width="327">Dataset Processing</th><th>Explanation</th></tr></thead><tbody><tr><td>dataset_processes</td><td>The <mark style="color:yellow;"><code>dataset_processes</code></mark> field allows you to define the <mark style="color:blue;">maximum number of processes to use during preprocessing of the input dataset.</mark> If not set, it defaults to the number of CPU cores available, which can optimize data preparation for training.</td></tr></tbody></table>

<table data-full-width="false"><thead><tr><th width="336">Push Checkpoints to Hub</th><th>Explanation</th></tr></thead><tbody><tr><td>hub_model_id</td><td>The <mark style="color:yellow;"><code>hub_model_id</code></mark> field specifies the <mark style="color:blue;">repository path to which the finetuned model checkpoints should be pushed</mark>. It facilitates the sharing of finetuned models through the HuggingFace model hub, making them accessible to others.</td></tr><tr><td>hub_strategy</td><td>The <mark style="color:yellow;"><code>hub_strategy</code></mark> field, not specified in this example, is intended to define the strategy for pushing checkpoints to the hub. It allows you to customize the behavior when pushing the checkpoints.</td></tr></tbody></table>

<table data-full-width="false"><thead><tr><th width="339">Authentication Token</th><th>Explanation</th></tr></thead><tbody><tr><td>hf_use_auth_token</td><td>The <mark style="color:yellow;"><code>hf_use_auth_token</code></mark> field is a boolean value that determines whether to use HuggingFace's <code>u</code><mark style="color:yellow;"><code>se_auth_token</code></mark> for loading datasets. This is particularly useful for fetching private datasets and is required to be set to "true" when used with <mark style="color:yellow;"><code>push_dataset_to_hub</code>.</mark></td></tr></tbody></table>

<table data-full-width="false"><thead><tr><th width="342">Validation Set Size</th><th>Explanation</th></tr></thead><tbody><tr><td>val_set_size</td><td><mark style="color:yellow;"><code>val_set_size</code></mark> <mark style="color:blue;">specifies the fraction of the dataset that should be set aside for evaluation purposes.</mark> For example, a value of 0.04 means that 4% of the dataset will be reserved for evaluation, helping assess model performance.</td></tr></tbody></table>

<table data-full-width="false"><thead><tr><th width="344">Dataset Sharding</th><th>Explanation</th></tr></thead><tbody><tr><td>dataset_shard_num</td><td>The <mark style="color:yellow;"><code>dataset_shard_num</code></mark> and <mark style="color:yellow;"><code>dataset_shard_idx</code></mark> fields, although not specified in this example, may be intended for configuring dataset sharding. <mark style="color:yellow;"><code>dataset_shard_num</code></mark> could define the number of shards to use, and <mark style="color:yellow;"><code>dataset_shard_idx</code></mark> could specify the index of the shard to use for the entire dataset. These options can be helpful for efficient data processing and training with large datasets.</td></tr></tbody></table>


# Sequence Configuration

<table data-full-width="false"><thead><tr><th width="261">Field Name</th><th>Explanation</th></tr></thead><tbody><tr><td>sequence_len</td><td>The <code>sequence_len</code> field specifies the maximum length of an input sequence to use during training. This <mark style="color:yellow;">value should typically be less than 2048</mark>, as most models have a token/context limit of 2048. Setting it appropriately ensures that input sequences are within the model's capacity.</td></tr><tr><td>pad_to_sequence_len</td><td>The <code>pad_to_sequence_len</code> field is intended to <mark style="color:yellow;">pad inputs to a constant size</mark>, which can reduce memory fragmentation and potentially prevent Out of Memory (OOM) errors. It helps re-use memory more efficiently by ensuring that all input sequences have a uniform length. However, this field appears empty in the provided configuration and should have a value (e.g., <code>true</code> or <code>false</code>) to specify whether to pad to the <code>sequence_len</code>.</td></tr><tr><td>max_packed_sequence_len</td><td><code>max_packed_sequence_len</code> sets the maximum sequence length for concatenating training samples together. This concept is inspired by StackLLaMA and can be useful for efficient training. However, it's important to note that there is a FutureWarning that suggests this field will soon be deprecated.</td></tr><tr><td>sample_packing</td><td>The <code>sample_packing</code> field, when set to 'true,' indicates the use of <mark style="color:blue;">efficient multi-packing with block diagonal attention and per-sequence position_ids.</mark> This setting is recommended for optimizing training and memory efficiency, especially when dealing with long sequences.</td></tr><tr><td>eval_sample_packing</td><td><code>eval_sample_packing</code> allows you to enable or disable sample packing during evaluation. If you encounter errors during evaluation when sample packing is on, you can set this field to 'false' to disable it.</td></tr><tr><td>sample_packing_eff_est</td><td>This field is not fully specified in the provided configuration. It appears to be related to estimating the efficiency of sample packing optimizations but is left empty.</td></tr><tr><td>total_num_tokens</td><td><code>total_num_tokens</code> seems to be intended for specifying the total number of tokens in the dataset. However, this field is also left empty in the configuration.</td></tr></tbody></table>


# Lora Configuration

<table data-full-width="false"><thead><tr><th width="213">Field Name</th><th>Explanation</th></tr></thead><tbody><tr><td>adapter</td><td>The <code>adapter</code> field specifies whether to use 'lora' for fine-tuning or leave it blank to train all parameters in the original model. Using 'lora' allows for more fine-grained control over certain model parameters during training.</td></tr><tr><td>lora_model_dir</td><td>If you already have a trained LoRA model that you want to load, you can specify the directory path here. This is useful for testing the model after training. Ensure this value matches the path to the saved LoRA model.</td></tr><tr><td>lora_r</td><td><code>lora_r</code> is a hyperparameter that determines the number of repetitions or hops for the LoRA mechanism. It controls how many times the adapter parameters are applied to the input sequence. A higher value can capture longer dependencies but may require more resources.</td></tr><tr><td>lora_alpha</td><td><code>lora_alpha</code> is another hyperparameter that controls the strength of the LoRA mechanism. It determines how much influence the adapter parameters have on the original model's output. Higher values make the adapter more influential.</td></tr><tr><td>lora_dropout</td><td><code>lora_dropout</code> represents the dropout rate applied to the adapter's internal layers. Dropout is a regularization technique that helps prevent overfitting during training by randomly dropping some neurons.</td></tr><tr><td>lora_target_modules</td><td><code>lora_target_modules</code> is a list of module names within the transformer architecture that the LoRA adapter parameters should be applied to. In this example, it targets 'q_proj' and 'v_proj' modules. You can uncomment and add more modules as needed.</td></tr><tr><td>lora_target_linear</td><td>If set to 'true,' the LoRA adapter will target all linear layers within the specified <code>lora_target_modules</code>. This option can be used for a more comprehensive adaptation of linear layers.</td></tr><tr><td>lora_modules_to_save</td><td><code>lora_modules_to_save</code> specifies which LoRA modules should be saved. It's essential when adding new tokens to the tokenizer because certain LoRA modules need to be aware of these new tokens. In this example, the modules are commented out, but you should specify the modules to save as needed.</td></tr><tr><td>lora_out_dir</td><td>After completing training, the model will be saved to the directory specified by <code>lora_out_dir</code>. If you merge the adapter to the base model, a subdirectory named 'merged' will be created under this directory. Ensure that <code>lora_model_dir</code> points to this directory if you intend to use the trained model.</td></tr><tr><td>lora_fan_in_fan_out</td><td>When set to 'false,' this option disables the use of fan-in/fan-out scaling in the LoRA mechanism. Fan-in/fan-out scaling is a technique that helps stabilize training by scaling the adapter's parameters based on the input and output dimensions of the targeted modules.</td></tr><tr><td>relora_steps</td><td><code>relora_steps</code> specifies the number of steps per ReLoRA (Restartable LoRA) restart. ReLoRA is a technique used to restart the LoRA mechanism during training to improve convergence. This field sets the frequency of restarts.</td></tr><tr><td>relora_warmup_steps</td><td><code>relora_warmup_steps</code> sets the number of warm-up steps for each ReLoRA restart. Warm-up steps allow the LoRA mechanism to adapt gradually during a restart, which can improve training stability.</td></tr><tr><td>relora_cpu_offload</td><td>When set to 'true,' this option performs LoRA weight merges on the CPU during restarts, potentially saving GPU memory. It's useful for situations where GPU memory is a limiting factor during training.</td></tr></tbody></table>


# Logging

Weights and Biases

These are the configurations for tracking our fine tuning experiments.

<table data-header-hidden><thead><tr><th width="233"></th><th></th></tr></thead><tbody><tr><td><mark style="color:blue;"><strong>Class</strong></mark></td><td><mark style="color:blue;"><strong>Description</strong></mark> </td></tr><tr><td><strong>wandb_project</strong></td><td>Specifies the name of the project in Weights &#x26; Biases (WandB) for experiment tracking. </td></tr><tr><td><strong>wandb_entity</strong></td><td>Specifies the entity (user or organisation) in WandB under which the project will be created. </td></tr><tr><td><strong>wandb_watch</strong></td><td>A flag indicating whether to watch and log the model's progress in WandB. If set to "true," it means the experiment's progress and model performance will be actively monitored and logged in WandB.</td></tr><tr><td><strong>wandb_run_id</strong></td><td>Specifies a unique ID for the WandB run associated with this experiment. </td></tr><tr><td><strong>wandb_log_model</strong></td><td>A flag indicating whether to log the model to WandB. If set to "true," it suggests that the trained model's weights and configurations will be logged to WandB as part of the experiment's output.</td></tr></tbody></table>


# Training Configuration

<table data-full-width="false"><thead><tr><th width="270">Field Name</th><th>Explanation</th></tr></thead><tbody><tr><td>gradient_accumulation_steps</td><td><code>gradient_accumulation_steps</code> controls the number of forward and backward passes to skip before updating the model's weights. It is useful when the batch size is too large to fit into GPU memory at once. Gradients are accumulated over the specified number of steps before performing weight updates.</td></tr><tr><td>micro_batch_size</td><td><code>micro_batch_size</code> specifies the number of samples included in each batch sent to each GPU. It helps in determining the mini-batch size for training.</td></tr><tr><td>eval_batch_size</td><td><code>eval_batch_size</code> sets the batch size for evaluation. It determines how many samples are processed in each evaluation step.</td></tr><tr><td>num_epochs</td><td><code>num_epochs</code> defines the number of training epochs, which represent the number of times the entire dataset is processed during training.</td></tr><tr><td>warmup_steps</td><td><code>warmup_steps</code> specifies the number of warm-up steps for the learning rate scheduler. During the warm-up phase, the learning rate gradually increases to its full value.</td></tr><tr><td>learning_rate</td><td><code>learning_rate</code> sets the initial learning rate for training. It is a critical hyperparameter that determines the step size for weight updates during optimization.</td></tr><tr><td>lr_quadratic_warmup</td><td><code>lr_quadratic_warmup</code> is a field related to learning rate scheduling. It is used to specify a quadratic warm-up schedule for the learning rate, which can be beneficial for certain training scenarios.</td></tr><tr><td>logging_steps</td><td><code>logging_steps</code> sets the frequency at which training logs are generated. It controls how often training progress is reported.</td></tr><tr><td>save_strategy</td><td><code>save_strategy</code> determines when model checkpoints are saved during training. Setting it to 'no' skips checkpoint saves, while other options control the timing of saves.</td></tr><tr><td>save_steps</td><td><code>save_steps</code> specifies the frequency at which model checkpoints are saved. You can leave it empty to save at each epoch or specify a different number of steps.</td></tr><tr><td>eval_steps</td><td><code>eval_steps</code> controls the frequency of model evaluation during training. It can be specified as an integer for every N steps or as a decimal for a fraction of total steps.</td></tr><tr><td>save_total_limit</td><td><code>save_total_limit</code> limits the maximum number of checkpoints saved at a time. Older checkpoints are deleted to keep the total number within this limit.</td></tr><tr><td>max_steps</td><td><code>max_steps</code> defines the maximum number of iterations to train for. It takes precedence over <code>num_epochs</code>. For example, if you set <code>max_steps</code> to 100, the training will stop after 100 steps, regardless of the number of epochs.</td></tr><tr><td>eval_table_size</td><td><code>eval_table_size</code> specifies the approximate number of predictions sent to wandb (Weights and Biases) depending on the batch size. This field is enabled above 0 and is useful for tracking evaluation metrics.</td></tr><tr><td>eval_table_max_new_tokens</td><td><code>eval_table_max_new_tokens</code> sets the total number of tokens generated for predictions sent to wandb. It helps control the amount of data sent for monitoring.</td></tr><tr><td>save_safetensors</td><td>When specified, <code>save_safetensors</code> indicates saving the model as safetensors, requiring the safetensors package for compatibility.</td></tr><tr><td>train_on_inputs</td><td><code>train_on_inputs</code> determines whether to mask out or include the human's prompt from the training labels. Setting it to 'false' omits the prompt from training.</td></tr><tr><td>group_by_length</td><td>When set to 'true,' <code>group_by_length</code> groups data with similar sequence lengths together to minimize padding. This can help improve training efficiency but may lead to an oscillating training loss.</td></tr><tr><td>gradient_checkpointing</td><td><code>gradient_checkpointing</code> controls whether to use gradient checkpointing, a technique that can reduce memory consumption during training. When enabled, it trades off computation for memory.</td></tr><tr><td>early_stopping_patience</td><td><code>early_stopping_patience</code> determines when to stop training if evaluation losses increase consecutively for a specified number of times. It helps prevent overfitting.</td></tr><tr><td>lr_scheduler</td><td><code>lr_scheduler</code> specifies the learning rate scheduler to use during training. Options include 'one_cycle,' 'log_sweep,' or leaving it empty for cosine scheduling.</td></tr><tr><td>lr_scheduler_kwargs</td><td><code>lr_scheduler_kwargs</code> can be used to provide additional arguments to the learning rate scheduler, depending on the chosen scheduler type.</td></tr><tr><td>lr_div_factor</td><td>For 'one_cycle' optimizer, <code>lr_div_factor</code> determines the learning rate division factor during the one-cycle learning rate schedule.</td></tr><tr><td>log_sweep_min_lr</td><td>For 'log_sweep' optimizer, <code>log_sweep_min_lr</code> sets the minimum learning rate for the logarithmic learning rate sweep.</td></tr><tr><td>log_sweep_max_lr</td><td>For 'log_sweep' optimizer, <code>log_sweep_max_lr</code> sets the maximum learning rate for the logarithmic learning rate sweep.</td></tr><tr><td>optimizer</td><td><code>optimizer</code> specifies the optimizer to use for training. There are various optimizer options available, and the choice depends on the model and use case.</td></tr><tr><td>weight_decay</td><td><code>weight_decay</code> determines the weight decay applied during optimization. It is a regularization term that prevents overfitting by penalizing large model weights.</td></tr><tr><td>adam_beta1</td><td>For 'adamw' optimizer, <code>adam_beta1</code> sets the beta1 hyperparameter for the Adam optimizer. It controls the exponential moving average of past gradients.</td></tr><tr><td>adam_beta2</td><td>For 'adamw' optimizer, <code>adam_beta2</code> sets the beta2 hyperparameter for the Adam optimizer. It controls the exponential moving average of past squared gradients.</td></tr><tr><td>adam_epsilon</td><td>For 'adamw' optimizer, <code>adam_epsilon</code> sets the epsilon value added to the denominator to prevent division by zero.</td></tr><tr><td>max_grad_norm</td><td><code>max_grad_norm</code> specifies the maximum gradient norm value. Gradients are clipped to this value during training to prevent exploding gradients.</td></tr></tbody></table>


# Augmentation Techniques

<table data-full-width="false"><thead><tr><th width="248">Field Name</th><th>Explanation</th></tr></thead><tbody><tr><td>noisy_embedding_alpha</td><td><code>noisy_embedding_alpha</code> is used for applying noise to embeddings as part of data augmentation. It is based on the NEFT (Noisy Embedding Fine-Tuning) technique and can be set to a number (e.g., 5) to add noise to embeddings. This technique helps introduce variability into the training data, potentially improving robustness and generalization.</td></tr><tr><td>flash_optimum</td><td><code>flash_optimum</code> determines whether to use the "Optimum Layer-Order for Transformers" technique provided by Better Transformers. It's an advanced technique that optimizes the order of layers in the transformer model for improved performance.</td></tr><tr><td>xformers_attention</td><td><code>xformers_attention</code> specifies whether to use the attention patch from the XFormers library. XFormers is a library that provides optimized implementations of transformer components, including attention mechanisms.</td></tr><tr><td>flash_attention</td><td><code>flash_attention</code> controls whether to use the Flash Attention patch from the Flash Attention library. Flash Attention is another library that offers optimized attention mechanisms for transformers.</td></tr><tr><td>flash_attn_cross_entropy</td><td><code>flash_attn_cross_entropy</code> determines whether to use the Flash-Attention Cross Entropy implementation. This is an advanced option and should be used with caution, as it may require specific use cases.</td></tr><tr><td>flash_attn_rms_norm</td><td><code>flash_attn_rms_norm</code> specifies whether to use the Flash-Attention Root Mean Square (RMS) Norm implementation. RMS Norm is a technique for normalizing model activations.</td></tr><tr><td>flash_attn_fuse_qkv</td><td><code>flash_attn_fuse_qkv</code> controls whether to fuse the Query, Key, and Value (QKV) components of the attention mechanism into a single operation. This can potentially improve efficiency during training.</td></tr><tr><td>flash_attn_fuse_mlp</td><td><code>flash_attn_fuse_mlp</code> determines whether to fuse part of the Multi-Layer Perceptron (MLP) components of the attention mechanism into a single operation. Like the previous option, this aims to enhance efficiency.</td></tr><tr><td>sdp_attention</td><td><code>sdp_attention</code> specifies whether to use the Scaled Dot-Product Attention mechanism, which is a fundamental component of transformer models. The link provided points to the PyTorch documentation for this attention mechanism.</td></tr><tr><td>landmark_attention</td><td><code>landmark_attention</code> is used only with LLaMA and controls whether to use landmark attention. Landmark attention is a specialized attention mechanism designed for specific use cases.</td></tr><tr><td>xpos_rope</td><td><code>xpos_rope</code> is related to the RoPE (Relative Positional Encoding) technique and is specific to LLaMA. It appears to be related to modifying RoPE for positional encoding in the LLaMA model. The provided link points to an external resource for more details.</td></tr></tbody></table>


# Axolotl Fine-Tuning Tips & Tricks: A Comprehensive Guide

tips and tricks from the community

### <mark style="color:blue;">Dataset Preparation and Management</mark>

* Use `python -m axolotl.cli.preprocess your_config.yml --debug` for preprocessing data and debugging the output. This ensures data is in the correct format for training or fine-tuning.
* Utilize `dataset_shard_num` and `dataset_shard_idx` parameters to work with dataset fractions, especially for large datasets or limited computational resources.
* Reduce dataset size when working with extremely large datasets to manage training duration effectively while maintaining or improving quality.
* Be aware of the datasets library caching intermediate transforms during preparation, as it can affect disk space usage and training efficiency.
* Define custom dataset formats for unique data requirements to allow for more tailored data handling.
* Handle duplicate data entries appropriately to prevent data contamination or model confusion.
* Be cautious with datasets containing anomalies like duplicates or conflicting entries, as they can introduce noise into training. Design preprocessing steps to handle or remove such anomalies effectively.

### <mark style="color:blue;">Hardware and Environment Setup</mark>

* Set up multi-GPU support for efficient training, potentially using DeepSpeed for distributed training.
* Ensure compatibility between precision settings and available libraries or dependencies.
* Be mindful of how different libraries and tools interact, and consult relevant documentation for guidance.
* Ensure the correct CUDA toolkit version is installed and matches library and framework requirements.
* Optimize hardware use by selecting the correct GPU architecture and ensuring dependencies like CUDA are up to date. Use optimized tools like bitsandbytes-rocm for specific hardware.

### <mark style="color:blue;">Model Configuration and Training</mark>

* Create custom prompt fields for tailoring model responses to specific needs and context-relevant interactions.
* Structure data correctly using appropriate separator styles, custom tokens, and formats like JSON for efficient training.
* Integrate DeepSpeed, using versions like zero2 and zero3, to enhance training efficiency, especially in multi-GPU setups.
* Adjust evaluation dataset size or configuration to avoid issues like 'eval\_loss' returning NaN.
* Understand and correctly set configuration file parameters like `load_in_8bit`, `load_in_4bit`, and `strict` for specific use cases.
* Tweak adapter settings, such as changing to `adapter: lora`, for potentially significant impact on model performance or compatibility.
* Configure accelerator settings correctly for specific hardware and model requirements, using tools like the `accelerate` library.
* Manage sequence length and sample packing for models like Llama or Mixtral to impact performance and training efficiency. Larger sequence lengths require more memory but can improve contextual understanding, while sample packing can increase throughput at the cost of complexity.
* Experiment with learning rates and schedulers to help models quickly converge and refine predictions.
* Use techniques like gradient checkpointing, mixed precision training, and optimizers like AdamW with reduced precision to manage VRAM usage and balance training speed.
* Explore advanced optimization techniques like quantization and pruning to reduce model size and speed up inference without significantly impacting performance.
* Leverage DeepSpeed's ZeRO stages and ensure configuration is optimized for your hardware setup to reduce memory requirements and maximize GPU utilization.
* For large models, consider using Fully Sharded Data Parallel (FSDP) to distribute model parameters across multiple GPUs, reducing memory requirements per GPU.

### <mark style="color:blue;">Tokenizer and Special Tokens Configuration</mark>

* Specify tokenizer configuration in the YAML file to ensure data preprocessing aligns with model expectations, especially with custom features like ChatML.
* Ensure tokenizer configuration accurately reflects the tokens your model expects, especially with custom or complex models, to avoid unexpected behavior or inefficient training.
* Maintain consistency in special token configuration (e.g., BOS/EOS tokens) across data preprocessing, model training, and inference to avoid discrepancies.

### <mark style="color:blue;">Debugging and Problem Solving</mark>

* Use `python -m axolotl.cli.preprocess your_config.yml --debug` for preprocessing data and debugging the output.
* Monitor training progress and visualize results using tools like Weights & Biases (W\&B) to understand model performance over time.
* Double-check version compatibility between training scripts, the Axolotl library, and model architecture when encountering errors related to unexpected arguments or missing keys in state dictionaries.
* Use gradient checkpointing to manage memory usage effectively, especially when training very large models on limited hardware.
* Ensure training setup correctly handles resuming from checkpoints, especially with tools like DeepSpeed, to avoid lost progress or errors.
* Be aware of specific hardware requirements for features like Flash Attention, as incompatible hardware can lead to runtime errors.
* Maintain consistency in tokenization between training and inference stages to avoid poor model performance and unexpected behavior.
* Double-check tokenizer settings and input configurations when encountering errors related to tokenization or input processing.
* Use debugging tools like Python's pdb to step through code and find elusive bugs.
* Investigate memory usage, data loading issues, or potential deadlocks in the training loop if training gets stuck, especially after saving checkpoints or at specific steps.
* Ensure model resumption from checkpoints is configured correctly to avoid training starting anew or errors due to missing or mismatched state dictionaries.

### <mark style="color:blue;">Logging and Monitoring</mark>

* Set a fixed seed in the training configuration to ensure deterministic sample packing, stabilizing training and making loss curves more predictable.
* Use detailed logging and integrate tools like Weights & Biases (WandB) to track training progress and diagnose issues, particularly with complex models or large-scale datasets.
* Integrate training with monitoring tools like TensorBoard for real-time tracking of metrics to quickly identify overfitting, underfitting, or performance deviations.
* Set up detailed logging, especially for long training runs on large datasets, as logs provide invaluable insights when troubleshooting or replicating experiments.

### <mark style="color:blue;">Continuous Learning and Adaptation</mark>

* Keep training scripts, model definitions, and dependencies up-to-date with the latest practices and libraries to avoid compatibility issues and leverage improvements in architectures, algorithms, and hardware support.
* Regularly update libraries and stay informed about new features and best practices in machine learning and tools like Axolotl, as they are constantly evolving.

By applying these tips and tricks, you can enhance your model training process using Axolotl, address common pitfalls, and optimize your models for performance and accuracy effectively. Remember to stay updated with the latest developments in the field and adapt your approach based on the specific needs of your project.


# Axolotl debugging guide

The guide provides tips and tricks for debugging Axolotl, a tool for fine-tuning large language models. It covers general debugging tips as well as how to set up debugging in VSCode, both locally and remotely via Docker containers.

### <mark style="color:blue;">General debugging tips</mark>

* Use the latest version of Axolotl from the main branch
* Eliminate concurrency by restricting to a single GPU and dataset process
* Use a small dataset and model to speed up iteration time
* Minimize batch size and disable validation
* Clear data preprocessing and HuggingFace caches between runs

### <mark style="color:blue;">Debugging in VSCode</mark>

* Do an editable install of Axolotl
* Modify launch.json to:
  * Launch the train command with additional debugging flags
  * Set the working directory to devtools
  * Limit to a single GPU
  * Redirect outputs and caches to temp folders
* Use a pre-launch task in tasks.json to clean up temp folders between runs
* Remote debugging is supported via SSH
* The launch config can be customized for your specific use case

### <mark style="color:blue;">Debugging in Docker</mark>

* Clone Axolotl repo on the host running Docker
* Run the desired Axolotl Docker image, mounting the Axolotl directory
* Do an editable install inside the container
* If using a remote host, SSH in with VSCode
* Attach VSCode to the running Docker container
* Debug inside the container as you would locally

By following these tips and configuration examples, you can efficiently debug Axolotl to better understand how the code works behind the scenes. The ability to quickly iterate with a simple test setup is key to productive debugging.


# Hugging Face Hub API

The Hugging Face Hub API, accessed through the <mark style="color:yellow;">**`huggingface_hub`**</mark> package in Python, provides a comprehensive set of methods and classes to interact with the Hugging Face Hub platform.

It allows users to access and manage repositories, models, datasets, and other resources available on the Hub. Here's a detailed summary of the capabilities of the Hugging Face Hub API:

### <mark style="color:green;">Repository Management</mark>

* Create a new repository on the Hub using `create_repo()`.
* Delete a repository from the Hub using `delete_repo()`.
* Update the visibility (public/private) of a repository using `update_repo_visibility()`.
* Move a repository from one namespace to another using `move_repo()`.

### <mark style="color:green;">Model and Dataset Information</mark>

* Get information about a specific model using `model_info()`.
* Get information about a specific dataset using `dataset_info()`.
* Get information about a specific space using `space_info()`.
* Get generic repository information using `repo_info()`.

### <mark style="color:green;">Listing Resources</mark>

* List all available models on the Hub using `list_models()`.
* List all available datasets on the Hub using `list_datasets()`.
* List all available metrics on the Hub using `list_metrics()`.
* Filter models and datasets based on various criteria using `ModelFilter` and `DatasetFilter` classes.

### <mark style="color:green;">File Management</mark>

* Upload a local file to a repository using `upload_file()`.
* Delete a file from a repository using `delete_file()`.
* List all files in a repository using `list_repo_files()`.

### <mark style="color:green;">Authentication</mark>

* Set the access token for authentication using `set_access_token()`.
* Unset the access token using `unset_access_token()`.
* Get information about the authenticated user using `whoami()`.

### <mark style="color:green;">Local Storage</mark>

* The `HfFolder` class provides methods to manage local storage of authentication tokens.
* Save a token using `save_token()`, get a token using `get_token()`, and delete a token using `delete_token()`.

### <mark style="color:green;">Filtering Helpers</mark>

* The `ModelFilter` and `DatasetFilter` classes provide an easy way to construct filters for searching models and datasets based on various criteria such as author, task, language, tags, etc.
* The `ModelSearchArguments` and `DatasetSearchArguments` classes provide tab-completion and access to all possible values for properties of models and datasets hosted on the Hub.

### <mark style="color:green;">Technical Details</mark>

* The API communicates with the Hugging Face Hub platform using HTTP requests.
* Authentication is performed using access tokens, which can be obtained from the Hugging Face website and set using the <mark style="color:yellow;">**`set_access_token()`**</mark> method.
* The API uses JSON format for data exchange.
* The <mark style="color:yellow;">**`HfApi`**</mark> class serves as the main entry point for interacting with the API, and most methods are accessible directly from the <mark style="color:yellow;">**`huggingface_hub`**</mark> package.
* The API supports various parameters for filtering, sorting, and pagination when listing resources.
* Error handling is provided through specific exception classes such as <mark style="color:yellow;">**`RepositoryNotFoundError`**</mark> and <mark style="color:yellow;">**`RevisionNotFoundError`**</mark>.
* The API allows specifying the revision (e.g., branch or tag) when accessing repository-related information.
* The <mark style="color:yellow;">**`upload_file()`**</mark> method supports uploading files up to 5GB in size.

The Hugging Face Hub API provides a powerful and flexible way to interact with the Hugging Face Hub platform programmatically.&#x20;

It enables users to manage repositories, access model and dataset information, upload and delete files, and perform various search and filtering operations.&#x20;


# NCCL

Troubleshooting NCCL issues from the Axolotl documentation

[<mark style="color:blue;">NVIDIA Collective Communications Library</mark>](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/overview.html) (NCCL, pronounced “Nickel”)  is a library to facilitate and optimise multi-GPU communication operations, such as broadcast, all-gather, reduce, all-reduce.

Broadly, NCCL configuration is highly environment-specific and is configured via several [environment variables](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html).&#x20;

<details>

<summary><mark style="color:green;">NCCL Explanation</mark></summary>

NCCL, which stands for NVIDIA Collective Communications Library, is a library developed by NVIDIA to <mark style="color:yellow;">facilitate efficient communication and synchronization between multiple GPUs</mark>

It plays a crucial role in enabling fast and scalable training of deep neural networks across multiple GPUs and nodes.

One of the key features of NCCL is its <mark style="color:yellow;">topology-aware design</mark>.&#x20;

It takes into account the underlying hardware architecture and interconnects, such as PCIe, NVLink, InfiniBand, and Ethernet, to optimise communication paths and minimise latency.&#x20;

This is particularly important in deep learning *<mark style="color:yellow;">**workloads where large amounts of data need to be exchanged between GPUs during training.**</mark>*

NCCL provides a set of fundamental collective communication primitives that are commonly used in deep learning frameworks. These include:

1. <mark style="color:blue;">AllReduce:</mark> This operation combines the input tensors from all GPUs, performs a specified reduction operation (e.g., sum, max, min), and distributes the result back to all GPUs. It is extensively used in distributed training for aggregating gradients and updating model parameters.
2. <mark style="color:blue;">Broadcast:</mark> This operation broadcasts the input tensor from a single source GPU to all other GPUs. It is useful for synchronizing model parameters across GPUs.
3. <mark style="color:blue;">Reduce:</mark> This operation performs a reduction operation on the input tensors from all GPUs and stores the result on a specified root GPU.
4. <mark style="color:blue;">AllGather:</mark> This operation gathers the input tensors from all GPUs and concatenates them along a specified dimension, making the result available on all GPUs.
5. <mark style="color:blue;">ReduceScatter:</mark> This operation performs a reduction operation on the input tensors from all GPUs and scatters the result across all GPUs, such that each GPU receives a portion of the reduced tensor.

NCCL also supports point-to-point communication primitives, such as send and receive, which enable more flexible communication patterns like scatter, gather, and all-to-all operations.

One of the key advantages of NCCL is its <mark style="color:yellow;">ability to perform collective operations in a single kernel,</mark> combining communication and computation. This tight synchronization minimises the overhead of launching multiple kernels and memory copies, allowing NCCL to achieve near-peak bandwidth utilization.

From a programming perspective, NCCL provides a simple and intuitive API that closely follows the popular <mark style="color:blue;">Message Passing Interface (MPI)</mark> standard. This makes it easy for developers familiar with MPI to adopt NCCL in their deep learning frameworks. NCCL seamlessly integrates with the CUDA programming model, allowing collectives to be launched directly from CUDA streams.

The impact of NCCL on the deep learning community has been significant. It has become a fundamental building block for many popular deep learning frameworks, such as TensorFlow, PyTorch, and MXNet. These frameworks leverage NCCL to efficiently scale the training of large neural networks across multiple GPUs and nodes, enabling faster convergence and handling larger datasets.

NCCL's performance optimisations and topology-aware design have greatly accelerated the training of deep neural networks, allowing researchers and practitioners to train more complex models in shorter timeframes. It has become an essential tool for anyone working on large-scale deep learning projects.

</details>

A common NCCL-related problem occurs when a long-running operation times out causing the training process to abort:

{% code overflow="wrap" %}

```bash
Watchdog caught collective operation timeout: WorkNCCL(SeqNum=42, OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1806948 milliseconds before timing out.
```

{% endcode %}

Often, this timeout will happen after 30 minutes (the default setting) and is accompanied by below-average power consumption with near 100% GPU utilization before the error is raised.&#x20;

Nvidia recommends [disabling PCI access control services (ACS)](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs) as a possible solution if this is available to you.

Forcing cross-GPU communication via [NVLink](https://en.wikipedia.org/wiki/NVLink) may help without increasing timeouts.&#x20;

To verify that your configuration is leveraging NVLink run the following command:

```bash
nvidia-smi nvlink --status
```

To force NCCL to use NVLink, simply set this in the environment:

```bash
export NCCL_P2P_LEVEL=NVL
```

If NVLink is not available in your environment there are other options for <mark style="color:yellow;">`NCCL_P2P_LEVEL`</mark> in the table below:

<table><thead><tr><th width="251">NCCL_P2P_LEVEL</th><th>Description</th></tr></thead><tbody><tr><td>PIX</td><td>P2P data transfers through no more than a single PCIe bridge. Faster data transfer rates vs to paths involving multiple bridges, but slower compared to direct GPU-to-GPU communication.</td></tr><tr><td>PXB</td><td>P2P data transfers through multiple PCIe bridges but not going through the PCIe Host Bridge; this path involves a complex routing process, potentially incurring a moderate level of latency.</td></tr><tr><td>PHB</td><td>P2P data transfers occur over the PCIe and through a PCIe Host Bridge, typically involving the CPU, which can facilitate direct memory access but might introduce additional latency compared to more direct paths (ex PIX, NVL)</td></tr></tbody></table>

To validate that acceptable data transfer speeds exist for your training job, running [NCCL Tests](https://github.com/NVIDIA/nccl-tests/blob/master/README.md) can help pinpoint bottlenecks, for example:

```bash
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 3
```

It can be useful when debugging NCCL communication timeouts to activate additional logging in both PyTorch and NCCL:

```bash
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
export TORCH_DISTRIBUTED_DEBUG=INFO
export TORCHELASTIC_ERROR_FILE=/PATH/TO/torcherror.log
```

Finally, if you believe your training job needs more time you can increase the timeout past 30 minutes by setting the <mark style="color:yellow;">`ddp_timeout`</mark> value in the Axolotl configuration. See [PyTorch init\_process\_group](https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group) for documentation on this value.


# Training Phi 1.5 - Youtube

Great Youtube videos

## From hu-po

{% embed url="<https://www.youtube.com/watch?v=yxAcTRp9EyQ>" %}

## Fine Tuning Phi 1\_5 with PEFT and QLoRA - Large Language Model with PyTorch

{% embed url="<https://www.youtube.com/watch?v=J0RbOtLrJhQ>" %}

## Textbooks Are All You Need

{% embed url="<https://www.youtube.com/watch?v=24O1KcIO3FM>" %}


# JSON (JavaScript Object Notation)

JSON is a lightweight, human-readable, and widely used data interchange format.

It *<mark style="color:yellow;">**represents data as key-value pairs,**</mark>* making it suitable for <mark style="color:yellow;">storing structured or semi-structured data.</mark>&#x20;

JSON is language-independent and supported by various programming languages.

·         Storing and exchanging data between a server and a client in web applications.

·         Configuration files for applications or services.

·         Storing data with hierarchical relationships or complex structures.

JSON is useful for training LLMs when you need to store and process structured or semi-structured textual data with additional metadata or attributes.&#x20;

For instance, if you are working with a dataset containing articles with author, title, date, and content information, JSON can efficiently store and represent this data.

JSON and JSON Lines (JSONL) are both formats used for storing and exchanging data. However, they have distinct structures and use cases.

### <mark style="color:blue;">JSON (JavaScript Object Notation)</mark>

* <mark style="color:blue;">**Structure:**</mark> JSON is a lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language. JSON represents data as a single, cohesive entity.
* <mark style="color:blue;">**Usage:**</mark> Typically used to represent complex data structures containing multiple nested objects and arrays. Ideal for configurations, RESTful API responses, and data interchange between server and web applications.

<mark style="color:green;">**Format Example**</mark>

```json
[
    {
        "name": "John",
        "age": 30
    },
    {
        "name": "Jane",
        "age": 25
    }
]
```

* <mark style="color:blue;">**File Read/Write:**</mark> The entire JSON file must be read or written in one operation, which can be resource-intensive for large datasets.

### <mark style="color:blue;">JSON Lines (JSONL)</mark>

* <mark style="color:blue;">**Structure:**</mark> JSON Lines is a <mark style="color:yellow;">convenient format for storing structured data that may be processed one record at a time.</mark> It handles multiple, discrete JSON objects, with each object separated by a newline character.
* <mark style="color:blue;">**Usage:**</mark> Well-suited for log files and data streaming scenarios where each line can be processed independently. Ideal for large datasets and Unix-style text processing tools.

<mark style="color:green;">**Format Example**</mark>

```json
{"name": "John", "age": 30}
{"name": "Jane", "age": 25}
```

* <mark style="color:blue;">**File Read/Write:**</mark> Allows for processing data one line at a time, which is more efficient for large datasets. Each line is a valid JSON object, allowing for incremental reading/writing.

### <mark style="color:blue;">Key Differences</mark>

1. <mark style="color:blue;">**Data Representation:**</mark> JSON represents a single cohesive data structure, while JSONL represents multiple, independent JSON objects separated by newlines.
2. <mark style="color:blue;">**Read/Write Efficiency:**</mark> JSONL is more efficient for large datasets as it allows for processing one record at a time, whereas JSON requires handling the entire data structure at once.
3. <mark style="color:blue;">**Use Cases:**</mark> JSON is ideal for configurations and API responses, while JSONL is better for logging and streaming large datasets.

### <mark style="color:blue;">Commonalities</mark>

* Both formats use UTF-8 encoding.
* Both are based on the JSON standard and can be easily parsed using standard JSON parsers, with slight adaptations for JSONL.

JSONL is particularly advantageous when dealing with large datasets that need to be streamed or processed incrementally, avoiding the memory overhead of loading the entire dataset as required in standard JSON.


# General Tips

### <mark style="color:blue;">General Tips</mark>

While debugging it’s helpful to simplify your test scenario as much as possible. Here are some tips for doing so:

<mark style="color:yellow;">**Make sure you are using the latest version of axolotl**</mark>

This project changes often and bugs get fixed fast. Check your git branch and make sure you have pulled the latest changes from <mark style="color:yellow;">`main`</mark>.

### <mark style="color:green;">**Eliminate concurrency**</mark>

Restrict the number of processes to 1 for both training and data pre-processing:

* Set `CUDA_VISIBLE_DEVICES` to a single GPU, ex: `export CUDA_VISIBLE_DEVICES=0`.
* Set `dataset_processes: 1` in your axolotl config or run the training command with `--dataset_processes=1`.

### <mark style="color:green;">**Use a small dataset**</mark>

Construct or use a small dataset from HF Hub.&#x20;

When using a small dataset, you will often have to make sure <mark style="color:yellow;">`sample_packing: False`</mark> and <mark style="color:yellow;">`eval_sample_packing: False`</mark> to avoid errors.&#x20;

If you are in a pinch and don’t have time to construct a small dataset but want to use from the Huggingface Hub, you can <mark style="color:yellow;">shard the data</mark> (this will still tokenize the entire dataset, but will only use a fraction of the data for training.&#x20;

For example, to shard the dataset into 20 pieces, add the following to your axolotl config):&#x20;

<mark style="color:yellow;">`yaml dataset: ... shards: 20`</mark>

### <mark style="color:green;">**Use a small model**</mark>

A good example of a small model is [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0).

### <mark style="color:green;">**Minimize iteration time**</mark>

Make sure the training loop finishes as fast as possible, with these settings.

* <mark style="color:yellow;">`micro_batch_size: 1`</mark>
* <mark style="color:yellow;">`max_steps: 1`</mark>
* <mark style="color:yellow;">val\_set\_size</mark><mark style="color:yellow;">`: 0`</mark>

### <mark style="color:green;">**Clear Caches**</mark>

Axolotl caches certain steps and so does the underlying HuggingFace trainer.&#x20;

You may want to clear some of these caches when debugging.

* <mark style="color:blue;">Data pre-processing:</mark> When debugging data pre-processing, which includes prompt template formation, you may want to delete the directory set in <mark style="color:yellow;">`dataset_prepared_path:`</mark> in your axolotl config. If you didn’t set this value, the default is <mark style="color:yellow;">`last_run_prepared`</mark><mark style="color:yellow;">.</mark>
* Huggingface Hub: If you are debugging data pre-processing, you should clear the relevant HF cache [HuggingFace cache](https://huggingface.co/docs/datasets/cache), by deleting the appropriate <mark style="color:yellow;">`~/.cache/huggingface/datasets/...`</mark> folder(s).
* The recommended approach is to redirect all outputs and caches to a temporary folder and delete selected subfolders before each run. This is demonstrated in the example configuration below.


# Datasets

\
**Data Quality**

* **Relevance**: Ensure the data is relevant to the tasks your model will perform. For coding models, for instance, include various programming languages, frameworks, and problem-solving scenarios.
* **Accuracy**: Data should be accurately labeled or annotated. This is crucial in supervised learning scenarios where the model learns directly from the dataset labels.

#### **Data Diversity**

* **Variety**: Include a wide range of examples to cover different cases the model might encounter. For a coding model, this might mean different coding styles, libraries, and even code snippets with common errors to enhance error recognition capabilities.
* **Balance**: Avoid biases by balancing the dataset. For example, if training a multilingual model, ensure all languages are adequately represented.

#### **Data Volume**

* **Scalability**: More data generally leads to better model performance, but it’s crucial to balance quantity with quality. Ensure that the increase in data volume doesn’t dilute data quality.
* **Incremental Training**: Start with a smaller, high-quality dataset to observe performance improvements before scaling up. This approach helps in identifying which types of data are most beneficial.

#### **Data Formatting**

* **Consistency**: Keep data formats consistent, especially if you are aggregating data from multiple sources. Consistency in formatting helps in minimizing preprocessing errors.
* **Preprocessing**: Clean and preprocess data to remove unnecessary or misleading information. This might include stripping irrelevant metadata or converting all data to a uniform format.

#### **Use of Synthetic Data**

* **Augmentation**: In cases where data is scarce, consider using synthetic data generation techniques to augment your datasets. This can be particularly useful for edge cases or rare scenarios.
* **Validation**: Ensure that synthetic data is realistic and validated against real-world criteria to avoid introducing biases or unrealistic scenarios into the training process.

**Legal and Ethical Considerations**

* **Compliance**: Ensure that the dataset complies with data protection regulations (like GDPR in Europe) especially if it includes personally identifiable information.
* **Ethics**: Consider the ethical implications of your data collection and usage. Ensure the data does not reinforce stereotypes or biases.

#### **Tools and Frameworks**

* **Automation Tools**: Utilize tools for data scraping, cleaning, and annotation to streamline the data preparation process. Libraries like Beautiful Soup for web scraping or Pandas for data manipulation are very helpful.
* **Dataset Libraries**: Leverage existing datasets available on platforms like Hugging Face or Kaggle as a starting point or as augmentation to your proprietary data.

#### **Feedback Loop**

* **Continuous Improvement**: Use model performance feedback to continuously improve the dataset. Identify areas where the model underperforms and augment the dataset accordingly.

Data Structuring for Conversational Models

1. **Role Naming Flexibility**: As discussed in your conversation, the ability to flexibly define roles like "User" and "Assistant" instead of the standard "human" and "gpt" can be beneficial. This approach can align the data more closely with the specific conversational roles used in your application. Implementing custom fields such as `field_human` and `field_model` in your configuration can allow for this flexibility without needing significant code modifications.
2. **Using YAML for Custom Configurations**:
   * For integrating diverse data formats or custom naming conventions, leveraging YAML configurations can allow you to specify how each part of your dataset should be interpreted by the training framework. This can include renaming fields or defining specific preprocessing behaviors.
3. **Conversation Structure**:
   * Ensure that your data mimics the conversational flow you expect the model to handle post-training. This includes the correct sequencing of prompts and responses and embedding contextual cues when necessary.
4. **Inclusion of System Prompts**:
   * If your model needs to handle system-driven interactions (like instructions or prompts given by a system before a user input), including these in the training data can help the model learn appropriate responses based on the system's prompts.

#### Data Annotation and Format

1. **Annotation Guidelines**:
   * Maintain clear and consistent annotation guidelines to ensure that the data is uniformly annotated. This consistency is crucial when the data involves subjective judgments or when multiple annotators are involved.
2. **Format Consistency**:
   * Ensure all data shares a consistent format, especially if aggregated from multiple sources. This can involve standardizing the text encoding, the way dialogues are structured, or how metadata is attached.

#### Utilizing Metadata and Advanced Formatting

1. **Metadata Usage**:
   * Include relevant metadata in your training datasets to allow models to utilize contextual information which could improve their understanding and generation capabilities. This can include information like the timestamp of a conversation, the platform it was captured from, or the geographical location involved.
2. **Advanced Formatting Techniques**:
   * Consider using advanced text formatting techniques like embedding hidden context or control codes that can guide the model’s responses in subtle ways. This might involve coding certain behavioral traits or operational modes directly into the training data.

#### Validation and Testing

1. **Dataset Splitting**:
   * Ensure that your data is split appropriately into training, validation, and testing sets. This helps in tuning the model on realistic data scenarios and validating its performance on unseen data.
2. **Dynamic Dataset Updates**:
   * Implement mechanisms to update the dataset dynamically as new data becomes available or as the model’s application domain evolves. This can involve periodically retraining the model on updated datasets or incrementally training on new data.

\
**Semantic Consistency**

* Ensure the data remains semantically consistent after transformations or style changes. This is crucial when rewriting texts in different styles to maintain the original meaning.

#### 2. **Balanced Representation**

* Strive for a balanced representation of styles, topics, and lengths to avoid model bias towards any specific style or subject matter.

#### 3. **Iterative Refinement**

* Use iterative refinement of data samples based on model feedback. This could involve adjusting the style intensity or clarity based on how well models perform with initial datasets.

#### 4. **Contextual Richness**

* For RAG tasks, enrich the dataset with diverse contexts that can help the model learn to retrieve and integrate relevant information effectively. This can include adding background information, related facts, or even conflicting viewpoints to enhance learning depth.

#### 5. **Dynamic Sampling**

* Implement dynamic sampling techniques to expose the model to a variety of data samples during training sessions. This helps in improving generalization by preventing overfitting on specific styles or patterns.

#### 6. **Quality Control Mechanisms**

* Establish robust quality control mechanisms to regularly assess and ensure the quality of data being fed into the model. This includes checking for data corruption, mislabeling, and ensuring stylistic accuracy in case of style-based rewriting tasks.

#### 7. **Automated and Manual Annotations**

* Combine automated tools with manual review to annotate data, especially for complex tasks like style transfer where nuanced understanding of style and content is necessary.

#### 8. **Data Augmentation**

* Use data augmentation techniques judiciously to expand dataset size without compromising quality, such as paraphrasing, back-translation, or synthetic data generation.

#### 9. **Feedback Loops**

* Create feedback loops where initial model outputs are reviewed and corrections are fed back into the training regime to continually refine the model's understanding and generation capabilities.

#### 10. **Use of External Knowledge Bases**

* For RAG implementations, integrate external knowledge bases during training to simulate how the model should perform retrieval tasks during actual deployment. This can help the model learn to pull in relevant information when needed.

#### 11. **Preprocessing Pipelines**

* Develop robust preprocessing pipelines that standardize data before it enters the training workflow, ensuring consistency in how data is handled and reducing the chance of errors during model training.

#### 12. **Experimentation with Tokenization**

* Experiment with different tokenization strategies to understand their impact on the model’s ability to understand and generate text accurately, especially for different languages or specialized vocabularies.

. **Structured JSONL Format**

* Utilize structured JSONL format for your datasets as it facilitates easier parsing and manipulation of the data. This format allows you to represent conversation turns explicitly, which can be directly utilized by models without additional preprocessing.

#### 2. **Role-based Data Entries**

* Define clear roles within your dataset entries, such as "user" and "assistant", to help the model learn context-specific responses. This is particularly useful in dialog systems where maintaining the role context is crucial.

#### 3. **Incorporating External Knowledge**

* Consider integrating external knowledge sources directly into your dataset to enhance the model's ability to perform knowledge-based tasks. This can be done through links to additional resources or embedding supplementary data within the dataset entries.

#### 4. **Multi-format Support**

* Prepare your datasets in multiple formats to test the flexibility of your model across different data handling frameworks. This includes variations in tokenization, annotation styles, and data structuring.

#### 5. **Dynamic Data Generation**

* Use models to generate dynamic training data based on existing patterns. This can involve creating variations of data samples through model-generated paraphrasing, summarization, or style transfer.

#### 6. **Simulating Real-World Scenarios**

* Structure your data to simulate real-world usage scenarios as closely as possible. This includes creating datasets that reflect the diversity of real-world applications, from casual conversations to technical support scenarios.

#### 8. **Version Control and Experiment Tracking**

* Maintain version control for your datasets and use experiment tracking tools to log how changes in data affect model performance. This helps in understanding the impact of specific data adjustments and optimizing the training process.

#### 9. **Utilizing Instruction-Based Tuning**

* Leverage instruction-based tuning where the dataset is enriched with explicit instructions that guide the model's output. This is particularly effective in scenarios where the model needs to perform specific tasks based on varied inputs.

#### 10. **Quality and Consistency Checks**

* Regularly perform quality checks and consistency assessments on your datasets to ensure high data quality and reliability. This can include automated scripts to detect anomalies and manual reviews to ensure contextual accuracy.

#### 11. **Adaptive Learning**

* Design datasets that enable adaptive learning, where data samples are selected and utilized based on the model’s current learning state. This helps in focusing training on areas where the model is underperforming.

#### 12. **Explorative Data Analysis**

* Before finalizing your dataset, conduct explorative data analysis to understand the data's characteristics fully. This can help in identifying potential biases, imbalances, or other issues that could impact training effectiveness.

<br>