Platform Installation

The next step after ensuring NVIDIA Drivers and Enviornment are stable and compatible

With the virtual machine established and optimised, we will now establish the virtual environment for Axolotl

Download the axolotl library

git clone https://github.com/OpenAccess-AI-Collective/axolotl.git

Then navigate into the library:

cd axolotl

Reference: What does git clone do?

The command git clone is a command-line instruction used with Git, a version control system, to create a local copy of a remote repository. Let's break down this command to understand each part:

1. git clone:

   • git: This is the command-line tool for using Git.

   • clone: A Git command used to copy a remote repository. It not only downloads the content of the repository but also creates a new local repository with all the history and branch information.

2. Repository URL

   • https://github.com/OpenAccess-AI-Collective/axolotl.git: This is the URL of the remote Git repository.

   • In this case, the repository is hosted on GitHub, a web-based platform for version control using Git.

   • The repository belongs to a user or organization named OpenAccess-AI-Collective, and the specific repository is named axolotl.

3. Executing the Command

   • When you execute this command in a terminal or command prompt, Git will:

     • Reach out to the URL provided.

     • Download the contents of the repository, including all the files, folders, branches, and history (commits).

     • Create a new directory in your current working directory with the same name as the repository (axolotl in this case).

     • Initialize a local Git repository in that directory and link it to the remote repository. This link is important for future commands like git pull (to fetch updates) and git push (to upload local changes).

4. Post-Cloning

   • After cloning, you can navigate into the newly created directory (cd axolotl) and start working with the files. You can also switch between branches, update the local repository with changes from the remote repository, and commit new changes.

5. Purpose

   • Cloning is often the first step when you want to work on a project from a remote repository, contribute to it, or just have a local copy for reference or backup.

In summary, git clone https://github.com/OpenAccess-AI-Collective/axolotl.git is a Git command to create a local copy of the axolotl repository from the OpenAccess-AI-Collective GitHub page. It sets up a new folder with all the repository data and establishes a connection between the local and remote repositories for ongoing version control.

Virtual Environment

Check for Anaconda:

• Ensure Anaconda is installed and create and Anaconda environment called 'axolotl'

• Type the following command into the terminal to check if Anaconda is installed and press Enter:

conda --version

If Anaconda is installed, this command will return the version number:

conda 23.10.0

• If you get an error or message saying that conda is not recognised, it means Anaconda is not installed.

If Anaconda is not installed - follow the installation instructions below "Installing Miniconda":

Installing Miniconda

Overview

• Miniconda is a minimal installer for the Conda package manager. It provides a lightweight alternative to Anaconda, including only Conda, Python, and essential packages they depend on, plus a few other useful packages such as pip and zlib.

• Ideal for users who prefer a smaller footprint or want more control over installed packages.

• Additional packages can be installed using conda install from Anaconda’s public repository or other channels like conda-forge or bioconda.

Choosing Miniconda

• The decision between Anaconda and Miniconda depends on user needs. The Anaconda or Miniconda page provides guidance for choosing the most suitable installation.

System Requirements

• System requirements are available on the Miniconda documentation page, ensuring compatibility with user systems.

Installation Links and Release Notes

• The latest Miniconda installer links are provided for Python 3.11.5 as of November 16, 2023.

• Installers are available for various platforms, including Windows, macOS (Intel and Apple M1), and Linux (with multiple architecture support).

• SHA256 hashes are provided for verifying the integrity of the downloaded files.

• For older Python versions or an archive of Miniconda versions, links are available on the documentation page.

Quick Installation (Command Line)

• Quick installation instructions are provided for a streamlined setup process.

• For Linux, a typical installation involves moving to your home directory and then:

Create a directory

mkdir -p ~/miniconda3

Downloading the installer:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh

Running the installer

bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3

Cleaning up

rm -rf ~/miniconda3/miniconda.sh

• Post-installation, initializing Miniconda for bash and zsh shells is recommended:

~/miniconda3/bin/conda init bash

Additional Notes

• Graphical installer instructions and hash-checking guidelines are available in the Miniconda documentation.

• For specific system setups or advanced configurations, refer to the detailed Miniconda documentation.

This summary provides an overview of Miniconda’s key features, installation process, and system requirements, suitable for inclusion in GitBook documentation.

Create Environment

• Axolotl requires Pytorch >=3.9 and Pytorch >=2.0

• We are going to use Python 3.10 - which is a compromise between the latest Python and the minimum Python environment required

• In this case, we want to create a conda environment with a specific version of Python, so do not use the conda create --clone command to create a clone of the base environment.

When should you clone a base environment?

The base environment in Conda is the default environment that gets created when you install Conda. It is a special environment that contains the Conda package manager itself, along with a collection of installed packages.

What is the Base Environment?

1. Default Environment:

   • The base environment is the default environment active upon the installation of Anaconda or Miniconda. It is where Conda itself, along with a set of pre-installed packages and Python, are located.

2. Contains Conda:

   • Crucially, the base environment includes the Conda package manager. This means you can use Conda to create and manage other environments from the base environment.

3. Pre-installed Packages:

   • When you install Anaconda, the base environment comes with a wide array of commonly used data science packages pre-installed. This makes it a ready-to-use environment for a variety of tasks.

Why Do People Clone the Base Environment?

1. Consistency:

   • Cloning the base environment ensures a consistent starting point. You get a new environment with the same set of packages and configurations as the base, which is often a known, stable setup.

2. Safety:

   • Working directly in the base environment is generally discouraged because changes might affect the stability and functionality of the Conda system itself. Cloning it provides a safe playground where one can install, update, or remove packages without risking the integrity of the Conda installation.

3. Ease of Setup:

   • For users who want a quick setup that includes a broad range of pre-installed packages (like those found in Anaconda), cloning the base environment is a time-saver. It eliminates the need to manually install a long list of common packages.

4. Replicability:

   • Cloning can be used to replicate the base environment across different machines or for different users, ensuring that everyone is working with the same set of tools and libraries.

5. Experimentation and Testing:

   • Cloning the base environment to create a new one allows for experimentation and testing with different package versions or configurations. If something goes wrong, the base environment remains unaffected.

• Use the following command to create an environment named 'axolotl' the contains the base libraries

conda create -n axolotl python=3.10

Once the environment has been created, activate the environment:

conda activate axolotl

To ensure that there are no installed packages in the environment, enter the command:

pip3 freeze

To ensure we only download fresh copies of packages and do not use packages in the cache, enter this command.

pip3 cache purge

Because the environment may be empty; we need to install lit and cmake to install from wheels.

The purpose of running pip install lit cmake is to set up a development environment with specific tools needed for compiling, building, testing, or packaging software, particularly in projects that might involve C/C++ code or require complex build configurations.

pip3 install lit cmake

The output should be similar to this:

Successfully installed cmake-3.29.2 lit-18.1.3

What is 'wheels'? Python Package Format

Definition: A wheel is a built package format for Python, denoted with a .whl file extension. It's a packaging standard introduced by PEP 427, designed to replace the older .egg format.

Advantages: Wheels are more efficient than the traditional setup.py script for installing Python packages.

The key benefits include:

Faster Installation: Wheels are pre-built distributions, meaning the package does not need to be compiled on the end user's machine. This significantly speeds up the installation process, especially for packages that contain compiled extensions.

Consistency: They provide a more consistent installation experience since they include pre-built binaries. This reduces issues stemming from variations in build environments.

Usage: When you run pip install package_name, pip tries to find and download a wheel compatible with your platform and Python version. If it can't find a suitable wheel, it falls back to installing from the source distribution, which can be slower as it may require compiling code.

If you make an error building your conda environment, you can delete it and start again. Follow the instructions below to delete your conda environment and start again:

How to delete a conda environment

To delete a Conda environment, you can follow these steps:

Open your terminal

First deactivate any conda environment you may already be in:

conda deactivate

Activate the base environment (or any other environment you're not deleting) to ensure you don't accidentally delete the active environment. You can do this by running:

conda activate base

To list all your Conda environments, use the following command:

conda env list

This will display a list of all the environments installed on your system along with their paths. You should see the axolotl virtual environment.

To delete the axolotl environment, use the following command:

conda env remove --name axolotl

If asked, confirm the deletion by typing y and pressing Enter when prompted.

The Conda environment will be deleted, and all its packages and dependencies will be removed from your system.

Always make sure you select the Python interpreter in VSCode to the axolotl virtual environment

The Python Interpreter

Remember, when using Visual Studio Code with virtual environments, make sure that you ensure that you have set the Python Interpreter to your virtual environment

Access the Command Palette

Open VS Code, and from the menu, go to View > Command Palette, or use the keyboard shortcut Ctrl+Shift+P.

The Python: Select Interpreter command displays a list of available global environments, conda environments, and virtual environments.

The following image, for example, shows several Anaconda installations along with a conda environment and a virtual environment (env) that's located within the workspace folder:

1. Open VS Code, and from the menu, go to View > Command Palette, or use the keyboard shortcut Ctrl+Shift+P.

2. In the Command Palette, type "Python: Select Interpreter" and select this command when it appears in the list.

Python: Select Interpreter

Make sure you select the axolotl virtual environment we have created.

Reference: Python Interpreter

To select an interpreter for Python and ensure your Conda virtual environment works properly in Visual Studio Code (VS Code):

Access the Command Palette

Open VS Code, and from the menu, go to View > Command Palette, or use the keyboard shortcut Ctrl+Shift+P.

Use the Python: Select Interpreter Command

In the Command Palette, type "Python: Select Interpreter" and select this command when it appears in the list. Alternatively, you can use the keyboard shortcut Ctrl+Shift+P to bring up the Command Palette and directly type "Python: Select Interpreter."

Select an Interpreter

Upon selecting the "Python: Select Interpreter" command, a list of available Python environments will be displayed. This list includes global environments, Conda environments, and virtual environments. You can choose from the listed environments. The list will look similar to the example shown in the documentation.

Remember Your Selection

If you have a folder or workspace open in VS Code when you select an interpreter, the Python extension will remember your choice for that specific workspace. This means that the same interpreter will be used when you reopen that workspace in the future.

Status Bar Confirmation

After selecting an interpreter, the selected environment version will be displayed on the right side of the Status Bar in VS Code. This serves as a confirmation that the interpreter has been set.

Optional

Prevent Automatic Activation: By default, VS Code will automatically activate the selected environment when you open a terminal within VS Code. If you want to prevent this automatic activation, you can add "python.terminal.activateEnvironment": false to your settings.json file.

Manually Specify an Interpreter (if needed)

If VS Code doesn't automatically locate an interpreter you want to use, you can manually specify it. To do this, run the "Python: Select Interpreter" command and choose the "Enter interpreter path..." option from the top of the interpreters list. You can then either enter the full path to the Python interpreter directly or browse your file system to find it.

Selecting the Python interpreter in Visual Studio Code (VS Code) is necessary because there are various Python environments and installations that can coexist on a system.

The reason why it can't always be done automatically is due to the complexity of managing and identifying Python environments, especially in cases where multiple versions or virtual environments are present. Here are some key reasons:

User Preferences: Developers may have their preferred Python interpreter for different types of projects. For example, they may want to use Python 3.8 for one project and Python 3.9 for another. Automatic selection might not align with their preferences.

Path Variations: Python interpreters can be located in various directories, and their paths may not always follow a consistent pattern. Manually selecting the interpreter allows users to specify the exact path.

Conda Environments: Conda is a package manager that creates isolated environments for projects. Detecting and managing Conda environments automatically can be complex due to the additional layer of isolation.

Compatibility: Python extensions and packages can have compatibility issues with certain Python versions. Manually selecting the interpreter allows users to ensure compatibility with their specific project requirements.

Pytorch Installation

--> The environment should contain Pytorch 2.0

Go to the Pytorch website and select your variables - version, operating system, package (Conda), Language, Computer Platform. It will provide you the command to run in your activated Conda environment.

https://pytorch.org/get-started/locally/

Enter in the required details

Then take the output provided and go into your terminal and enter the command. Ensure that you are within the conda environment 'axolotl' (conda activate axolotl) if you are not enter the command:

conda activate axolotl

The command below is for installation of Pytorch 2.1 on a Linux terminal using Python and CUDA 12.1

conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia

Press (y) when asked to download the libraries suggested/

This is a breakdown of the command:

• conda install: This indicates that you're using Conda

• pytorch: This is the main PyTorch library

• torchvision: A package for PyTorch that provides utilities for image and video processing

• torchaudio: A package for PyTorch tailored for audio processing

• pytorch-cuda=12.1: This specifies that you want the PyTorch build that is compatible with CUDA 12.1.

• -c pytorch: This tells Conda to install from the pytorch channel on Anaconda Cloud.

• -c nvidia: This specifies to also use the nvidia channel, which is necessary for CUDA-related packages.

Instruction

When the pytorch installation command is executed, the terminal will ask if you want to install all of the Pytorch packages - reply yes.

By following these steps, you should be able to successfully install the Axolotl library on your Ubuntu 20.04 version.

Reference: How does Pytorch relate to CUDA?

• PyTorch is an open-source machine learning library for building and training neural networks.

• PyTorch uses CUDA to accelerate its operations on NVIDIA GPUs. This acceleration is particularly crucial for training large and complex deep learning models, a process that can be computationally intensive and time-consuming.

How They Work Together

• Acceleration of Tensor Operations: PyTorch performs a large number of operations on tensors, which are multi-dimensional arrays. CUDA accelerates these operations when executed on NVIDIA GPUs.

• Transparent Usage: PyTorch abstracts the complexity of CUDA, making it easier for users. Developers can write standard PyTorch code, and if a CUDA-enabled GPU is available, PyTorch will automatically use CUDA to accelerate operations.

• CUDA Tensors: PyTorch introduces 'CUDA tensors', similar to normal tensors but located in the GPU's memory. Operations on CUDA tensors are performed on the GPU, offering significant speed improvements.

How to Think About Their Interaction

• Complementary Roles: Think of CUDA as an underlying engine that provides horsepower to PyTorch. While PyTorch handles the creation and manipulation of tensors and neural networks, CUDA provides the capability to execute these operations quickly on a GPU.

• Ease of Development: From a developer’s perspective, CUDA’s complexities are mostly hidden. You write PyTorch code as usual, and PyTorch, coupled with CUDA, takes care of efficiently executing operations on the GPU.

• Scalability and Performance: In environments where training large models or processing large datasets is required, the CUDA-PyTorch integration becomes critical. It allows leveraging GPU acceleration for better performance, reducing training time from days to hours or hours to minutes.

• Conditional Usage: CUDA is utilized by PyTorch only if a compatible NVIDIA GPU is available. Otherwise, PyTorch defaults to using the CPU.

Install "Packaging" Package

Now install the packaging and ninja python packages.

The packaging library provides utilities for version handling, specifiers, markers, requirements, tags, and more, which are often used in package management and installation.

 install  ninja

Run the Axolotl setup.py file

• Ensure that you are in the base directory that contains the setup.py file.

• Run the installation command:

pip3 install -e '.[flash-attn,deepspeed]'

-e flag

Stands for "editable" mode. When you install a package in editable mode, Python installs the package in a way that allows you to modify the package source code and see the changes directly without needing to reinstall the package. This is particularly useful for development purposes.

This is important

The dot "." represents the current directory. In this context, it indicates that pip should install the package located in the current directory.

This directory contains the setup.py file, which is the build script for setuptools. It tells setuptools about your package (such as the name and version) and the files that belong to it.

[flash-attn,deepspeed]

These are extras.

Extras are additional dependencies that are relevant for optional features of the package.

In this case, flash-attn and deepspeed are optional dependencies. When you specify these extras, pip will also install the dependencies associated with these features as defined in the package's setup.py.

This command will install the Axolotl package along with its dependencies as specified in setup.py.

You can find a detailed explanation of the setup.py script here: setup.py decomposition

setup.py installation process

Key Actions

Installation Process: The script is executing the installation of the Axolotl package. This involves several steps:

• Running Egg Info: Gathering package information and writing it to the axolotl.egg-info directory.

• Building Distribution: Creating a build directory and compiling the package into an 'egg' format, a type of Python distribution.

• Copying and Installing: The compiled package is then copied to your Miniconda environment's site-packages directory.

• Processing Dependencies: The script is also handling dependencies, downloading, and installing them as required. This includes packages like gcsfs, s3fs, tensorboard, gradio, fschat, art, pynvml, scikit-learn, scipy, rouge-score, and evaluate.

Warnings and Recommendations

1. Deprecation of setup.py install: The script warns that setup.py install is deprecated. This is a Setuptools deprecation warning, encouraging the use of more modern, standards-based tools like pypa/build and pypa/installer.

   • Implication: While setup.py install still works, it's recommended to use newer tools for package building and installation in future projects to stay aligned with Python packaging standards.

2. Deprecation of easy_install: Similar to the previous warning, there’s a notice about the deprecation of easy_install, a legacy package installation method.

Installation Summary

• The Axolotl library and its dependencies are being installed into your specified conda environment.

• The package is being built and installed using an older method (setup.py install), which, although operational, is not recommended due to deprecation warnings.

• The process involves the creation of a package distribution in the 'egg' format and the installation of various dependencies listed in the package's requirements.txt file.

Install Logging

We will be using Weights and Biases for training logging.

Weights and Biases should already be installed, if it is not - to install Weights and Biases:

pip3 install wandb

Login to Weights and Biases

wandb login

When asked, complete the following fields for your Weights and Biases account:

Username:

Password:

API Token:

References: Weights and Biases

To set up Weights and Biases (W&B) for tracking experiments, follow these steps:

Create an Account and Install W&B

• Sign up for a free account at https://wandb.ai/site and log in to your W&B account.

• Install the W&B library on your machine in a Python 3 environment using pip:

pip install wandb

Log in to W&B

• In your Python script or notebook, import the W&B Python SDK and log in using wandb.login(). You will be prompted to provide your API key.

import wandb
wandb.login()

Start a Run and Track Hyperparameters

• Initialise a W&B Run object with wandb.init(). You can specify project information and track hyperparameters in the configuration.

run = wandb.init(
    project="my-awesome-project",
    config={
        "learning_rate": 0.01,
        "epochs": 10,
    },
)

A run is a fundamental unit in W&B used to track metrics, logs, jobs, and more.

Example Script: Below is an example script that demonstrates how to use W&B for tracking hyperparameters and metrics in a training script.

import wandb
import random

wandb.login()

epochs = 10
lr = 0.01

run = wandb.init(
    project="my-awesome-project",
    config={
        "learning_rate": lr,
        "epochs": epochs,
    },
)

offset = random.random() / 5
print(f"lr: {lr}")

for epoch in range(2, epochs):
    acc = 1 - 2**-epoch - random.random() / epoch - offset
    loss = 2**-epoch + random.random() / epoch + offset
    print(f"epoch={epoch}, accuracy={acc}, loss={loss}")
    wandb.log({"accuracy": acc, "loss": loss})

# run.log_code()

View Results: After running your script, you can view the tracked metrics, including accuracy and loss, in the W&B App at https://wandb.ai/home.

What's Next?: Explore other features of the W&B ecosystem, such as integrations, reports, artifacts, sweeps, and more to enhance your machine learning experiments.

Additionally, the documentation provides answers to common questions, such as finding your API key, using W&B in automated environments, setting up local installations, and temporarily disabling W&B logging.

Reference: Weights and Biases python libraries

When you installed the Weights and Biases (W&B) library using the pip install wandb command, the following libraries were installed along with W&B:

Click: A library for creating command-line interfaces.

GitPython: A Python library to interact with Git repositories.

psutil: A cross-platform library for retrieving information on system utilization.

sentry_sdk: A Python SDK for Sentry, which helps with error tracking and monitoring.

docker-pycreds: A library for working with Docker credentials.

setproctitle: A library for setting the process title on Unix platforms.

appdirs: A library for determining appropriate platform-specific directories.

protobuf: A library for working with Protocol Buffers, used for efficient data serialization.

six: A Python 2 and 3 compatibility library.

gitdb: A library for interacting with Git object databases.

smmap: A library for managing sparse memory-mapped files.

These libraries are dependencies that are required by W&B to function properly.

They help provide various functionalities, including version control, system monitoring, and efficient data serialisation, which are useful for tracking machine learning experiments effectively using W&B.

What are the main classes within the Transformer Library?

The main classes within the library are:

1. Model Classes: These classes correspond to specific transformer models. Each model class in the library is usually named after the model it represents, such as BertModel for BERT, GPT2Model for GPT-2, T5Model for T5, etc. These classes are used to instantiate model architectures, either from pretrained weights or from scratch.

2. Tokenizer Classes: Tokenizers are responsible for preprocessing text input for transformer models. They convert text into a format that is understandable by the model. Classes like BertTokenizer and GPT2Tokenizer handle this task. There are also "fast" tokenizers (e.g., BertTokenizerFast), which are optimized for speed and provide additional functionalities.

3. Configuration Classes: Configuration classes store the configurations of a model — for example, BertConfig. These classes contain all the settings necessary to build a model, such as the number of layers, hidden units, and attention heads.

4. Pipeline Classes: The pipeline class abstracts away much of the preprocessing and postprocessing work involved in using models. For instance, the pipeline function allows users to perform tasks like text classification, question answering, and text generation using a simple API.

5. Trainer Class: The Trainer class provides an easy-to-use interface for training, evaluating, and fine-tuning transformer models. It handles many training details like data collation, optimization, and logging.

6. Data Collator Classes: These are utility classes used to collate batches of data. They are especially useful when dealing with variable-sized input, such as in language modeling tasks.

7. Optimizer and Scheduler Classes: While not exclusive to Transformers, these classes are often used in conjunction with model training. They define the optimization algorithm and learning rate schedule used during training.

If you are interested in understanding the libraries and processes installed via the setup.py script, please review the following pages.