Introduction

This is Continuum's documentation for training large language models.

This training platform has been put together by a dedicated group of people, whose generosity has allowed a community to form and become able to fine tune a variety of large language models.

GitHub - OpenAccess-AI-Collective/axolotl: Go ahead and axolotl questionsGitHub

Link to Axolotl GitHub Repository

Background

The GitHub repository "Axolotl" provides a versatile tool designed for the fine-tuning of various AI models, specifically targeting ease of use and flexibility in handling different model configurations and architectures.

Core Purpose

Axolotl is aimed at streamlining the fine-tuning process of AI models, offering compatibility with a wide range of Huggingface models and fine-tuning techniques.

Supported Features

• Model Support: It supports training with various Huggingface models

• Fine-tuning Techniques: The tool supports several fine-tuning methods

• Configuration Flexibility: Users can customise configurations using a YAML file or override settings via the CLI.

• Dataset Compatibility: Axolotl can load different dataset formats, support custom formats, or handle user-provided tokenized datasets.

• Integration with Advanced Tools: The tool integrates with xformer, flash attention, rope scaling, and multipacking for enhanced model performance and efficiency.

• Multi-GPU Support: It facilitates training on single or multiple GPUs using FSDP or Deepspeed.

• Docker Support: Axolotl can be easily run with Docker, either locally or on the cloud.

• Experiment Tracking: It allows logging results and optionally checkpoints to WandB or MLflow.