Phi 2.0 - Model Quantization

With the model configured, we next have to determine whether whether we will be using quantization in the training process.

Given Phi 2.0 is such a small model, in this case we will not be using quantization during the training process.

Model Quantization

#These are the default values

llm_int8_has_fp16_weight: false
bnb_4bit_quant_type: nf4  #4 bit normal float data type
bnb_4bit_use_double_quant: true

#You can override the default values as per below

load_in_8bit: true
load_in_4bit: false
strict: false

load_in_8bit: true or false

This is configuration flag that determines whether the model should be loaded in 8-bit precision. If it is set to "true," the model will be loaded in 8-bit precision.

Memory Efficiency

8-bit precision reduces the memory footprint of the model compared to higher precision formats (like the default 16-bit). This is because it requires less memory to store each weight in the model.

Loading a model in 8-bit precision can accelerate model loading and inference times. This is due to the reduced computational load compared to higher precision formats.

While 8-bit precision is more efficient, it can slightly reduce the accuracy of the model compared to full precision (32-bit). This happens because of the reduced resolution in representing the weights and activations.

Reference: BitsAndBytesConfig Class from the Transformers Library

This class is a wrapper for configuring and managing the quantization settings when loading a model using the bitsandbytes library.

Quantization is a technique used to reduce the memory footprint and computational cost of deep learning models by representing weights and activations with lower-precision data types, such as int8 or 4-bit floating-point numbers.

The bitsandbytes library provides methods for quantizing models, and the BitsAndBytesConfig class acts as a configuration object to control the quantization settings.

Let's go through the main aspects of the BitsAndBytesConfig class:

Initialization

• The class takes several arguments in its constructor to configure the quantization settings.

• The main arguments are load_in_8bit and load_in_4bit, which are mutually exclusive and determine whether to use 8-bit or 4-bit quantization.

• Other arguments include threshold values, module exclusion lists, and settings specific to the bitsandbytes library.

Properties and Setters

• The class provides properties and setters for the load_in_4bit and load_in_8bit attributes.

• The setters enforce the mutual exclusivity of these attributes and validate the input values.

Post-initialization

• The post_init() method is called after initialization to perform safety checks on the provided arguments.

• It ensures that the arguments have the correct data types and raises a ValueError if any inconsistencies are found.

• It also checks the version of the bitsandbytes library to ensure compatibility with 4-bit quantization.

Quantization Methods

• The is_quantizable() method returns True if the model is quantizable based on the load_in_8bit or load_in_4bit flags.

• The quantization_method() method returns the specific quantization method used, such as "llm_int8", "fp4", or "nf4", based on the configuration.

Serialization

• The to_dict() method serializes the configuration instance to a Python dictionary, converting the PyTorch data types to strings for serialization.

• The to_diff_dict() method serializes only the attributes that differ from the default configuration, providing a more concise representation.

Representation

• The __repr__() method provides a string representation of the configuration instance, displaying the class name and the serialized dictionary.

The BitsAndBytesConfig class is designed to work seamlessly with the Transformers library and the bitsandbytes library for quantizing models. It provides a convenient way to configure and manage the quantization settings when loading a model.

Here are a few key points to note:

• The class supports both 8-bit and 4-bit quantization, controlled by the load_in_8bit and load_in_4bit flags.

• It allows specifying threshold values for outlier detection in 8-bit quantization, which can help maintain performance for large models.

• It provides options to exclude certain modules from quantization and to enable offloading of non-quantized parts to CPU.

• The class performs validation checks to ensure the consistency and compatibility of the provided arguments.

Overall, the BitsAndBytesConfig class is an important component in the Transformers library for enabling quantization of models using the bitsandbytes library. It provides a flexible and configurable interface to control the quantization settings and optimize the performance and memory usage of deep learning models.

The supporting academic paper

This November 2022 paper paper presents a novel quantization method for large language models (LLMs) that enables efficient inference without performance degradation. The key points of the paper are:

The authors develop a two-part quantization procedure called LLM.int8() that allows for the use of 8-bit matrix multiplication in feed-forward and attention projection layers of transformers, reducing memory requirements by half while maintaining full precision performance.

The authors demonstrate that LLM.int8() enables inference in LLMs with up to 175B parameters without any performance degradation, making such models more accessible to researchers and practitioners.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScalearXiv.org

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

load_in_4bit: true or false

This is configuration flag that determines whether the model should be loaded in 4-bit precision. If it is set to "true," the model will be loaded in 4-bit precision.

4-bit precision takes the concept of memory efficiency further, halving the memory requirements compared to 8-bit. This can be crucial for deploying large models on limited hardware.

Similar to 8-bit, 4-bit precision can lead to even faster loading and inference times due to the further reduced computational requirements.

The trade-off in accuracy might be more pronounced in 4-bit precision. The reduced bit-depth means that the model's ability to represent nuanced information in weights and activations is more limited. This might affect tasks that require high precision or are sensitive to small changes in weights.

strict: true or false

If set to false, default weights will be chosen where missing in adapters. This is a component of 'bits and bytes' library

Reference: What is 'bits and bytes'?

Overview

The bitsandbytes repository provides a lightweight wrapper around CUDA custom functions, primarily focusing on 8-bit optimizers, matrix multiplication (LLM.int8()), and quantization functions.

This tool is designed to enhance the performance and efficiency of machine learning models, particularly in the context of CUDA-enabled computing environments.

Key Features

• 8-bit Optimizers: Specialised for reducing memory usage and improving computational efficiency.

• Matrix Multiplication (LLM.int8()): Offers optimized matrix multiplication capabilities.

• Quantization Functions: Includes various methods for quantizing models, contributing to reduced model sizes and potentially faster inference times.

Requirements

• Python version 3.8 or higher.

• Linux distribution (Ubuntu, MacOS, etc.) with CUDA version greater than 10.0.

• Note: CUDA 10.0 is deprecated, and future support is focused on CUDA >= 11.0 with release 0.39.0.

Installation

• Installable via pip (pip install bitsandbytes).

• In cases where compilation from source is necessary, users are encouraged to submit a bug report and follow the provided compilation instructions.

Usage Highlights

• Int8 Inference with HuggingFace Transformers: Allows models to load in 8-bit for reduced memory usage.

• 8-bit Optimizer Usage: Users can easily switch to 8-bit optimizers by replacing their existing optimizers with the corresponding 8-bit version from bitsandbytes.

• Mixed 8-bit Training and Int8 Inference: The library supports both mixed 8-bit training with 16-bit main weights and full 8-bit inference.

Features

• Advanced techniques for 8-bit matrix multiplication and LLM.int8() inference.

• A range of 8-bit optimizers including Adam, AdamW, RMSProp, LARS, LAMB, and Lion.

• A stable embedding layer feature for improved stability in NLP models.

• Fast and efficient algorithms for quantile estimation.

Requirements & Hardware Compatibility

• Requires Anaconda, cudatoolkit, and PyTorch.

• Compatible with NVIDIA GPUs, specifically Turing or newer for LLM.int8(), and Kepler or newer for 8-bit optimizers and quantization.

• Supports CUDA versions from 10.2 to 12.0.

• Note: The library is currently supported only on Linux distributions.

Summary of Tim Dettmers' Presentation on 8-Bit Methods for Efficient Deep Learning

His main thesis is that computationally efficient methods will accelerate progress in understanding deep learning.

Key Points from Tim's Presentation

8-Bit Methods for Large Models: Tim highlights the importance of making large models more accessible through quantization, which reduces the memory footprint.

Quantization Explained: He explains quantization as a process of converting floating-point or real representations into discrete buckets, akin to histogram binning.

Linear vs. Nonlinear Quantization: Linear (integer) quantization involves equally wide bins, while nonlinear quantization allows varying bin widths.

Error Reduction in Quantization: Tim illustrates how the choice of bins impacts precision and error distribution in quantized values.

4-Bit Inference: His recent work shows that 4-bit inference is highly effective for large transformers.

Floating Point Data Types: The presentation delves into the structure of floating point data types, explaining the roles of exponent bits and fraction bits.

Dynamic Exponent Data Type: Tim introduces a unique data type he developed with a dynamic exponent, which offers flexibility in approximating large and small values with varying precision.

8-Bit Optimizers: The focus shifts to 8-bit optimizers, crucial for memory efficiency in training large models, particularly in language modeling.

Tim discusses reducing memory usage by approximately 40% by converting 32-bit Adam optimizer buffers to 8-bit.

This reduction is significant as it helps make large models more memory-efficient.

Outliers in Adam optimizer buffers cause issues in quantization, leading to increased error and ineffective 8-bit quantization.

Tim presents an example showing how outliers can skew the data, leading to a waste of bits and loss of effective representation.

To address the problem of outliers, Tim proposes chunking Adam states into blocks and quantizing each block independently.

This method isolates the impact of outliers to specific blocks, enhancing the stability of 8-bit optimizers.

The process involves chunking state into blocks, finding the maximum value for normalization, and storing the index for 8-bit representation.

This method ensures compact yet effective optimization, comparable to 32-bit optimizers.

This achievement indicates significant memory savings without compromising performance.

8-bit optimizers are efficient in mapping onto hardware, with the main overhead being the dequantization process.

Outliers become a significant problem in models larger than 6.7 billion parameters, causing performance drops.

Tim's research identifies systematic outliers that emerge with scale and become problematic at specific model sizes.

Outliers in large models exhibit systematic and emergent properties, affecting the same dimensions across layers.

These outliers impact all layers in a transformer model once a certain scale is reached.

The emergence of outliers follows an exponential trend, leading to a phase shift-like effect at a certain scale.

Understanding and addressing this exponential trend is key to managing outliers in large models.

A novel approach was developed to identify and process these outliers in 16-bit while handling the rest in 8-bit, effectively maintaining efficiency while addressing the problem.

Efficiency of 8-Bit Matrix Multiplication

• By applying this method, 99.9% of weights are computed in 8-bit, with a small portion in 16-bit for outliers. This approach achieves performance equivalent to 16-bit computations while halving memory size.

• This makes large models like Llama 65B accessible on consumer hardware, significantly lowering the barrier to entry for working with such models.

Few-Shot and Zero-Shot Performance

• The few-shot performance of models using 8-bit methods is comparable to 16-bit models. Tim highlighted a strong correlation between zero-shot performance and perplexity in language models, indicating that complexity evaluations can reliably predict zero-shot performance.

Understanding Outliers in Transformer Models

• Outliers tend to be concentrated in specific columns of the input batch and are more prevalent in larger models. These outliers are crucial for attention mechanisms in transformers.

• They are context-independent, aiding the attention mechanism in focusing on specific values by providing predictable patterns for the model to cancel out unnecessary information.

Trade-Offs in Activation Functions

• Replacing traditional activation functions like softmax with more stable alternatives can increase stability but may lead to a drop in performance.

• This presents a research challenge in balancing stability with maintaining or enhancing model performance.

Impact of Precision and Parameter Count on Model Efficiency

• An interesting finding is that models with the same number of bits but different distributions of precision and parameter count (e.g., 8-bit with more parameters vs. 4-bit with fewer parameters) exhibit the same inference latency.

• This equivalence in performance is due to the nature of GPU computations, where memory loading is significantly more costly than computation. Therefore, the memory used during inference, rather than the computational complexity, often dictates the performance.