LogoLogo
Continuum Knowledge BankContinuum Applications
  • Introduction
  • Creation of Environment
    • Platform Installation
    • Axolotl Dependencies
    • setup.py objectives
      • script analysis
  • Huggingface Hub
  • Download the dataset
    • Types of Dataset Structures
    • Structuring Datasets for Fine-Tuning Large Language Models
    • Downloading Huggingface Datasets
    • Use Git to download dataset
    • Popular Datasets
    • Download cleaned Alpaca dataset
    • Template-free prompt construction
  • Downloading models
    • Phi 2.0 details
    • Downloading Phi 2.0
    • Available Models
  • Configuration for Training
  • Datasets
  • Model Selection - General
  • Phi 2.0
    • Phi 2.0 - Model Configuration
    • Phi 2.0 - Model Quantization
    • Phi 2.0 - Data Loading and Paths
    • Phi 2.0 - Sequence Configuration
    • Phi 2.0 - Lora Configuration
    • Phi 2.0 - Logging
    • Phi 2.0 - Training Configuration
    • Phi 2.0 - Data and Precision
    • Phi 2.0 - Optimisations
    • Phi 2.0 - Extra Hyperparameters
    • Phi 2.0 - All Configurations
    • Phi 2.0 - Preprocessing
    • Phi 2.0 - Training
    • Uploading Models
  • Llama2
    • Llama2 - Model Configuration
    • Llama2 - Model Quantization
    • Llama2 - Data Loading and Paths
    • Llama2 - Sequence Configuration
    • Llama2 - Lora Configuration
    • Llama2 - Logging
    • Llama2 - Training Configuration
    • Llama2 - Data and Precision
    • Llama2 - Optimisations
    • Llama2 - Extra Hyperparameters
    • Llama2- All Configurations
    • Llama2 - Training Configuration
    • Llama2 - Preprocessing
    • Llama2 - Training
  • Llama3
    • Downloading the model
    • Analysis of model files
      • Model Analysis - Configuration Parameters
      • Model Analysis - Safetensors
      • Tokenizer Configuration Files
        • Model Analysis - tokenizer.json
        • Model Analysis - Special Tokens
    • Llama3 - Model Configuration
    • Llama3 - Model Quantization
    • Llama3 - Data Loading and Paths
    • Llama3 - Sequence Configuration
    • Llama3 - Lora Configuration
    • Llama3 - Logging
    • Llama3 - Training Configuration
    • Llama3 - Data and Precision
    • Llama3 - Optimisations
    • Llama3 - Extra Hyperparameters
    • Llama3- All Configurations
    • Llama3 - Preprocessing
    • Llama3 - Training
    • Full Fine Tune
  • Special Tokens
  • Prompt Construction for Fine-Tuning Large Language Models
  • Memory-Efficient Fine-Tuning Techniques for Large Language Models
  • Training Ideas around Hyperparameters
    • Hugging Face documentation on loading PEFT
  • After fine tuning LLama3
  • Merging Model Weights
  • Merge Lora Instructions
  • Axolotl Configuration Files
    • Configuration Options
    • Model Configuration
    • Data Loading and Processing
    • Sequence Configuration
    • Lora Configuration
    • Logging
    • Training Configuration
    • Augmentation Techniques
  • Axolotl Fine-Tuning Tips & Tricks: A Comprehensive Guide
  • Axolotl debugging guide
  • Hugging Face Hub API
  • NCCL
  • Training Phi 1.5 - Youtube
  • JSON (JavaScript Object Notation)
  • General Tips
  • Datasets
Powered by GitBook
LogoLogo

This documentation is for the Axolotl community

On this page
  • Repository Management
  • Model and Dataset Information
  • Listing Resources
  • File Management
  • Authentication
  • Local Storage
  • Filtering Helpers
  • Technical Details

Was this helpful?

Hugging Face Hub API

The Hugging Face Hub API, accessed through the huggingface_hub package in Python, provides a comprehensive set of methods and classes to interact with the Hugging Face Hub platform.

It allows users to access and manage repositories, models, datasets, and other resources available on the Hub. Here's a detailed summary of the capabilities of the Hugging Face Hub API:

Repository Management

  • Create a new repository on the Hub using create_repo().

  • Delete a repository from the Hub using delete_repo().

  • Update the visibility (public/private) of a repository using update_repo_visibility().

  • Move a repository from one namespace to another using move_repo().

Model and Dataset Information

  • Get information about a specific model using model_info().

  • Get information about a specific dataset using dataset_info().

  • Get information about a specific space using space_info().

  • Get generic repository information using repo_info().

Listing Resources

  • List all available models on the Hub using list_models().

  • List all available datasets on the Hub using list_datasets().

  • List all available metrics on the Hub using list_metrics().

  • Filter models and datasets based on various criteria using ModelFilter and DatasetFilter classes.

File Management

  • Upload a local file to a repository using upload_file().

  • Delete a file from a repository using delete_file().

  • List all files in a repository using list_repo_files().

Authentication

  • Set the access token for authentication using set_access_token().

  • Unset the access token using unset_access_token().

  • Get information about the authenticated user using whoami().

Local Storage

  • The HfFolder class provides methods to manage local storage of authentication tokens.

  • Save a token using save_token(), get a token using get_token(), and delete a token using delete_token().

Filtering Helpers

  • The ModelFilter and DatasetFilter classes provide an easy way to construct filters for searching models and datasets based on various criteria such as author, task, language, tags, etc.

  • The ModelSearchArguments and DatasetSearchArguments classes provide tab-completion and access to all possible values for properties of models and datasets hosted on the Hub.

Technical Details

  • The API communicates with the Hugging Face Hub platform using HTTP requests.

  • Authentication is performed using access tokens, which can be obtained from the Hugging Face website and set using the set_access_token() method.

  • The API uses JSON format for data exchange.

  • The HfApi class serves as the main entry point for interacting with the API, and most methods are accessible directly from the huggingface_hub package.

  • The API supports various parameters for filtering, sorting, and pagination when listing resources.

  • Error handling is provided through specific exception classes such as RepositoryNotFoundError and RevisionNotFoundError.

  • The API allows specifying the revision (e.g., branch or tag) when accessing repository-related information.

  • The upload_file() method supports uploading files up to 5GB in size.

The Hugging Face Hub API provides a powerful and flexible way to interact with the Hugging Face Hub platform programmatically.

It enables users to manage repositories, access model and dataset information, upload and delete files, and perform various search and filtering operations.

PreviousAxolotl debugging guideNextNCCL

Last updated 1 year ago

Was this helpful?

Page cover image