Welcome to Bitorch Engine’s documentation!

Welcome to the documentation of Bitorch Engine (BIE): a cutting-edge computation library for neural networks that enhances PyTorch by integrating specialized layers and functions for Low-Bit quantized neural network operations. This is where you can find all the information you need about how to use BIE.

Building on the foundational strengths of Bitorch Engine, the technology has been employed in pioneering projects that push the boundaries of neural network training and inference. For instance,

• green-bit-llm-trainer: In this project, BIE represents a significant leap in the field of Large Language Model (LLM) fine-tuning. Unlike traditional approaches that either quantize a fully trained model or introduce a few additional trainable parameters for LoRA style fine-tuning, this project innovates by directly fine-tuning the quantized parameters of LLMs. This paradigm shift allows for the full-scale quantization fine-tuning of LLMs, ensuring that the training process tightly integrates with the quantization schema from the outset.

• green-bit-llm-inference also showcases the BIE’s adeptness at supporting inference for models quantized from 4 to 2-bits without any significant loss in accuracy compared to the original 32 or 16-bits models. It stands as a testament to BIE’s capability to maintain the delicate balance between model size, computational efficiency, and accuracy, addressing one of the key challenges in deploying sophisticated neural networks in resource-constrained environments.

All changes are tracked in the changelog.

Contents:

• Installation
  • Binary Release
  • Build From Source
    • Conda on Linux (with CUDA)
    • Docker (with CUDA)
    • Conda on MacOS (with MLX)
• Build options
  • Building Specific Extensions
  • Building for a Specific CUDA Architecture
  • Force Building CUDA Modules
  • Skip Library File Building
• Full Documentation
  • bitorch_engine
    • bitorch_engine.functions
      • bitorch_engine.functions.cuda
    • bitorch_engine.layers
      • bitorch_engine.layers.qconv
      • bitorch_engine.layers.qembedding
      • bitorch_engine.layers.qlinear
      • bitorch_engine.layers.qmha
    • bitorch_engine.optim
      • bitorch_engine.optim.diode_beta
      • bitorch_engine.optim.galore_projector
    • bitorch_engine.utils
      • bitorch_engine.utils.arch_helper
      • bitorch_engine.utils.convert
      • bitorch_engine.utils.cpp_extension
      • bitorch_engine.utils.cuda_extension
      • bitorch_engine.utils.cutlass_path
      • bitorch_engine.utils.mlx_extension
      • bitorch_engine.utils.mlx_path
      • bitorch_engine.utils.model_helper
      • bitorch_engine.utils.quant_operators
      • bitorch_engine.utils.safe_import

Indices and tables

• Index

• Module Index

• Search Page

Enjoy exploring our documentation!