SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

5X faster 60% less memory QLoRA finetuning

A high-throughput and memory-efficient inference and serving engine for LLMs

BitBLAS is a library to support mixed-precision matrix multiplications, especially for quantized LLM deployment.

SOTA Weight-only Quantization Algorithm for LLMs

Official implementation of Half-Quadratic Quantization (HQQ)

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

QLoRA: Efficient Finetuning of Quantized LLMs

FlashInfer: Kernel Library for LLM Serving

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

Enforce the output format (JSON Schema, Regex etc) of a language model

The official PyTorch implementation of Google's Gemma models

Fast and memory-efficient exact attention

Instruct-tune LLaMA on consumer hardware

GPTQ inference Triton kernel

4 bits quantization of LLaMa using GPTQ

Port of Facebook's LLaMA model in C/C++

A hyper-fast local vector database for use with LLM Agents. Now accepting SAFEs at $35M cap.

The release repo for "Vicuna: An Open Chatbot Impressing GPT-4"

Source code for Twitter's Recommendation Algorithm

gpt4all: a chatbot trained on a massive collection of clean assistant data including code, stories and dialogue

🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production

A Smart Ethernet Switch for Earth

This dataset contains synthetic training data for grammatical error correction. The corpus is generated by corrupting clean sentences from C4 using a tagged corruption model. The approach and the d…

libheif is an HEIF and AVIF file format decoder and encoder.