🦾 OpenLLM: Self-Hosting LLMs Made Easy

OpenLLM helps developers run any open-source LLMs, such as Llama 2 and Mistral, as OpenAI-compatible API endpoints, locally and in the cloud, optimized for serving throughput and production deployment.

• 🚂 Support a wide range of open-source LLMs including LLMs fine-tuned with your own data
• ⛓️ OpenAI compatible API endpoints for seamless transition from your LLM app to open-source LLMs
• 🔥 State-of-the-art serving and inference performance
• 🎯 Simplified cloud deployment via BentoML

For starter, we provide two ways to quickly try out OpenLLM:

Try this OpenLLM tutorial in Google Colab: Serving Llama 2 with OpenLLM.

We provide a docker container that helps you start running OpenLLM:

docker run --rm -it -p 3000:3000 ghcr.io/bentoml/openllm start facebook/opt-1.3b --backend pt

docker run --rm --gpus all -p 3000:3000 -it ghcr.io/bentoml/openllm start HuggingFaceH4/zephyr-7b-beta --backend vllm

The following provides instructions for how to get started with OpenLLM locally.

You have installed Python 3.8 (or later) and pip. We highly recommend using a Virtual Environment to prevent package conflicts.

Install OpenLLM by using pip as follows:

pip install openllm

To verify the installation, run:

$ openllm -h

Usage: openllm [OPTIONS] COMMAND [ARGS]...

   ██████╗ ██████╗ ███████╗███╗   ██╗██╗     ██╗     ███╗   ███╗
  ██╔═══██╗██╔══██╗██╔════╝████╗  ██║██║     ██║     ████╗ ████║
  ██║   ██║██████╔╝█████╗  ██╔██╗ ██║██║     ██║     ██╔████╔██║
  ██║   ██║██╔═══╝ ██╔══╝  ██║╚██╗██║██║     ██║     ██║╚██╔╝██║
  ╚██████╔╝██║     ███████╗██║ ╚████║███████╗███████╗██║ ╚═╝ ██║
   ╚═════╝ ╚═╝     ╚══════╝╚═╝  ╚═══╝╚══════╝╚══════╝╚═╝     ╚═╝.

  An open platform for operating large language models in production.
  Fine-tune, serve, deploy, and monitor any LLMs with ease.

Options:
  -v, --version  Show the version and exit.
  -h, --help     Show this message and exit.

Commands:
  build       Package a given models into a BentoLLM.
  import      Setup LLM interactively.
  models      List all supported models.
  prune       Remove all saved models, (and optionally bentos) built with OpenLLM locally.
  query       Query a LLM interactively, from a terminal.
  start       Start a LLMServer for any supported LLM.

Extensions:
  build-base-container  Base image builder for BentoLLM.
  dive-bentos           Dive into a BentoLLM.
  get-containerfile     Return Containerfile of any given Bento.
  get-prompt            Get the default prompt used by OpenLLM.
  list-bentos           List available bentos built by OpenLLM.
  list-models           This is equivalent to openllm models...
  playground            OpenLLM Playground.

OpenLLM allows you to quickly spin up an LLM server using openllm start. For example, to start a phi-2 server, run the following:

TRUST_REMOTE_CODE=True openllm start microsoft/phi-2

This starts the server at http://0.0.0.0:3000/. OpenLLM downloads the model to the BentoML local Model Store if it has not been registered before. To view your local models, run bentoml models list.

To interact with the server, you can visit the web UI at http://0.0.0.0:3000/ or send a request using curl. You can also use OpenLLM’s built-in Python client to interact with the server:

import openllm

client = openllm.client.HTTPClient('http://localhost:3000')
client.query('Explain to me the difference between "further" and "farther"')

Alternatively, use the openllm query command to query the model:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'Explain to me the difference between "further" and "farther"'

OpenLLM seamlessly supports many models and their variants. You can specify different variants of the model to be served. For example:

openllm start <model_id> --<options>

DTYPE=float32 openllm start microsoft/phi-2

OpenLLM currently supports the following models. By default, OpenLLM doesn't include dependencies to run all models. The extra model-specific dependencies can be installed with the instructions below.

TRUST_REMOTE_CODE=True openllm start baichuan-inc/baichuan-7b

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start thudm/chatglm-6b

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start databricks/dbrx-instruct

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start databricks/dolly-v2-3b

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start tiiuae/falcon-7b

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start google/flan-t5-large

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start google/gemma-7b

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start eleutherai/gpt-neox-20b

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start NousResearch/llama-2-7b-hf

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start mistralai/Mistral-7B-Instruct-v0.1

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start mistralai/Mixtral-8x7B-Instruct-v0.1

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start mosaicml/mpt-7b-instruct

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

openllm start facebook/opt-1.3b

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start microsoft/phi-2

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start qwen/Qwen-7B-Chat

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start stabilityai/stablelm-tuned-alpha-3b

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start bigcode/starcoder

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

TRUST_REMOTE_CODE=True openllm start 01-ai/Yi-6B

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

More models will be integrated with OpenLLM and we welcome your contributions if you want to incorporate your custom LLMs into the ecosystem. Check out Adding a New Model Guide to learn more.

OpenLLM allows you to start your model server on multiple GPUs and specify the number of workers per resource assigned using the --workers-per-resource option. For example, if you have 4 available GPUs, you set the value as one divided by the number as only one instance of the Runner server will be spawned.

TRUST_REMOTE_CODE=True openllm start microsoft/phi-2 --workers-per-resource 0.25

When using the --workers-per-resource option with the openllm build command, the environment variable is saved into the resulting Bento.

For more information, see Resource scheduling strategy.

Different LLMs may support multiple runtime implementations. Models that have vLLM (vllm) supports will use vLLM by default, otherwise it fallback to use PyTorch (pt).

To specify a specific runtime for your chosen model, use the --backend option. For example:

openllm start meta-llama/Llama-2-7b-chat-hf --backend vllm

Note:

1. To use the vLLM backend, you need a GPU with at least the Ampere architecture or newer and CUDA version 11.8.
2. To see the backend options of each model supported by OpenLLM, see the Supported models section or run openllm models.

Quantization is a technique to reduce the storage and computation requirements for machine learning models, particularly during inference. By approximating floating-point numbers as integers (quantized values), quantization allows for faster computations, reduced memory footprint, and can make it feasible to deploy large models on resource-constrained devices.

OpenLLM supports the following quantization techniques

• LLM.int8(): 8-bit Matrix Multiplication through bitsandbytes
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression through bitsandbytes
• AWQ: Activation-aware Weight Quantization,
• GPTQ: Accurate Post-Training Quantization
• SqueezeLLM: Dense-and-Sparse Quantization.

With PyTorch backend, OpenLLM supports int8, int4, and gptq.

For using int8 and int4 quantization through bitsandbytes, you can use the following command:

TRUST_REMOTE_CODE=True openllm start microsoft/phi-2 --quantize int8

To run inference with gptq, simply pass --quantize gptq:

openllm start TheBloke/Llama-2-7B-Chat-GPTQ --quantize gptq

With vLLM backend, OpenLLM supports awq, squeezellm

To run inference with awq, simply pass --quantize awq:

openllm start TheBloke/zephyr-7B-alpha-AWQ --quantize awq

To run inference with squeezellm, simply pass --quantize squeezellm:

openllm start squeeze-ai-lab/sq-llama-2-7b-w4-s0 --quantize squeezellm --serialization legacy

PEFT, or Parameter-Efficient Fine-Tuning, is a methodology designed to fine-tune pre-trained models more efficiently. Instead of adjusting all model parameters, PEFT focuses on tuning only a subset, reducing computational and storage costs. LoRA (Low-Rank Adaptation) is one of the techniques supported by PEFT. It streamlines fine-tuning by using low-rank decomposition to represent weight updates, thereby drastically reducing the number of trainable parameters.

With OpenLLM, you can take advantage of the fine-tuning feature by serving models with any PEFT-compatible layers using the --adapter-id option. For example:

openllm start facebook/opt-6.7b --adapter-id aarnphm/opt-6-7b-quotes:default

OpenLLM also provides flexibility by supporting adapters from custom file paths:

openllm start facebook/opt-6.7b --adapter-id /path/to/adapters:local_adapter

To use multiple adapters, use the following format:

openllm start facebook/opt-6.7b --adapter-id aarnphm/opt-6.7b-lora:default --adapter-id aarnphm/opt-6.7b-french:french_lora

By default, all adapters will be injected into the models during startup. Adapters can be specified per request via adapter_name:

curl -X 'POST' \
  'http://localhost:3000/v1/generate' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "What is the meaning of life?",
  "stop": [
    "philosopher"
  ],
  "llm_config": {
    "max_new_tokens": 256,
    "temperature": 0.75,
    "top_k": 15,
    "top_p": 1
  },
  "adapter_name": "default"
}'

To include this into the Bento, you can specify the --adapter-id option when using the openllm build command:

openllm build facebook/opt-6.7b --adapter-id ...

If you use a relative path for --adapter-id, you need to add --build-ctx.

openllm build facebook/opt-6.7b --adapter-id ./path/to/adapter_id --build-ctx .

OpenLLM is not just a standalone product; it's a building block designed to integrate with other powerful tools easily. We currently offer integration with OpenAI's Compatible Endpoints, LlamaIndex, LangChain, and Transformers Agents.

OpenLLM Server can be used as a drop-in replacement for OpenAI's API. Simply specify the base_url to llm-endpoint/v1 and you are good to go:

import openai

client = openai.OpenAI(
  base_url='http://localhost:3000/v1', api_key='na'
)  # Here the server is running on localhost:3000

completions = client.completions.create(
  prompt='Write me a tag line for an ice cream shop.', model=model, max_tokens=64, stream=stream
)

The compatible endpoints supports /completions, /chat/completions, and /models

To start a local LLM with llama_index, simply use llama_index.llms.openllm.OpenLLM:

import asyncio
from llama_index.llms.openllm import OpenLLM

llm = OpenLLM('HuggingFaceH4/zephyr-7b-alpha')

llm.complete('The meaning of life is')


async def main(prompt, **kwargs):
  async for it in llm.astream_chat(prompt, **kwargs):
    print(it)


asyncio.run(main('The time at San Francisco is'))

If there is a remote LLM Server running elsewhere, then you can use llama_index.llms.openllm.OpenLLMAPI:

from llama_index.llms.openllm import OpenLLMAPI

Spin up an OpenLLM server, and connect to it by specifying its URL:

from langchain.llms import OpenLLM

llm = OpenLLM(server_url='http://44.23.123.1:3000', server_type='http')
llm('What is the difference between a duck and a goose? And why there are so many Goose in Canada?')

OpenLLM seamlessly integrates with Transformers Agents.

import transformers

agent = transformers.HfAgent('http://localhost:3000/hf/agent')  # URL that runs the OpenLLM server

agent.run('Is the following `text` positive or negative?', text="I don't like how this models is generate inputs")

There are several ways to deploy your LLMs:

1. Building a Bento: With OpenLLM, you can easily build a Bento for a specific model, like mistralai/Mistral-7B-Instruct-v0.1, using the build command.:

   openllm build mistralai/Mistral-7B-Instruct-v0.1

   A Bento, in BentoML, is the unit of distribution. It packages your program's source code, models, files, artefacts, and dependencies.

2. Containerize your Bento

   bentoml containerize <name:version>

   This generates a OCI-compatible docker image that can be deployed anywhere docker runs. For best scalability and reliability of your LLM service in production, we recommend deploy with BentoCloud。

Deploy OpenLLM with BentoCloud, the inference platform for fast moving AI teams.

1. Create a BentoCloud account: sign up here

2. Log into your BentoCloud account:

   bentoml cloud login --api-token <your-api-token> --endpoint <bento-cloud-endpoint>

3. Bulding a Bento: With OpenLLM, you can easily build a Bento for a specific model, such as mistralai/Mistral-7B-Instruct-v0.1:

   openllm build mistralai/Mistral-7B-Instruct-v0.1

4. Pushing a Bento: Push your freshly-built Bento service to BentoCloud via the push command:

   bentoml push <name:version>

5. Deploying a Bento: Deploy your LLMs to BentoCloud with a single bentoml deployment create command following the deployment instructions.

Engage with like-minded individuals passionate about LLMs, AI, and more on our Discord!

OpenLLM is actively maintained by the BentoML team. Feel free to reach out and join us in our pursuit to make LLMs more accessible and easy to use 👉 Join our Slack community!

We welcome contributions! If you're interested in enhancing OpenLLM's capabilities or have any questions, don't hesitate to reach out in our discord channel.

Checkout our Developer Guide if you wish to contribute to OpenLLM's codebase.

OpenLLM collects usage data to enhance user experience and improve the product. We only report OpenLLM's internal API calls and ensure maximum privacy by excluding sensitive information. We will never collect user code, model data, or stack traces. For usage tracking, check out the code.

You can opt out of usage tracking by using the --do-not-track CLI option:

openllm [command] --do-not-track

Or by setting the environment variable OPENLLM_DO_NOT_TRACK=True:

export OPENLLM_DO_NOT_TRACK=True

If you use OpenLLM in your research, we provide a citation to use:

@software{Pham_OpenLLM_Operating_LLMs_2023,
author = {Pham, Aaron and Yang, Chaoyu and Sheng, Sean and  Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost},
license = {Apache-2.0},
month = jun,
title = {{OpenLLM: Operating LLMs in production}},
url = {https://github.com/bentoml/OpenLLM},
year = {2023}
}