Skip to content

Commit

Permalink
Update XGrammar Blog with SGLang integration information (#15)
Browse files Browse the repository at this point in the history
  • Loading branch information
Ubospica authored Nov 26, 2024
1 parent 9cb380f commit ab6316c
Showing 1 changed file with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,9 @@ We also provide ready-to-use Python and TypeScript libraries.
It already powers many LLM frameworks:

- [MLC-LLM](https://github.com/mlc-ai/mlc-llm) leveraged the C++ backend to build cross-platform support for high-performance structured generation, enabling support on a diverse set of server, laptop, and edge platforms.
- [SGLang](https://github.com/sgl-project/sglang) integrated the Python library and showed a significant reduction of JSON Schema generation overhead compared to its previous backend.
- [SGLang](https://github.com/sgl-project/sglang) integrated the Python library and showed a significant reduction of JSON Schema generation overhead compared to its previous backend. To use it:
- Add `--grammar-backend xgrammar` when you launch the server
- Query it with OpenAI-compatible API <https://sgl-project.github.io/backend/openai_api_completions.html#JSON>
- [WebLLM](https://webllm.mlc.ai/) integrated the typescript library and enabled in-browser local structured generation. You can check out a demo in [WebLLM Structured Generation Playground](https://huggingface.co/spaces/mlc-ai/WebLLM-Structured-Generation-Playground).
- We are working with the [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) team to bring first-class XGrammar support, which will soon be integrated into the preview main branch.
- We are working with the [vLLM](https://github.com/vllm-project/vllm) team to enable first-class XGrammar integration in vLLM.
Expand Down

0 comments on commit ab6316c

Please sign in to comment.