Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Grammar] Integrate with XGrammar #635

Merged
merged 10 commits into from
Nov 22, 2024
Merged

[Grammar] Integrate with XGrammar #635

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
8d530cd
[Grammar] Integrate with web-xgrammar
CharlieFRuan Oct 7, 2024
2715813
Rebase to new xgrammar, use tokenizerInfo and add full bitmask size
CharlieFRuan Oct 13, 2024
308660a
Allow grammar response format, on top of schema
CharlieFRuan Oct 14, 2024
8a987f3
Add metrics for grammar overhead, overlap grammar init
CharlieFRuan Oct 14, 2024
81ce64c
Trivial
CharlieFRuan Oct 14, 2024
deb5a26
Add TTFT and TPOT to usage extra
CharlieFRuan Oct 15, 2024
6480f52
Trivial, use root instead of main in ebnf
CharlieFRuan Oct 25, 2024
a63a534
Add metric E2E latency to usage extra
CharlieFRuan Oct 25, 2024
6e23f5d
Support ignore_eos
CharlieFRuan Oct 28, 2024
2fed3e8
Update to new XGrammar pre-release APIs
CharlieFRuan Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[Grammar] Integrate with web-xgrammar
  • Loading branch information
@CharlieFRuan
CharlieFRuan committed Nov 22, 2024
commit 8d530cd00c5b136cd8b5c7237f16e8e0808bceef
5 changes: 4 additions & 1 deletion examples/json-mode/src/json_mode.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@ async function main() {
const initProgressCallback = (report: webllm.InitProgressReport) => {
setLabel("init-label", report.text);
};
const selectedModel = "Llama-3.1-8B-Instruct-q4f32_1-MLC";
// Pick any one of these models to start trying -- most models in WebLLM support grammar
const selectedModel = "Llama-3.2-3B-Instruct-q4f16_1-MLC";
// const selectedModel = "Qwen2.5-1.5B-Instruct-q4f16_1-MLC";
// const selectedModel = "Phi-3.5-mini-instruct-q4f16_1-MLC";
const engine: webllm.MLCEngineInterface = await webllm.CreateMLCEngine(
selectedModel,
{ initProgressCallback: initProgressCallback },
Expand Down
18 changes: 14 additions & 4 deletions examples/json-schema/src/json_schema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,13 @@ async function simpleStructuredTextExample() {
const initProgressCallback = (report: webllm.InitProgressReport) => {
setLabel("init-label", report.text);
};

// Pick any one of these models to start trying -- most models in WebLLM support grammar
// const selectedModel = "Llama-3.2-3B-Instruct-q4f16_1-MLC";
// const selectedModel = "Qwen2.5-1.5B-Instruct-q4f16_1-MLC";
const selectedModel = "Phi-3.5-mini-instruct-q4f16_1-MLC";
const engine: webllm.MLCEngineInterface = await webllm.CreateMLCEngine(
"Llama-3.1-8B-Instruct-q4f16_1-MLC",
selectedModel,
{ initProgressCallback: initProgressCallback },
);

Expand Down Expand Up @@ -106,8 +111,13 @@ async function harryPotterExample() {
setLabel("init-label", report.text);
};

// Pick any one of these models to start trying -- most models in WebLLM support grammar
// const selectedModel = "Llama-3.2-3B-Instruct-q4f16_1-MLC";
// const selectedModel = "Qwen2.5-1.5B-Instruct-q4f16_1-MLC";
const selectedModel = "Phi-3.5-mini-instruct-q4f16_1-MLC";

const engine: webllm.MLCEngineInterface = await webllm.CreateMLCEngine(
"Llama-3.1-8B-Instruct-q4f16_1-MLC",
selectedModel,
{ initProgressCallback: initProgressCallback },
);

Expand Down Expand Up @@ -216,8 +226,8 @@ async function functionCallingExample() {

async function main() {
// await simpleStructuredTextExample();
// await harryPotterExample();
await functionCallingExample();
await harryPotterExample();
// await functionCallingExample();
}

main();
28 changes: 28 additions & 0 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
"ts-jest": "^29.1.2",
"tslib": "^2.3.1",
"@mlc-ai/web-runtime": "0.18.0-dev2",
"@mlc-ai/web-xgrammar": "../xgrammar/web",
"typescript": "^4.9.5"
},
"dependencies": {
Expand Down
203 changes: 0 additions & 203 deletions src/grammar.ts
View file
Open in desktop

This file was deleted.

Loading