Code

Upstream merge oct31#41

Code

Merged

masahi merged 75 commits intooctoml:batch-servingoctoml/mlc-llm:batch-servingfrom masahi:upstream-merge-oct31masahi/mlc-llm:upstream-merge-oct31Copy head branch name to clipboardOct 30, 2023

+8,943-2,078

Commits

Commits on Oct 8, 2023

• [API] Add GenerationConfig (mlc-ai#1024)

  davidpissarra

  authored
  898db76

  Copy full SHA for 898db76

• Fix two bugs in kv-cache backtrack loop (mlc-ai#856)

  Show description for ad3a6b9
  shenberg

  authored
  ad3a6b9

  Copy full SHA for ad3a6b9

• [Build] Added --pdb flag to build.py, drop into pdb on error (mlc-ai#1017)

  Show description for 6e40c21
  Lunderberg

  authored
  6e40c21

  Copy full SHA for 6e40c21

• [Android] Use AlertDialog instead of Toast (mlc-ai#1039)

  cyx-6

  authored
  bae37b3

  Copy full SHA for bae37b3

Commits on Oct 9, 2023

• Add doc for ChatConfig, ConvConfig, GenerationConfig, BuildArgs (mlc-ai#1040)

  Show description for b44f679
  CharlieFRuan

  authored
  b44f679

  Copy full SHA for b44f679

• [Android] Add Llama2 q4f16_0 (mlc-ai#1041)

  Show description for 3a9849a
  spectrometerHBH

  authored
  3a9849a

  Copy full SHA for 3a9849a

• [Docs] Model prebuilts tracking page revamp (mlc-ai#1000)

  CharlieFRuan

  authored
  bed9e60

  Copy full SHA for bed9e60

• Update compile_models.rst (mlc-ai#1038)

  Show description for c02fdaf
  yongjer

  authored
  c02fdaf

  Copy full SHA for c02fdaf

• Support for the Stable LM 3B model (mlc-ai#1008)

  Show description for 85001ed
  jeethu

  authored
  85001ed

  Copy full SHA for 85001ed

• [Docs] Iterate model prebuilts docs (mlc-ai#1043)

  Show description for a032d40
  CharlieFRuan

  authored
  a032d40

  Copy full SHA for a032d40

• Update README.md

  junrushao

  authored
  a58605f

  Copy full SHA for a58605f

• [CPP] Separate common utils out from llm_chat.cc (mlc-ai#1044)

  Show description for bdd9d9b
  MasterJH5574

  authored
  bdd9d9b

  Copy full SHA for bdd9d9b

• Update README.md (mlc-ai#1045)

  Show description for 20131fb
  junrushao

  authored
  20131fb

  Copy full SHA for 20131fb

Commits on Oct 11, 2023

• add verbose stats to mlc-chat REST API (mlc-ai#1049)

  Show description for 1e6fb11
  denise-k

  authored
  1e6fb11

  Copy full SHA for 1e6fb11

Commits on Oct 12, 2023

• [Transform] Apply split_rotary optimization on prefill (mlc-ai#1033)

  Show description for b9179cf
  Lunderberg

  authored
  b9179cf

  Copy full SHA for b9179cf

• [Docs] Add mlc.ai/package to DEPENDENCY INSTALLATION group (mlc-ai#1055)

  Show description for 98ebd28
  

  LeshengJin

  and

  junrushao

  authored
  98ebd28

  Copy full SHA for 98ebd28

• Revert "[Transform] Apply split_rotary optimization on prefill (mlc-ai#1033)" (mlc-ai#1058)

  Show description for bfaa5b9
  MasterJH5574

  authored
  bfaa5b9

  Copy full SHA for bfaa5b9

Commits on Oct 13, 2023

• [BugFix] Set the right max_sequence_length for both Llama-1 and Llama-2 families (mlc-ai#1032)

  Show description for ca8c11b
  

  sunggg

  and

  “Sunghyun

  authored
  ca8c11b

  Copy full SHA for ca8c11b

• [Doc] Use -U instead of --force-reinstall (mlc-ai#1062)

  Show description for edab9b5
  junrushao

  authored
  edab9b5

  Copy full SHA for edab9b5

Commits on Oct 14, 2023

• [Model] Initial batching support for Llama (mlc-ai#1048)

  Show description for d854105
  MasterJH5574

  authored
  d854105

  Copy full SHA for d854105

• Fix Stable LM 3B build (mlc-ai#1061)

  Show description for c2b8cbc
  jeethu

  authored
  c2b8cbc

  Copy full SHA for c2b8cbc

• [Core] Remove duplication in MODEL.get_model calls (mlc-ai#1054)

  Show description for 481cd92
  Lunderberg

  authored
  481cd92

  Copy full SHA for 481cd92

• [ParamManager] Cleanup creation of quantization IRModule (mlc-ai#1053)

  Show description for 8184431
  Lunderberg

  authored
  8184431

  Copy full SHA for 8184431

Commits on Oct 15, 2023

• Minor typo fix (mlc-ai#1064)

  jeethu

  authored
  9010d48

  Copy full SHA for 9010d48

• Add links to Python API Reference (mlc-ai#1068)

  junrushao

  authored
  b0bfc88

  Copy full SHA for b0bfc88

• [Fix] ChatModule incorrect temperature buffer shape (mlc-ai#1070)

  Show description for 204860b
  MasterJH5574

  authored
  204860b

  Copy full SHA for 204860b

Commits on Oct 16, 2023

• [ParamManager] Added progress bar for get_item/set_item (mlc-ai#1063)

  Lunderberg

  authored
  d202077

  Copy full SHA for d202077

• [Python] Extract common device str parse function in ChatModule (mlc-ai#1074)

  Show description for 9872c48
  MasterJH5574

  authored
  9872c48

  Copy full SHA for 9872c48

Commits on Oct 17, 2023

• [Bugfix] Compilation Error in q4f32_1 (mlc-ai#1078)

  Show description for 3aefd9f
  junrushao

  authored
  3aefd9f

  Copy full SHA for 3aefd9f

Commits on Oct 19, 2023

• Establish mlc_chat.compiler (mlc-ai#1082)

  Show description for 2625945
  junrushao

  authored
  2625945

  Copy full SHA for 2625945

• Update README.md for Multi-GPU (mlc-ai#1090)

  junrushao

  authored
  56a8004

  Copy full SHA for 56a8004

• Support lib_path override in C++. Improvements on docs and error messages (mlc-ai#1086)

  Show description for b0373d1
  rickzx

  authored
  b0373d1

  Copy full SHA for b0373d1

• StreamIterator (mlc-ai#1057)

  Show description for 830656f
  

  varshith15

  and

  Varshith

  authored
  830656f

  Copy full SHA for 830656f

• Update benchmark.py according to mlc-ai#1086 (mlc-ai#1091)

  Show description for 9bf5723
  junrushao

  authored
  9bf5723

  Copy full SHA for 9bf5723

Commits on Oct 20, 2023

• Disable Disco for q4f16_ft and q8f16_ft quantization (mlc-ai#1094)

  LeshengJin

  authored
  62d0c03

  Copy full SHA for 62d0c03

• [Format] Apply isort and black for python/ (mlc-ai#1097)

  Show description for cf39bf6
  junrushao

  authored
  cf39bf6

  Copy full SHA for cf39bf6

Commits on Oct 21, 2023

• More formatting (mlc-ai#1099)

  junrushao

  authored
  e9b85ce

  Copy full SHA for e9b85ce

• Enable Python Linter (mlc-ai#1098)

  Show description for 03c641a
  junrushao

  authored
  03c641a

  Copy full SHA for 03c641a

• Add Basic Pylint and Mypy Tooling (mlc-ai#1100)

  Show description for 46d11e6
  junrushao

  authored
  46d11e6

  Copy full SHA for 46d11e6

Commits on Oct 22, 2023

• [CI] Add clang-format (mlc-ai#1103)

  junrushao

  authored
  6159cc4

  Copy full SHA for 6159cc4

Commits on Oct 23, 2023

• [Slim-LM] Smart path finding for config and weight (mlc-ai#1088)

  LeshengJin

  authored
  16dd2ae

  Copy full SHA for 16dd2ae

• [Transform] Provide IRModule transform for rewrite_attention (mlc-ai#1052)

  Show description for f57c9c9
  Lunderberg

  authored
  f57c9c9

  Copy full SHA for f57c9c9

• [ParamManager] Use BundleModelParams for transform_dequantize (mlc-ai#1056)

  Show description for e5927ce
  Lunderberg

  authored
  e5927ce

  Copy full SHA for e5927ce

• [Slim-LM] Introduce HFLoad for loading Pytorch and SafeTensor weights (mlc-ai#1113)

  LeshengJin

  authored
  7ae8c6d

  Copy full SHA for 7ae8c6d

Commits on Oct 24, 2023

• [WINDOWS] reduce noise in windows build (mlc-ai#1115)

  tqchen

  authored
  5a7dcd8

  Copy full SHA for 5a7dcd8

• Add CLI commands for compilation (mlc-ai#1109)

  junrushao

  authored
  61179a0

  Copy full SHA for 61179a0

• Auto updated submodule references

  

  Git bot

  committed
  8ce7793

  Copy full SHA for 8ce7793

• fix mismatched argument name (mlc-ai#1117)

  Show description for 488017d
  Sing-Li

  authored
  488017d

  Copy full SHA for 488017d

• [Docs] Add doc for max and mean gen len, shift factor; and buildArgs (mlc-ai#1119)

  Show description for 206103b
  CharlieFRuan

  authored
  206103b

  Copy full SHA for 206103b

• Revert "[ParamManager] Use BundleModelParams for transform_dequantize" (mlc-ai#1120)

  Show description for 2aa6809
  junrushao

  authored
  2aa6809

  Copy full SHA for 2aa6809

• Remove inaccurate warning message (mlc-ai#1121)

  Show description for 9cb8e8e
  junrushao

  authored
  9cb8e8e

  Copy full SHA for 9cb8e8e

• [REST] OpenAI compatible Rest API (mlc-ai#1107)

  Show description for 9166edb
  Kartik14

  authored
  9166edb

  Copy full SHA for 9166edb

Commits on Oct 25, 2023

• Add --opt flag parsing to CLI (mlc-ai#1123)

  junrushao

  authored
  a4279e3

  Copy full SHA for a4279e3

• [ParamManager][Redo] Use BundleModelParams for transform_dequantize (mlc-ai#1127)

  Show description for 973f9fc
  Lunderberg

  authored
  973f9fc

  Copy full SHA for 973f9fc

Commits on Oct 27, 2023

• added details to windows installation (mlc-ai#1133)

  Show description for 24f795e
  goutham2688

  authored
  24f795e

  Copy full SHA for 24f795e

Commits on Oct 28, 2023

• Grammatical and Typographical improvements (mlc-ai#1139)

  Show description for 2c492e5
  tmsagarofficial

  authored
  2c492e5

  Copy full SHA for 2c492e5

• Minor enhancements to ChatModule (mlc-ai#1132)

  Show description for 2ec0cc8
  YuchenJin

  authored
  2ec0cc8

  Copy full SHA for 2ec0cc8

Commits on Oct 29, 2023

• Updating tvm install docs (mlc-ai#1143)

  Show description for 27ac5ac
  David-Sharma

  authored
  27ac5ac

  Copy full SHA for 27ac5ac

• Make the help info consistent with program name (mlc-ai#1137)

  Show description for 2b6d832
  fennecJ

  authored
  2b6d832

  Copy full SHA for 2b6d832

• Support parameter packing (mlc-ai#1146)

  junrushao

  authored
  878ae84

  Copy full SHA for 878ae84

• [Slim-LM] Enable Group Quant (mlc-ai#1129)

  Show description for c0c3a8d
  zxybazh

  authored
  c0c3a8d

  Copy full SHA for c0c3a8d

• Enable Mypy and Pylint in mlc_chat Python Package (mlc-ai#1149)

  junrushao

  authored
  2193767

  Copy full SHA for 2193767

Commits on Oct 30, 2023

• Migrate Compiler Passes (mlc-ai#1150)

  junrushao

  authored
  0a25374

  Copy full SHA for 0a25374

• Compile Model Preset without External config.json (mlc-ai#1151)

  Show description for 1a79a53
  junrushao

  authored
  1a79a53

  Copy full SHA for 1a79a53

• Update attention layer (mlc-ai#1153)

  Show description for ba67835
  junrushao

  authored
  ba67835

  Copy full SHA for ba67835

• Add batched Llama model definition using vLLM paged attention (mlc-ai#1134)

  Show description for fee2cb5
  masahi

  authored
  fee2cb5

  Copy full SHA for fee2cb5

• [Transform][Redo] Apply split_rotary optimization on prefill (mlc-ai#1125)

  Show description for ece97b1
  Lunderberg

  authored
  ece97b1

  Copy full SHA for ece97b1

• Apply rewrite for normal attention and MQA (mlc-ai#1138)

  Show description for b190578
  Lunderberg

  authored
  b190578

  Copy full SHA for b190578

• [Rest] Fix emoji handling in Rest API. (mlc-ai#1142)

  YuchenJin

  authored
  8ca0176

  Copy full SHA for 8ca0176

• [Utility] Check for isinstance(exc, Exception) before entering pdb (mlc-ai#1095)

  Show description for 3cf5605
  Lunderberg

  authored
  3cf5605

  Copy full SHA for 3cf5605

• [Utils] Remove conversion to numpy array in utils.save_params (mlc-ai#1083)

  Show description for 0a9d6c7
  Lunderberg

  authored
  0a9d6c7

  Copy full SHA for 0a9d6c7

• [Fix][REST] Use lowered-cased "app" (mlc-ai#1159)

  junrushao

  authored
  425a2cb

  Copy full SHA for 425a2cb

• sync llama def

  masahi

  committed
  ed10571

  Copy full SHA for ed10571

• Merge remote-tracking branch 'mlc-ai/main' into upstream-merge-oct31

  masahi

  committed
  bab2e50

  Copy full SHA for bab2e50

• fi

  masahi

  committed
  2252c66

  Copy full SHA for 2252c66