Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(breaking): unify LLM API #283

Merged
merged 9 commits into from
Sep 1, 2023
Merged

refactor(breaking): unify LLM API #283

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
03c402e
refactor: initial work to _gen
aarnphm Aug 30, 2023
939a1dc
chore: remove bettertransformer
aarnphm Aug 30, 2023
14dda15
merge: branch 'main' of github.com:bentoml/OpenLLM into refactor/unif…
aarnphm Aug 30, 2023
9d4ec15
fix: run format
aarnphm Aug 30, 2023
f224e70
fix: rename backend and cleanup runtime [wip]
aarnphm Aug 31, 2023
025633a
refactor: update naming and envvar
aarnphm Sep 1, 2023
973c29f
chore: add breaking change notes
aarnphm Sep 1, 2023
e292fe0
chore: update overload types
aarnphm Sep 1, 2023
20a9c0e
Merge branch 'main' into refactor/unify-llm-impl
aarnphm Sep 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore: add breaking change notes 
Signed-off-by: aarnphm-ec2-dev <29749331+aarnphm@users.noreply.github.com>
  • Loading branch information
@aarnphm
aarnphm committed Sep 1, 2023
commit 973c29f0826e50b9527ab2096bc89775d733f99d
20 changes: 20 additions & 0 deletions changelog.d/283.breaking.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
All environment variable now will be more simplified, without the need for the specific model prefix

For example: OPENLLM_LLAMA_GENERATION_MAX_NEW_TOKENS now becomes OPENLLM_GENERATION_MAX_NEW_TOKENS

Unify some misc environment variable. To switch different backend, one can use `--backend` for both `start` and `build`

```bash
openllm start llama --backend vllm
```

or the environment variable `OPENLLM_BACKEND`

```bash
OPENLLM_BACKEND=vllm openllm start llama
```

`openllm.Runner` now will default to try download the model the first time if the model is not available, and get the cached in model store consequently

Model serialisation now updated to a new API version with more clear name change, kindly ask users to do `openllm prune -y --include-bentos` and update to
this current version of openllm