Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate distributed inference with chat/server #1381

Merged
merged 20 commits into from
Dec 19, 2024
Merged

Integrate distributed inference with chat/server #1381

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
7b4d5c5
Integrate distributed inference without introducing abstraction
mreso Nov 16, 2024
e7670c3
Cleanup old distributed inference integration
mreso Nov 16, 2024
08a8e03
Merge branch 'main' into refactor/distributed_inference_without_abstr…
mreso Nov 16, 2024
d5bca9b
Read distribution from model_config
mreso Nov 18, 2024
76895cc
Declare distribution_path if args.model is not given
mreso Nov 18, 2024
3ef1296
Address some nits from PR review
mreso Nov 19, 2024
04cdfd0
Merge branch 'main' into refactor/distributed_inference_without_abstr…
mreso Dec 2, 2024
924a096
Merge branch 'main' into refactor/distributed_inference_without_abstr…
mreso Dec 4, 2024
99c33e8
Merge branch 'main' into refactor/distributed_inference_without_abstr…
Jack-Khuu Dec 11, 2024
773894f
Merge remote-tracking branch 'origin/main' into refactor/distributed_…
mreso Dec 16, 2024
7cb98c9
Added comment on model size all reduce + type hint
mreso Dec 16, 2024
10fb55a
Apply suggestions from code review
mreso Dec 16, 2024
28d7836
Make sure speculative decoding is disable for pp >1 and remark this i…
mreso Dec 17, 2024
68eec0b
Refactor conditions in pp
mreso Dec 17, 2024
3ad31e8
Rename and alter signature of setup_env to reflect that it also runs …
mreso Dec 17, 2024
e07b03d
Rename setup_env in server + fix condition
mreso Dec 17, 2024
daf902c
Merge branch 'main' into refactor/distributed_inference_without_abstr…
Jack-Khuu Dec 19, 2024
db5fd1b
Merge branch 'main' into refactor/distributed_inference_without_abstr…
Jack-Khuu Dec 19, 2024
7ac16f9
Update generate.py
Jack-Khuu Dec 19, 2024
7650153
Add default value to add_generation_prompt to preserve bc
mreso Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Rename setup_env in server + fix condition
  • Loading branch information
@mreso
mreso committed Dec 17, 2024
commit e07b03d6837b01a9124ae229e7b3b5a693f3ad28
5 changes: 2 additions & 3 deletions torchchat/usages/server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from flask import Flask, request, Response

from torchchat.cli.builder import BuilderArgs, TokenizerArgs
from torchchat.distributed.utils import setup_env
from torchchat.distributed.utils import run_in_dist_env
from torchchat.generate import GeneratorArgs

from torchchat.usages.openai_api import (
Expand Down Expand Up @@ -73,11 +73,10 @@ def create_app(args): # noqa: C901
mp_context = mp.get_context('spawn')
queue = mp_context.Queue()

mreso marked this conversation as resolved.
Show resolved Hide resolved
if builder_args.distributed:
for i in range(1, world_size):
fn = partial(run_worker, args, i, queue)
mp_context = mp.get_context('spawn')
procs.append(mp_context.Process(target=setup_env, args=(world_size, i, fn)))
procs.append(mp_context.Process(target=run_in_dist_env, args=(world_size, i, fn)))
procs[-1].start()

environ["MASTER_ADDR"] = "localhost"
Expand Down
Loading