Add Intel XPU device support to generate and serve #1361

jenniew · 2024-11-09T06:27:45Z

Add XPU device support exclude distributed mode, workflow, documentation.

pytorch-bot · 2024-11-09T06:27:48Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1361

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit 8edfeb2 with merge base ef82697 ():

NEW FAILURE - The following job has failed:

pull / test-build-runner-et-android / linux-job (gh)
RuntimeError: Command docker exec -t 026763def57a020a2a1909d507c220f807344e7972a7a05d77c574d19fa3a0c0 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / test-tinystories-executorch (16-core-ubuntu) (gh) (trunk failure)
##[error]The operation was canceled.
pull / test-tinystories-executorch (macos-14-xlarge) (gh) (trunk failure)
RuntimeError: Failed to load ET compiled /Users/runner/work/torchchat/torchchat/stories15M.pte

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2024-11-09T06:27:50Z

Hi @jenniew!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

…device

mikekgfb · 2024-11-09T18:13:57Z

Is there a way to run at least a few simple tests on an xpu to avoid inadvertent breakage?

jenniew · 2024-11-12T07:26:10Z

Is there a way to run at least a few simple tests on an xpu to avoid inadvertent breakage?

For generate, run a simple test by python3 torchchat.py generate llama3.1 --prompt "write me a story about a boy and his bear --device xpu
For serve, run a simple test by python3 torchchat.py server llama3.1 --device xpu

Jack-Khuu · 2024-11-12T09:18:30Z

install/install_requirements.sh

-PYTORCH_NIGHTLY_VERSION=dev20241002
+if [[ -x "$(command -v xpu-smi)" ]];
+then
+  PYTORCH_NIGHTLY_VERSION=dev20241001


Why does xpu need an older PYTORCH_NIGHTLY?

when install torch==2.6.0.dev20241002 and torchvision==0.20.0.dev20241002+xpu, it will get error:
ERROR: Cannot install torch==2.6.0.dev20241002 and torchvision==0.20.0.dev20241002+xpu because these package versions have conflicting dependencies.

The conflict is caused by:
The user requested torch==2.6.0.dev20241002
torchvision 0.20.0.dev20241002+xpu depends on torch==2.6.0.dev20241001

So for xpu, I changed the torch nightly version to dev20241001

Let me see if I can get you a fresher version on XPU, the torch/vision discrepancy shouldn't be a normal thing

Jack-Khuu · 2024-11-12T09:19:06Z

install/install_requirements.sh

+  REQUIREMENTS_TO_INSTALL=(
+    torch=="2.6.0.${PYTORCH_NIGHTLY_VERSION}"
+    torchvision=="0.20.0.${VISION_NIGHTLY_VERSION}"
+    torchtune=="0.3.1"


Context on the varying tune version?

On xpu nightly URL, it does not have nightly version of torchtune, so just install 0.3.1 release for xpu environment.

Hmm we should add support for nightly, let me ping some torchtune folk

cc: @ebsmothers

Jack-Khuu · 2024-11-12T09:26:07Z

Welcome to torchchat and thanks for adding @jenniew!!

Super stoked to see that it didn't require much lift to get XPU set up. Added a few questions on the versioning difference.
I'll try to get some folks to test and see if we can get some recurring jobs set up.

What device did you test on btw?

Tagging a few folk who I'm trying to help taking a larger role

Jack-Khuu · 2024-11-12T22:18:16Z

@jenniew Also do you mind filling out the CLA? It'll allow you to contribute to Meta repos

jenniew · 2024-11-13T20:07:10Z

@jenniew Also do you mind filling out the CLA? It'll allow you to contribute to Meta repos

Yes, I just signed the CLA

…o xpu_device

jenniew · 2024-11-13T23:09:00Z

Welcome to torchchat and thanks for adding @jenniew!!

Super stoked to see that it didn't require much lift to get XPU set up. Added a few questions on the versioning difference. I'll try to get some folks to test and see if we can get some recurring jobs set up.

What device did you test on btw?

Tagging a few folk who I'm trying to help taking a larger role

I tested on Intel Data Center GPU Max 1100.

Jack-Khuu · 2024-11-15T20:01:48Z

Just an update:

I'm looking into getting a PT pin bumped Multi Pin Bumps across PT/AO/tune/ET #1367, which should allow XPU to use a fresher version
https://github.com/pytorch/torchtune just release 0.4.0, which I'm asking someone to bump torchchat to so XPU can have an updated version over there as well

mikekgfb · 2024-12-10T05:22:15Z

Would be nice to run as a test as well. Could be as easy as enabling and adding a runner for xpu to test-readme-pr.yml if it is available. Alternatively, may require a copy of that file because the test spec has several target-runner related specs, with updates to all fields for values representative of xpu environment:

      runner: linux.g5.4xlarge.nvidia.gpu
      gpu-arch-type: cuda
      gpu-arch-version: "12.1"

…device

Jack-Khuu · 2025-01-15T19:30:41Z

install/install_requirements.sh

+  REQUIREMENTS_TO_INSTALL=(
+    torch=="2.6.0.${PYTORCH_NIGHTLY_VERSION}"
+    torchvision=="0.22.0.${VISION_NIGHTLY_VERSION}"
+    torchtune=="0.4.0"


Hi @songhappy, are we able to bump this to 0.5.0?

I also noticed that over in tune pytorch/torchtune#1280, another intel eng (Ankur-singh) is checking in on nightly stsatus

Hi @Jack-Khuu , I have updated torchtune version to 0.5.0. Update to now, nightly version is still unavailable. Can you review this PR and check if it can be merged. If the torchtune nightly on XPU is ready, we can update the installation later in a new PR.

…device

jenniew · 2025-01-16T22:58:47Z

Would be nice to run as a test as well. Could be as easy as enabling and adding a runner for xpu to test-readme-pr.yml if it is available. Alternatively, may require a copy of that file because the test spec has several target-runner related specs, with updates to all fields for values representative of xpu environment:
      runner: linux.g5.4xlarge.nvidia.gpu
      gpu-arch-type: cuda
      gpu-arch-version: "12.1"

@mikekgfb , yes, we need to add xpu test as well. Since the test validation may take a few efforts, I think we may do it in a separate PR after this PR is merged.

jenniew · 2025-02-03T20:31:32Z

Would be nice to run as a test as well. Could be as easy as enabling and adding a runner for xpu to test-readme-pr.yml if it is available. Alternatively, may require a copy of that file because the test spec has several target-runner related specs, with updates to all fields for values representative of xpu environment:
      runner: linux.g5.4xlarge.nvidia.gpu
      gpu-arch-type: cuda
      gpu-arch-version: "12.1"
@mikekgfb , yes, we need to add xpu test as well. Since the test validation may take a few efforts, I think we may do it in a separate PR after this PR is merged.

@mikekgfb We're trying to add xpu test. Do you know if there is existing xpu runner that we can use. If there is, can you provide the xpu runner info to us? Thanks!

Jack-Khuu · 2025-02-03T23:30:53Z

Give linux.idc.xpu runner a shot

If you spin up a PR, we can loop in the dev-infra folk if needed

* add xpu * add xpu device * update * profile * update install * update * update * update --------- Co-authored-by: Jack-Khuu <jack.khuu.7@gmail.com> Co-authored-by: Guoqiong <guoqiong.song@intel.com>

jenniew added 2 commits November 6, 2024 22:48

add xpu

778efd6

add xpu device

7c4e42b

jenniew added 2 commits November 9, 2024 06:29

update

6ed3cda

Merge branch 'main' of https://github.com/pytorch/torchchat into xpu_…

6e73400

…device

Jack-Khuu requested review from vmpuri, Jack-Khuu and Gasoonjia November 12, 2024 09:19

Jack-Khuu reviewed Nov 12, 2024

View reviewed changes

Merge branch 'main' into xpu_device

4735bff

facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 13, 2024

jenniew added 2 commits November 13, 2024 22:58

merge

6ef7cd5

Merge branch 'xpu_device' of https://github.com/jenniew/torchchat int…

4d16351

…o xpu_device

Jack-Khuu mentioned this pull request Nov 13, 2024

Can we get XPU Nightlies? pytorch/torchtune#2005

Open

songhappy added 7 commits December 11, 2024 00:21

update

11222dd

profile

20e3eae

merge

5e0b073

merge

069d393

Merge branch 'main' of https://github.com/pytorch/torchchat into xpu_…

c0530d0

…device

update install

ab61f0f

update

9f4cde3

update

5c3306e

Jack-Khuu reviewed Jan 15, 2025

View reviewed changes

jenniew added 2 commits January 16, 2025 22:11

update

30a38ec

Merge branch 'main' of https://github.com/pytorch/torchchat into xpu_…

b784239

…device

Merge branch 'main' into xpu_device

8edfeb2

Jack-Khuu approved these changes Jan 18, 2025

View reviewed changes

Jack-Khuu merged commit 2fc98f7 into pytorch:main Jan 18, 2025
54 of 57 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Intel XPU device support to generate and serve #1361

Add Intel XPU device support to generate and serve #1361

jenniew commented Nov 9, 2024

pytorch-bot bot commented Nov 9, 2024 •

edited

Loading

facebook-github-bot commented Nov 9, 2024

mikekgfb commented Nov 9, 2024

jenniew commented Nov 12, 2024

Jack-Khuu Nov 12, 2024

jenniew Nov 13, 2024

Jack-Khuu Nov 13, 2024

Jack-Khuu Nov 12, 2024

jenniew Nov 13, 2024

Jack-Khuu Nov 13, 2024 •

edited

Loading

Jack-Khuu commented Nov 12, 2024

Jack-Khuu commented Nov 12, 2024

jenniew commented Nov 13, 2024

jenniew commented Nov 13, 2024

Jack-Khuu commented Nov 15, 2024 •

edited

Loading

mikekgfb commented Dec 10, 2024 •

edited

Loading

Jack-Khuu Jan 15, 2025

jenniew Jan 16, 2025

jenniew commented Jan 16, 2025

jenniew commented Feb 3, 2025

Jack-Khuu commented Feb 3, 2025

Add Intel XPU device support to generate and serve #1361

Add Intel XPU device support to generate and serve #1361

Conversation

jenniew commented Nov 9, 2024

pytorch-bot bot commented Nov 9, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1361

❌ 1 New Failure, 2 Unrelated Failures

facebook-github-bot commented Nov 9, 2024

Action Required

Process

mikekgfb commented Nov 9, 2024

jenniew commented Nov 12, 2024

Jack-Khuu Nov 12, 2024

Choose a reason for hiding this comment

jenniew Nov 13, 2024

Choose a reason for hiding this comment

Jack-Khuu Nov 13, 2024

Choose a reason for hiding this comment

Jack-Khuu Nov 12, 2024

Choose a reason for hiding this comment

jenniew Nov 13, 2024

Choose a reason for hiding this comment

Jack-Khuu Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Jack-Khuu commented Nov 12, 2024

Jack-Khuu commented Nov 12, 2024

jenniew commented Nov 13, 2024

jenniew commented Nov 13, 2024

Jack-Khuu commented Nov 15, 2024 • edited Loading

mikekgfb commented Dec 10, 2024 • edited Loading

Jack-Khuu Jan 15, 2025

Choose a reason for hiding this comment

jenniew Jan 16, 2025

Choose a reason for hiding this comment

jenniew commented Jan 16, 2025

jenniew commented Feb 3, 2025

Jack-Khuu commented Feb 3, 2025

pytorch-bot bot commented Nov 9, 2024 •

edited

Loading

Jack-Khuu Nov 13, 2024 •

edited

Loading

Jack-Khuu commented Nov 15, 2024 •

edited

Loading

mikekgfb commented Dec 10, 2024 •

edited

Loading