Add platform markers for linux only extra_install_requires #88826

JacobHayes · 2022-11-10T18:04:27Z

#85097 added new extra dependencies on nvidia-*. They are linux (GPU) only packages, but were not marked as such, causing issues installing pytorch 1.13 via Poetry (and possibly other tools that follow PyPI's metadata API) on non-Linux systems. This "fixes" the issue by adding the ; platform_system = 'Linux' marker on these dependencies, but the main problem of different metadata for different wheels is a somewhat larger issue.

#85097 used ; as a delimiter for splitting the different deps, but that is the delimiter used in markers, so I changed to split on |.

pytorch-bot · 2022-11-10T18:04:31Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88826

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

test_jit_cuda_fuser often crashes with SIGIOT

❌ 3 Failures

As of commit 9d8956b:

The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

lalmei

LGTM

hmih · 2022-11-16T13:11:32Z

This PR fixes a huge nuisance for me, can I do something to help push this PR through?

atalman · 2022-11-16T21:11:39Z

we need to attach ciflow/binaries_wheel label and produce the new wheel. Once this wheel is build it needs to be tested. If it works, we can ship this PR

JacobHayes · 2022-11-17T04:01:38Z

I think the test failures are flakes / unrelated (RuntimeError: functorch/test_aotdispatch failed!) and I'm not sure where to find the built wheels (I might be missing them in the GH UI?), but let me know if there's anything else I can do to help on this!

atalman · 2022-11-18T14:05:48Z

Tested the resulting binary installation worked:

pip install torch-1.14.0.dev20221116+cu117.with.pypi.cudnn-cp39-cp39-linux_x86_64.whl
Processing ./torch-1.14.0.dev20221116+cu117.with.pypi.cudnn-cp39-cp39-linux_x86_64.whl
Collecting networkx
  Downloading networkx-2.8.8-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 18.2 MB/s eta 0:00:00
Collecting sympy
  Downloading sympy-1.11.1-py3-none-any.whl (6.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.5/6.5 MB 36.8 MB/s eta 0:00:00
Collecting nvidia-cudnn-cu11==8.5.0.96
  Using cached nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
Collecting typing-extensions
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting nvidia-cuda-runtime-cu11
  Using cached nvidia_cuda_runtime_cu11-11.8.89-py3-none-manylinux1_x86_64.whl (875 kB)
Collecting nvidia-cublas-cu11==11.10.3.66
  Using cached nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
Requirement already satisfied: setuptools in ./miniconda3/envs/poetry/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==1.14.0.dev20221116+cu117.with.pypi.cudnn) (65.5.0)
Requirement already satisfied: wheel in ./miniconda3/envs/poetry/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==1.14.0.dev20221116+cu117.with.pypi.cudnn) (0.37.1)
Collecting mpmath>=0.19
  Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 532.6/532.6 kB 6.8 MB/s eta 0:00:00
Installing collected packages: mpmath, typing-extensions, sympy, nvidia-cuda-runtime-cu11, nvidia-cublas-cu11, networkx, nvidia-cudnn-cu11, torch


Successfully installed mpmath-1.2.1 networkx-2.8.8 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-runtime-cu11-11.8.89 nvidia-cudnn-cu11-8.5.0.96 sympy-1.11.1 torch-1.14.0.dev20221116+cu117.with.pypi.cudnn typing-extensions-4.4.0

Smoke test passed:

python smoke_test.py 
torch: 1.14.0.dev20221116+cu117-with-pypi-cudnn
torch cuda: 11.7
torch cudnn: 8500

Metadata looks good as well:

Requires-Dist: nvidia-cuda-runtime-cu11 ; platform_system == "Linux"
Requires-Dist: nvidia-cudnn-cu11 (==8.5.0.96) ; platform_system == "Linux"
Requires-Dist: nvidia-cublas-cu11 (==11.10.3.66) ; platform_system == "Linux"

atalman · 2022-11-18T14:07:07Z

@pytorchbot merge -f "Test failures are not related"

pytorchmergebot · 2022-11-18T14:09:12Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

varunagrawal · 2022-11-21T19:48:25Z

Can we cut a patch release that includes this fix? Would really help with package managers like poetry.

… torch to ~1.12.1 again because of platform issue: pytorch/pytorch#88826

…8826) Fixes pytorch#88049 pytorch#85097 added new extra dependencies on `nvidia-*`. They are linux (GPU) only packages, but were not marked as such, causing issues installing pytorch 1.13 via Poetry (and possibly other tools that follow PyPI's metadata API) on non-Linux systems. This "fixes" the issue by adding the `; platform_system = 'Linux'` marker on these dependencies, but the main problem of different metadata for different wheels is a [somewhat larger issue](pytorch#88049 (comment)). pytorch#85097 used `;` as a delimiter for splitting the different deps, but that is the delimiter used in markers, so I changed to split on `|`. Pull Request resolved: pytorch#88826 Approved by: https://github.com/neersighted, https://github.com/lalmei, https://github.com/malfet

…89924) Fixes #88049 #85097 added new extra dependencies on `nvidia-*`. They are linux (GPU) only packages, but were not marked as such, causing issues installing pytorch 1.13 via Poetry (and possibly other tools that follow PyPI's metadata API) on non-Linux systems. This "fixes" the issue by adding the `; platform_system = 'Linux'` marker on these dependencies, but the main problem of different metadata for different wheels is a [somewhat larger issue](#88049 (comment)). #85097 used `;` as a delimiter for splitting the different deps, but that is the delimiter used in markers, so I changed to split on `|`. Pull Request resolved: #88826 Approved by: https://github.com/neersighted, https://github.com/lalmei, https://github.com/malfet Co-authored-by: Jacob Hayes <jacob.r.hayes@gmail.com>

…8826) Fixes pytorch#88049 pytorch#85097 added new extra dependencies on `nvidia-*`. They are linux (GPU) only packages, but were not marked as such, causing issues installing pytorch 1.13 via Poetry (and possibly other tools that follow PyPI's metadata API) on non-Linux systems. This "fixes" the issue by adding the `; platform_system = 'Linux'` marker on these dependencies, but the main problem of different metadata for different wheels is a [somewhat larger issue](pytorch#88049 (comment)). pytorch#85097 used `;` as a delimiter for splitting the different deps, but that is the delimiter used in markers, so I changed to split on `|`. Pull Request resolved: pytorch#88826 Approved by: https://github.com/neersighted, https://github.com/lalmei, https://github.com/malfet

lawrenceong · 2022-12-21T02:17:56Z

NOTE: This fix does not address platform_system == 'Linux' and platform_machine != 'x86_64' as the nvidia-cublas-cu11 does not exist except for x86_64.

huydhn · 2022-12-21T02:44:31Z

NOTE: This fix does not address platform_system == 'Linux' and platform_machine != 'x86_64' as the nvidia-cublas-cu11 does not exist except for x86_64.

This looks related to this open PR #91083, but the change doesn't seem to work out just yet

jspaezp · 2022-12-21T03:02:50Z

@huydhn in my system it works setting the version to >=1.3.1, can you confirm it does not work for you in that case?

Add platform markers for linux only extra_install_requires

9d8956b

JacobHayes requested a review from a team as a code owner November 10, 2022 18:04

pytorch-bot bot added the topic: not user facing topic category label Nov 10, 2022

JacobHayes mentioned this pull request Nov 10, 2022

PyTorch 1.13 cannot be installed using poetry on Mac M1 #88049

Closed

pytorchbot added the open source label Nov 10, 2022

neersighted approved these changes Nov 10, 2022

View reviewed changes

JacobHayes mentioned this pull request Nov 10, 2022

Instructions for installing PyTorch python-poetry/poetry#6409

Open

1 task

ngimel added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 10, 2022

lalmei approved these changes Nov 11, 2022

View reviewed changes

atalman added the ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR label Nov 16, 2022

malfet added this to the 1.13.1 milestone Nov 16, 2022

malfet approved these changes Nov 16, 2022

View reviewed changes

pytorchmergebot added the Merged label Nov 18, 2022

pytorchmergebot closed this in 2e358cc Nov 18, 2022

JacobHayes deleted the fix-linux-only-extra-dep-metadata branch November 18, 2022 14:43

jspaezp mentioned this pull request Nov 18, 2022

Poetry unable to find installation candidates for Nvidia c11 dependencies python-poetry/poetry#6939

Closed

4 tasks

lsickert added a commit to inseq-team/inseq that referenced this pull request Nov 23, 2022

Merge branch 'main' into attention-attribution\n\nNeeded to downgrade…

bb13036

… torch to ~1.12.1 again because of platform issue: pytorch/pytorch#88826

lsickert mentioned this pull request Nov 23, 2022

Basic Attention attribution inseq-team/inseq#148

Merged

11 tasks

This was referenced Nov 30, 2022

Add platform markers for linux only extra_install_requires (#88826) #89924

Merged

[v.1.13.1] Release Tracker #89855

Closed

philswatton mentioned this pull request Dec 2, 2022

System Dependencies for PyTorch (or TensorFlow) alan-turing-institute/arc-model-similarities-phase-2#8

Closed

lsickert mentioned this pull request Dec 12, 2022

[WIP] Fix issues with pytorch 1.13 on MacOs inseq-team/inseq#151

Merged

11 tasks

atalman mentioned this pull request Dec 14, 2022

Changes to prep for pypi script for release 1.13.1 pytorch/builder#1231

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add platform markers for linux only extra_install_requires #88826

Add platform markers for linux only extra_install_requires #88826

JacobHayes commented Nov 10, 2022

pytorch-bot bot commented Nov 10, 2022 •

edited

Loading

lalmei left a comment

hmih commented Nov 16, 2022 •

edited

Loading

atalman commented Nov 16, 2022 •

edited

Loading

JacobHayes commented Nov 17, 2022

atalman commented Nov 18, 2022 •

edited

Loading

atalman commented Nov 18, 2022

pytorchmergebot commented Nov 18, 2022

varunagrawal commented Nov 21, 2022

lawrenceong commented Dec 21, 2022

huydhn commented Dec 21, 2022

jspaezp commented Dec 21, 2022

Add platform markers for linux only extra_install_requires #88826

Add platform markers for linux only extra_install_requires #88826

Conversation

JacobHayes commented Nov 10, 2022

pytorch-bot bot commented Nov 10, 2022 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88826

❗ 1 Active SEVs

❌ 3 Failures

lalmei left a comment

Choose a reason for hiding this comment

hmih commented Nov 16, 2022 • edited Loading

atalman commented Nov 16, 2022 • edited Loading

JacobHayes commented Nov 17, 2022

atalman commented Nov 18, 2022 • edited Loading

atalman commented Nov 18, 2022

pytorchmergebot commented Nov 18, 2022

Merge started

varunagrawal commented Nov 21, 2022

lawrenceong commented Dec 21, 2022

huydhn commented Dec 21, 2022

jspaezp commented Dec 21, 2022

pytorch-bot bot commented Nov 10, 2022 •

edited

Loading

hmih commented Nov 16, 2022 •

edited

Loading

atalman commented Nov 16, 2022 •

edited

Loading

atalman commented Nov 18, 2022 •

edited

Loading