Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix ExecuTorch CI after landing #6564 #139700

Closed
wants to merge 3 commits into from
Closed

Fix ExecuTorch CI after landing #6564 #139700

wants to merge 3 commits into from

Conversation

huydhn
Copy link
Contributor

@huydhn huydhn commented Nov 5, 2024

After landing pytorch/executorch#6564, we need to update the pinned ExecuTorch commit on PyTorch is fix the regression on PyTorch side. The change to .ci/docker/common/install_executorch.sh is needed because it's how the dependencies are setup on ExecuTorch CI now.

@huydhn huydhn requested a review from jeffdaily as a code owner November 5, 2024 01:57
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Nov 5, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139700

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 9f94cea with merge base c92de3b (image):

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@huydhn
Copy link
Contributor Author

huydhn commented Nov 5, 2024

@guangy10 @larryliu0820 After pytorch/executorch#6564, we need this change to fix the regression on PyTorch side. However, I think during the gap when PyTorch wasn't installed correctly, some changes has already landed that doesn't work with ExecuTorch yet. Specifically, I'm referring to this failure https://github.com/pytorch/pytorch/actions/runs/11676105984/job/32512808907?pr=139700#step:22:7900

ERROR examples/models/llama3_2_vision/preprocess/test_preprocess.py
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_add_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.add' has no overload name 'out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_choose_qparams_tensor_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.choose_qparams' has no overload name 'Tensor_out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_dequantize_per_channel_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.dequantize_per_channel' has no overload name 'out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_dequantize_per_tensor_tensor_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.dequantize_per_tensor' has no overload name 'Tensor_out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_dequantize_per_tensor_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.dequantize_per_tensor' has no overload name 'out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_mixed_linear_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.mixed_linear' has no overload name 'out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_mixed_mm_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.mixed_mm' has no overload name 'out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_quantize_per_channel_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.quantize_per_channel' has no overload name 'out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_quantize_per_tensor_tensor_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.quantize_per_tensor' has no overload name 'Tensor_out'
FAILED kernels/quantized/test/test_out_variants.py::TestOutVariants::test_quantize_per_tensor_to_out_variant - AttributeError: The underlying op of 'quantized_decomposed.quantize_per_tensor' has no overload name 'out'
FAILED extension/pybindings/test/test_pybindings.py::PybindingsTest::test - RuntimeError: Missing out variants: {'quantized_decomposed::dequantize_per_tensor', 'quantized_decomposed::add', 'quantized_decomposed::quantize_per_tensor'}

Is there a way to ignore them to land this change, then follow up with proper fixes later? I want to land this change as early as possible to avoid further misses.

@huydhn huydhn requested a review from a team as a code owner November 6, 2024 17:56
@huydhn
Copy link
Contributor Author

huydhn commented Nov 6, 2024

Chat with @larryliu0820, let's land this and mark this as unstable for a forward fix

@huydhn huydhn added the ciflow/unstable Run all experimental or flaky jobs on PyTorch unstable workflow label Nov 6, 2024
@huydhn
Copy link
Contributor Author

huydhn commented Nov 6, 2024

@pytorchbot merge -f 'Land this first and will follow up with ExecuTorch team later on the fix'

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@huydhn
Copy link
Contributor Author

huydhn commented Nov 6, 2024

@larryliu0820 I think I understand what happens now. The test that is failing examples/models/llama3_2_vision/preprocess/test_preprocess.py requires torchtune and torchao, otherwise, it would fail. It makes sense now as the quantized ops are from torchao. From what I see, both repos are currently not included in https://github.com/.../blob/main/install_requirements.py, so I assume that they are optional dependency. Given that, I think it's better to skip the test if torchtune and ao are not available and run it only when they are installed. We need that flexibility when running ET tests on PT CI.

pytest -v examples/models/llama3_2_vision/preprocess/test_preprocess.py works locally for me when I have them installed.

@huydhn huydhn mentioned this pull request Nov 7, 2024
Closed
atalman pushed a commit to atalman/pytorch that referenced this pull request Nov 11, 2024
@huydhn @atalman
Fix ExecuTorch CI after landing pytorch#6564 (pytorch#139700)
3825946 
After landing pytorch/executorch#6564, we need to update the pinned ExecuTorch commit on PyTorch is fix the regression on PyTorch side.  The change to `.ci/docker/common/install_executorch.sh` is needed because it's how the dependencies are setup on ExecuTorch CI now.
Pull Request resolved: pytorch#139700
Approved by: https://github.com/larryliu0820, https://github.com/malfet
zero000064 pushed a commit to zero000064/pytorch that referenced this pull request Nov 14, 2024
@huydhn @zero000064
Fix ExecuTorch CI after landing pytorch#6564 (pytorch#139700)
2ded4d1 
After landing pytorch/executorch#6564, we need to update the pinned ExecuTorch commit on PyTorch is fix the regression on PyTorch side.  The change to `.ci/docker/common/install_executorch.sh` is needed because it's how the dependencies are setup on ExecuTorch CI now.
Pull Request resolved: pytorch#139700
Approved by: https://github.com/larryliu0820, https://github.com/malfet
Ryo-not-rio pushed a commit to Ryo-not-rio/pytorch that referenced this pull request Dec 2, 2024
@huydhn @Ryo-not-rio
Fix ExecuTorch CI after landing pytorch#6564 (pytorch#139700)
97e4d83 
After landing pytorch/executorch#6564, we need to update the pinned ExecuTorch commit on PyTorch is fix the regression on PyTorch side.  The change to `.ci/docker/common/install_executorch.sh` is needed because it's how the dependencies are setup on ExecuTorch CI now.
Pull Request resolved: pytorch#139700
Approved by: https://github.com/larryliu0820, https://github.com/malfet
pobin6 pushed a commit to pobin6/pytorch that referenced this pull request Dec 5, 2024
@huydhn
Fix ExecuTorch CI after landing pytorch#6564 (pytorch#139700)
40f3f6e 
After landing pytorch/executorch#6564, we need to update the pinned ExecuTorch commit on PyTorch is fix the regression on PyTorch side.  The change to `.ci/docker/common/install_executorch.sh` is needed because it's how the dependencies are setup on ExecuTorch CI now.
Pull Request resolved: pytorch#139700
Approved by: https://github.com/larryliu0820, https://github.com/malfet
fmo-mt pushed a commit to fmo-mt/pytorch that referenced this pull request Dec 11, 2024
@huydhn @fmo-mt
Fix ExecuTorch CI after landing pytorch#6564 (pytorch#139700)
2ee4e6a 
After landing pytorch/executorch#6564, we need to update the pinned ExecuTorch commit on PyTorch is fix the regression on PyTorch side.  The change to `.ci/docker/common/install_executorch.sh` is needed because it's how the dependencies are setup on ExecuTorch CI now.
Pull Request resolved: pytorch#139700
Approved by: https://github.com/larryliu0820, https://github.com/malfet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@larryliu0820 larryliu0820 larryliu0820 approved these changes

@malfet malfet malfet approved these changes

@guangy10 guangy10 Awaiting requested review from guangy10

@jeffdaily jeffdaily Awaiting requested review from jeffdaily jeffdaily is a code owner

Assignees
No one assigned
Labels
ciflow/inductor ciflow/unstable Run all experimental or flaky jobs on PyTorch unstable workflow Merged test-config/default test-config/executorch topic: not user facing topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@huydhn @pytorchmergebot @malfet @larryliu0820