Use "vendoring" to maintain Spack's pure Python dependencies #33951

alalazo · 2022-11-16T19:52:48Z

depends on #34029

In this PR I am using vendoring to manage our vendored dependencies. The important modifications are in the first commit. The other two are just deleting the old dependencies and adding back the new ones.

To regenerate the tree of dependencies, from the root of our repo:

$ vendoring sync -v
Working in .
Load configuration... 
  Will attempt to load pyproject.toml.
  Read configuration file.
  Parsed configuration file.
  Validated configuration.
  Done!
Clean existing libraries... 
  Done!
Add vendored libraries... 
  Running pip install -t lib/spack/external/_vendoring -r lib/spack/external/vendor.txt --no-compile --no-deps
    Collecting distro==1.8.0
      Using cached distro-1.8.0-py3-none-any.whl (20 kB)
    Collecting jsonschema==3.0.2
      Using cached jsonschema-3.0.2-py2.py3-none-any.whl (54 kB)
[ ... ]

The list of our vendored dependencies is maintained in the lib/spack/external/vendor.txt file. Licenses etc. are copied automatically by the vendoring tool in the folder created for the package.

alalazo · 2022-11-16T19:54:37Z

Currently I am not vendoring pytest and erroring out with a silly error message. Before marking this PR as "Ready for review" I'll either add it back or find a way to bootstrap it as we do for the tools used in spack style.

EDIT: #34029 adds capabilities to bootstrap pytest

alalazo · 2022-12-06T15:15:14Z

@spackbot fix style

spackbot-app · 2022-12-06T15:15:17Z

Let me see if I can fix that for you!

spackbot-app · 2022-12-06T15:16:55Z

I encountered an error attempting to format style.

Details

Error: <class 'sh.SignalException_SIGKILL'>, 

  RAN: /tmp/tmpekn986ei/spack-develop/bin/spack --color never style --fix --root /tmp/tmpekn986ei/spack

  STDOUT:


  STDERR:

Stack trace:
  File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 1075, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 854, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 880, in _execute
    coro_result = loop.run_until_complete(result)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "./spackbot/workers.py", line 336, in fix_style_task
    spack, ["--color", "never", "style", "--fix", "--root", check_dir]
  File "./spackbot/helpers.py", line 164, in run_command
    raise inst
  File "./spackbot/helpers.py", line 157, in run_command
    control(*cmd, _out=res, _err=err, _ok_code=ok_codes)
  File "/usr/local/lib/python3.7/site-packages/sh.py", line 1524, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/local/lib/python3.7/site-packages/sh.py", line 788, in __init__
    self.wait()
  File "/usr/local/lib/python3.7/site-packages/sh.py", line 845, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/local/lib/python3.7/site-packages/sh.py", line 869, in handle_command_exit_code
    raise exc

alalazo · 2022-12-07T08:49:18Z

@spackbot run pipeline

spackbot-app · 2022-12-07T08:49:21Z

I'm sorry, gitlab does not have your latest revision yet, I can't run that pipeline for you right now.

One likely possibility is that your PR pipeline has been temporarily deferred, in which case, it is awaiting a develop pipeline, and will be run when that finishes.

Please check the gitlab commit status message to see if more information is available.

Details

Unexpected response from gitlab: {'message': '404 Commit Not Found'}

alalazo · 2022-12-07T15:24:48Z

@spackbot run pipeline

spackbot-app · 2022-12-07T15:24:51Z

I'm sorry, gitlab does not have your latest revision yet, I can't run that pipeline for you right now.

One likely possibility is that your PR pipeline has been temporarily deferred, in which case, it is awaiting a develop pipeline, and will be run when that finishes.

Please check the gitlab commit status message to see if more information is available.

Details

Unexpected response from gitlab: {'message': '404 Commit Not Found'}

tgamblin

This LGTM as-is. A couple comments but honestly this is great. Thanks to @pradyunsg for vendoring.

My only worry is this, from here:

This tool has no stability promises -- it has only one intended user: pip. There may be unannounced changes to this codebase at any time, as long as the intended user (i.e. the pip project) is prepared for those changes.

But we deal with more rapidly changing dependencies than this anyway... and if vendoring disappears it won't break spack... the externals are already copied in and committed here.

tgamblin · 2023-01-04T17:03:03Z

lib/spack/docs/developer_guide.rst

@@ -232,7 +232,7 @@ Spack Subcommands
 Unit tests
 ^^^^^^^^^^

-:mod:`spack.test`
+``spack.test``


why not a module reference?

tgamblin · 2023-01-04T17:38:46Z

pyproject.toml

+  "pkg_resources/extern/",
+  # trim vendored pygments styles and lexers
+  "pygments/styles/[!_]*.py",
+  '^pygments/lexers/(?!python|__init__|_mapping).*\.py$',


Random comment: I have thought a few times that it would be nice to vendor pygments so we could colorize YAML and JSON output.

The only reason pip trims this is for distribution sizes.

pradyunsg · 2023-01-04T21:17:43Z

You can have my word that I’ll bump the major version when I make a backwards incompatible change. Pin with a ~=1.2 or whatever variant you prefer. :)

We aren't using Spack (heavily) in these images, all we use it for is to install from the buildcache. Spack doesn't need to know about *all* the compilers we installed to do that much. This also fixes an issue regarding the Spack code for OpenSUSE Leap 15, it used to be called `os=opensuse_leap15` but now is `os=opensuse15`. Bisection reveals the difference appeared when the distro vendered dependency was updated in spack/spack#33951 to include python-distro/distro#319. The change in Spack occured in January 2023, years after the change in distro was merged in October 2021, over a year after OpenSUSE Leap 15.2 was released in July 2020. Sigh.

spackbot-app bot added commands core PR affects Spack core functionality documentation Improvements or additions to documentation stand-alone-tests Stand-alone (or smoke) tests for installed packages vendored-dependencies labels Nov 16, 2022

alalazo mentioned this pull request Nov 21, 2022

Bootstrap most of Spack dependencies using environments #34029

Merged

5 tasks

alalazo force-pushed the maintainers/use_vendoring_to_update_dependencies branch from 117e2b5 to 25dcd6d Compare December 6, 2022 12:00

spackbot-app bot added the architecture label Dec 6, 2022

alalazo marked this pull request as ready for review December 6, 2022 12:00

alalazo force-pushed the maintainers/use_vendoring_to_update_dependencies branch from fc1ea7b to 5aba414 Compare December 6, 2022 15:01

alalazo force-pushed the maintainers/use_vendoring_to_update_dependencies branch from 5aba414 to d29ee45 Compare December 6, 2022 22:16

alalazo force-pushed the maintainers/use_vendoring_to_update_dependencies branch from d29ee45 to d470046 Compare December 21, 2022 15:25

spackbot-app bot added the tests General test capability(ies) label Dec 21, 2022

alalazo added 4 commits December 29, 2022 11:31

Use "vendoring" to manage 3rd party dependencies

3c9508f

Delete outdated externals

c9ea9dd

Add vendored packages back

4cca9e1

Fix building docs

d7d2631

alalazo force-pushed the maintainers/use_vendoring_to_update_dependencies branch from d470046 to d7d2631 Compare December 29, 2022 10:54

tgamblin self-requested a review January 4, 2023 16:53

tgamblin approved these changes Jan 4, 2023

View reviewed changes

tgamblin merged commit 6a44a14 into spack:develop Jan 4, 2023

alalazo deleted the maintainers/use_vendoring_to_update_dependencies branch January 4, 2023 17:52

alalazo mentioned this pull request Jan 4, 2023

Update Jinja2 to v3.0.3 and MarkupSafe to v2.0.1 #33922

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use "vendoring" to maintain Spack's pure Python dependencies #33951

Use "vendoring" to maintain Spack's pure Python dependencies #33951

alalazo commented Nov 16, 2022 •

edited

Loading

alalazo commented Nov 16, 2022 •

edited

Loading

alalazo commented Dec 6, 2022

spackbot-app bot commented Dec 6, 2022

spackbot-app bot commented Dec 6, 2022

alalazo commented Dec 7, 2022

spackbot-app bot commented Dec 7, 2022

alalazo commented Dec 7, 2022

spackbot-app bot commented Dec 7, 2022

tgamblin left a comment

tgamblin Jan 4, 2023

tgamblin Jan 4, 2023

pradyunsg Jan 4, 2023

pradyunsg commented Jan 4, 2023 •

edited

Loading

Use "vendoring" to maintain Spack's pure Python dependencies #33951

Use "vendoring" to maintain Spack's pure Python dependencies #33951

Conversation

alalazo commented Nov 16, 2022 • edited Loading

alalazo commented Nov 16, 2022 • edited Loading

alalazo commented Dec 6, 2022

spackbot-app bot commented Dec 6, 2022

spackbot-app bot commented Dec 6, 2022

alalazo commented Dec 7, 2022

spackbot-app bot commented Dec 7, 2022

alalazo commented Dec 7, 2022

spackbot-app bot commented Dec 7, 2022

tgamblin left a comment

Choose a reason for hiding this comment

tgamblin Jan 4, 2023

Choose a reason for hiding this comment

tgamblin Jan 4, 2023

Choose a reason for hiding this comment

pradyunsg Jan 4, 2023

Choose a reason for hiding this comment

pradyunsg commented Jan 4, 2023 • edited Loading

alalazo commented Nov 16, 2022 •

edited

Loading

alalazo commented Nov 16, 2022 •

edited

Loading

pradyunsg commented Jan 4, 2023 •

edited

Loading