[engsys] Migrate to tox4 (#30159)

* Remove references to {envbindir} tox prepends the {env_bin_dir} to $PATH, so commands will default to using the virtual environment first. See: https://tox.wiki/en/4.4.11/config.html#commands One of tox's core maintainers also discourages its use: tox-dev/tox#2909 (comment) * refactor: Write paths relative to git root, instead of package dir Uses an inline plugin (toxfile.py) to make a computed config value avaiable to tox environments. * refactor: Rename {toxinidir} to {tox_root} {toxinidir} is an alias for {tox_root}, but {tox_root} is: * More clear: {tox_root} isn't necessarily where the ini file is * Shorter * refactor: Flatten some commands that are no longer super long * refactor: Make a pytest section * refactor: Remove `changedir = {tox_root}` This is the default behavior, see: https://tox.wiki/en/4.4.11/config.html#change_dir-external * fix: {distdir} is deprecated See: https://tox.wiki/en/latest/upgrading.html#removed-tox-ini-keys * fix: Remove --ignore-installed from pip command --ignore-installed can break a python installation if multiple conflicting versions of a package are installed * refactor: Add descriptions to all tox environments * fix: Remove ineffective platform config platform = linux: linux macos: darwin windows: win32 Setting the above in the base environment has no effect: * None of the environments have {linux,macos,windows} in the name, so the platform config is always empty. * chore: Bump `tox` to `>4.4.11` and no longer install `tox-monorepo` tox4 natively solves what `tox-monorepo` was written to solve: `--root` lets you set `toxinidir` independently of the config file in use. 4.4.11 was chosen as the cutoff since it was the first release to include a fix to `--root` that prevented `{work_dir}` from being changed when `{toxinidir}` was changed by `--root`. `tox-monorepo` should no longer be needed * docs: Update CONTRIBUTING.md to use tox 4 commands * References to tox-monorepo were removed * Replaced discussion of `tox -l` with tox4's `tox list` * Updated examples to use `--root` * docs: Make `azure-media-{videoanalyzer,analytics}-edge`'s DevTips.md point to CONTRIBUTING.md for tox * docs: Remove remaining references to tox-monorepo * doc: Use `--root` throughout documentation * engsys: Make tox_harness set `{toxinidir}` when invoking tox * engsys: Set required tox version in config * docs: Use the new url for tox docs * docs: Add a callout that TOX_* environment variables control defaults * TOX_CONFIG_DIR can be used to permanently set --conf * TOX_ROOT_DIR can be used to permanently set --root * update packaging requirement everywhere it is used. * fix cspell * docs: Use `tox run` syntax throughout documentation * engsys: Use `tox run` syntax in tox_harness.py * docs: Replace – (U+2013 : EN DASH) with - (ASCII code 45) * update pytest requirements + coverage. * update virtualenv * more conflicts resolved * bringing in urllib3 restriction, aligning ci_tools.txt with test_tools.txt * check_call requires a list of type string. if we append ['blah', 'blah'] we are actually inserting another array. just use regular array concatenation * engsys: Tox uses `run` for serial execution and `run-parallel` for parallel` * azure-core: Use `@pytest_asyncio.fixture` to mark an asyncio fixture See https://pytest-asyncio.readthedocs.io/en/latest/concepts.html#strict-mode * pin coverage to newer version with no conflicts * identify issue with missing code coverage being caused by wrongly named namespace (#30344) * remove fix of dotfile! * apply black formatting to resolve failing analyze step * tox installing tox. probably not a good thing :) * engsys: Remove `tox` from `azure-media-analytics-edge` dev_requirements * align regression version of ci/test_tools.txt * new coverage format requires that the package sources be present when combining .coverage files. amend our tox tree cleanup to exclude the whl directory when running coverage. * ensure that missing hidden folder can't break cleanup operation * ensure coverage isn't generated on whl_no_aio * ensure that coverage is generated without wonky package paths --------- Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>
Azure · May 23, 2023 · 982234b · 982234b
1 parent 62bbb9f
commit 982234b
Show file tree

Hide file tree

Showing 31 changed files with 343 additions and 339 deletions.
diff --git a/.vscode/cspell.json b/.vscode/cspell.json
@@ -416,6 +416,12 @@
         "pytestmarkparametrize"
       ]
     },
+    {
+      "filename": "doc/dev/pylint_checking.md",
+      "words": [
+        "pylintrc"
+      ]
+    },
     {
       "filename": "tools/azure-sdk-tools/devtools_testutils/proxy_startup.py",
       "words": [

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -11,13 +11,13 @@ If you want to contribute to a file that is generated (header contains `Code gen
 
 The Azure SDK team's Python CI leverages the tool `tox` to distribute tests to virtual environments, handle test dependency installation, and coordinate tooling reporting during PR/CI builds. This means that a dev working locally can reproduce _exactly_ what the build machine is doing.
 
-[A Brief Overview of Tox](https://tox.readthedocs.io/en/latest/)
+[A Brief Overview of Tox](https://tox.wiki/en/latest/)
 
 #### A Monorepo and Tox in Harmony
 
 Traditionally, the `tox.ini` file for a package sits _alongside the setup.py_ in source code. The `azure-sdk-for-python` necessarily does not adhere to this policy. There are over one-hundred packages contained here-in. That's a lot of `tox.ini` files to maintain!
 
-Instead, the CI system leverages an tox plugin called `tox-monorepo`. This plugin allows `tox` to act as if the `tox.ini` is located in whatever directory you executed tox in!
+Instead, the CI system leverages the `--root` argument which is new to `tox4`. The `--root` argument allows `tox` to act as if the `tox.ini` is located in whatever directory you specify!
 
 #### Tox Environments
 
@@ -32,40 +32,62 @@ Internally `tox` leverages `virtualenv` to create each test environment's virtua
 
 This means that once the `tox` workflow is in place, all tests will be executed _within a virtual environment._
 
-To see the default environments from a specific `tox.ini` file, use the command `tox -l` in the same directory as the file itself.
-
-> sdk-for-python/eng/tox> tox -l
-
-```
-
-whl
-sdist
-
+You can use the command `tox list` to list all the environments provided by a `tox.ini` file. You can either use that command in the
+same directory as the file itself, or use the `--conf` argument to specify the path to it directly.
+
+
+Sample output of `tox list`:
+
+```
+sdk-for-python/eng/tox> tox list
+default environments:
+whl              -> Builds a wheel and runs tests
+sdist            -> Builds a source distribution and runs tests
+
+additional environments:
+pylint           -> Lints a package with a pinned version of pylint
+next-pylint      -> Lints a package with pylint (version 2.15.8)
+mypy             -> Typechecks a package with mypy (version 1.0.0)
+next-mypy        -> Typechecks a package with the latest version of mypy
+pyright          -> Typechecks a package with pyright (version 1.1.287)
+next-pyright     -> Typechecks a package with the latest version of static type-checker pyright
+verifytypes      -> Verifies the "type completeness" of a package with pyright
+whl_no_aio       -> Builds a wheel without aio and runs tests
+develop          -> Tests a package
+sphinx           -> Builds a package's documentation with sphinx
+depends          -> Ensures all modules in a target package can be successfully imported
+verifywhl        -> Verify directories included in whl and contents in manifest file
+verifysdist      -> Verify directories included in sdist and contents in manifest file. Also ensures that py.typed configuration is correct within the setup.py
+devtest          -> Tests a package against dependencies installed from a dev index
+latestdependency -> Tests a package against the released, upper-bound versions of its azure dependencies
+mindependency    -> Tests a package against the released, lower-bound versions of its azure dependencies
+apistub          -> Generate an api stub of a package ( for https://apiview.dev )
+bandit           -> Runs bandit, a tool to find common security issues, against a package
+samples          -> Runs a package's samples
+breaking         -> Runs the breaking changes checker against a package
 ```
 
-Unfortunately, the command `tox -l` only returns the _default_ test builds. The common `tox.ini` file also supports `pylint` and `mypy` environments.
-
 ### Example Usage of the common Azure SDK For Python `tox.ini`
 
 Basic usage of `tox` within this monorepo is:
 
-1. `pip install tox<4 tox-monorepo`
-2. `cd` to target package folder
-3. run `tox -c path/to/tox.ini`
+1. `pip install tox<5`
+2. Run `tox run -e ENV_NAME -c path/to/tox.ini --root path/to/python_package`
+  * **Note**: You can use environment variables to provide defaults for tox config values
+    * With `TOX_CONFIG_FILE` set to the absolute path of `tox.ini`, you can avoid needing `-c path/to/tox.ini` in your tox invocations
+    * With `TOX_ROOT_DIR` set to the absolute path to your python package, you can avoid needing `--root path/to/python_package`
 
 The common `tox.ini` location is `eng/tox/tox.ini` within the repository.
 
 If at any time you want to blow away the tox created virtual environments and start over, simply append `-r` to any tox invocation!
 
 #### Example `azure-core` mypy
 
-1. `cd` to `sdk/core/azure-core`
-2. Run `tox -e mypy -c ../../../eng/tox/tox.ini`
+1. Run `tox run -e mypy -c ../../../eng/tox/tox.ini --root sdk/core/azure-core`
 
 #### Example `azure-storage-blob` tests
 
-1. `cd` to `sdk/storage/azure-storage-blob`
-2. Execute `tox -c ../../../eng/tox/tox.ini`
+2. Execute `tox run -c ../../../eng/tox/tox.ini --root sdk/storage/azure-storage-blob`
 
 Note that we didn't provide an `environment` argument for this example. Reason here is that the _default_ environment selected by our common `tox.ini` file is one that runs `pytest`.
 
@@ -75,7 +97,7 @@ Used for test execution across the spectrum of all the platforms we want to supp
 * Installs the wheel, runs tests using the wheel
 
 ```
-\> tox -e whl -c <path to tox.ini>
+\> tox run -e whl -c <path to tox.ini> --root <path to python package>
 
 ```
 
@@ -87,30 +109,30 @@ Used for the local dev loop.
 
 ```
 
-\> tox -e sdist -c <path to tox.ini>
+\> tox run -e sdist -c <path to tox.ini> --root <path to python package>
 
 ```
 
 #### `pylint` environment
 Pylint install and run.
 
 ```
-\> tox -e pylint -c <path to tox.ini>
+\> tox run -e pylint -c <path to tox.ini> --root <path to python package>
 ```
 
 
 #### `mypy` environment
 Mypy install and run.
 
 ```
-\> tox -e mypy -c <path to tox.ini>
+\> tox run -e mypy -c <path to tox.ini> --root <path to python package>
 ```
 
 #### `sphinx` environment
 Generate sphinx doc for this package.
 
 ```
-\> tox -e sphinx -c <path to tox.ini>
+\> tox run -e sphinx -c <path to tox.ini> --root <path to python package>
 ```
 
 ### Custom Pytest Arguments
@@ -120,7 +142,7 @@ Generate sphinx doc for this package.
 [Tox Documentation on Positional Arguments](https://tox.wiki/en/latest/config.html#substitutions-for-positional-arguments-in-commands)
 
 **Example: Invoke tox, breaking into the debugger on failure**
-`tox -e whl -c ../../../eng/tox/tox.ini -- --pdb`
+`tox run -e whl -c <path to tox.ini> --root <path to python package> -- --pdb`
 
 ### Performance Testing
 
@@ -150,7 +172,7 @@ a. cd to package root folder
 b. run tox environment devtest
 
 ```
-\> tox -e devtest -c <path to tox.ini>
+\> tox run -e devtest -c <path to tox.ini> --root <path to python package>
 ```
 
 This tox test( devtest) will fail if installed dependent packages are not dev build version.

diff --git a/doc/dev/dev_setup.md b/doc/dev/dev_setup.md
@@ -33,11 +33,11 @@ or execute the various commands available in the toolbox.
 
 4. Setup your development environment
 
-    Install the development requirements for a specific library (located in the `dev_requirements.txt` file at the root of the library), [Tox][tox], [Tox monorepo][tox_monorepo] and an editable install of your library. For example, to install requirements for `azure-ai-formrecognizer`:
+    Install the development requirements for a specific library (located in the `dev_requirements.txt` file at the root of the library), [Tox][tox] and an editable install of your library. For example, to install requirements for `azure-ai-formrecognizer`:
     ```
     azure-sdk-for-python> cd sdk/formrecognizer/azure-ai-formrecognizer
     azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install -r dev_requirements.txt
-    azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install tox tox-monorepo
+    azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install tox<5
     azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install -e .
     ```
 
@@ -58,6 +58,5 @@ Follow the instructions in the [Perform one-time test proxy setup][proxy_setup]
 [python_39]: https://www.microsoft.com/p/python-39/9p7qfqmjrfp7
 [proxy_setup]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md#perform-one-time-test-proxy-setup
 [tests]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md
-[tox]: https://tox.readthedocs.io/en/latest/
-[tox_monorepo]: https://pypi.org/project/tox-monorepo/
+[tox]: https://tox.wiki/en/latest/
 [virtual_environment]: https://docs.python.org/3/tutorial/venv.html
diff --git a/doc/dev/pylint_checking.md b/doc/dev/pylint_checking.md
@@ -24,7 +24,7 @@ In the Azure SDK for Python repository, in addition to the standard pylint libra
 
 One way to run pylint is to run at the package level with tox:
 
-    .../azure-sdk-for-python/sdk/eventgrid/azure-eventgrid>tox -e pylint -c ../../../eng/tox/tox.ini
+    .../azure-sdk-for-python/sdk/eventgrid/azure-eventgrid>tox run -e pylint -c ../../../eng/tox/tox.ini --root .
 
 If you don't want to use tox, you can also install and run pylint on its own:
 
@@ -58,9 +58,9 @@ In addition to being a part of the CI, the custom pylint checkers are also integ
 
 There is now a new step on the CI pipeline called `Run Pylint Next`. This is merely a duplicate of the `Run Pylint` step with the exception that `Run Pylint Next` uses the latest version of pylint and the latest version of the custom pylint checkers.
 
-This next-pylint enviornment can also be run locally through tox:
+This next-pylint environment can also be run locally through tox:
 
-    tox -e next-pylint -c ../../../eng/tox/tox.ini
+    tox run -e next-pylint -c ../../../eng/tox/tox.ini --root <path to python package>
 
 The errors generated by the `Run Pylint Next` step will not break your weekly test pipelines, but make sure to fix the warnings so that your client library is up to date for the next pylint release. 
 

diff --git a/doc/dev/sample_guide.md b/doc/dev/sample_guide.md
@@ -52,13 +52,13 @@ The given `START`/`END` keywords can be used in a [sphinx literalinclude][sphinx
 [Literalinclude example][literalinclude]
 
 The rendered code snippets are sensitive to the indentation in the sample file. Adjust the `dedent` accordingly to ensure the sample is captured accurately and not accidentally trimmed.
-You can preview how published reference documentation will look by running [tox][tox]: `tox -e sphinx -c ../../../eng/tox/tox.ini`.
+You can preview how published reference documentation will look by running [tox][tox]: `tox run -e sphinx -c ../../../eng/tox/tox.ini --root <path to python package>`.
 
 ## Test run samples in CI live tests
 Per the [Python guidelines][snippet_guidelines], sample code and snippets should be test run in CI to ensure they remain functional. Samples should be run in the package's live test pipeline which is scheduled to run daily.
 To ensure samples do get tested as part of regular CI runs, add these [lines][live_tests] to the package's tests.yml. 
 
-You can test this CI step locally first, by utilizing [tox][tox] and running `tox -e samples -c ../../../eng/tox/tox.ini` at the package-level.
+You can test this CI step locally first, by utilizing [tox][tox] and running `tox run -e samples -c ../../../eng/tox/tox.ini --root <path to python package>`.
 
 The `Test Samples` step in CI will rely on the resources provisioned and environment variables used for running the package's tests.
 

diff --git a/doc/dev/static_type_checking.md b/doc/dev/static_type_checking.md
@@ -175,17 +175,17 @@ version of the type checker ships. All client libraries in the Python SDK repo a
 
 The easiest way to install and run the type checkers locally is
 with [tox](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md#tox). This reproduces the exact type checking
-environment run in CI and brings in the third party stub packages necessary. To begin, first install `tox` and `tox-monorepo`:
+environment run in CI and brings in the third party stub packages necessary. To begin, first install `tox`:
 
-`pip install tox tox-monorepo`
+`pip install tox<5`
 
 ### Run mypy
 
 mypy is currently pinned to version [1.0.0](https://pypi.org/project/mypy/1.0.0/).
 
 To run mypy on your library, run the tox mypy env at the package level:
 
-`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e mypy -c ../../../eng/tox/tox.ini`
+`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e mypy -c ../../../eng/tox/tox.ini --root .`
 
 If you don't want to use `tox` you can also install and run mypy on its own:
 
@@ -217,7 +217,7 @@ Note that pyright requires that node is installed. The command-line [wrapper pac
 
 To run pyright on your library, run the tox pyright env at the package level:
 
-`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e pyright -c ../../../eng/tox/tox.ini`
+`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e pyright -c ../../../eng/tox/tox.ini --root .`
 
 If you don't want to use `tox` you can also install and run pyright on its own:
 
@@ -249,7 +249,7 @@ from the code in the PR vs. the latest release on PyPi.
 
 To run verifytypes on your library, run the tox verifytypes env at the package level:
 
-`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e verifytypes -c ../../../eng/tox/tox.ini`
+`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e verifytypes -c ../../../eng/tox/tox.ini --root .`
 
 If you don't want to use `tox` you can also install and run pyright/verifytypes on its own:
 

diff --git a/doc/dev/tests.md b/doc/dev/tests.md
@@ -123,20 +123,20 @@ If you have print statements in your tests for debugging you can add the `-s` fl
 
 ## Tox
 
-The Python SDK uses the [tox project](https://tox.readthedocs.io/en/latest/) to automate releases, run tests, run linters, and build our documentation. The `tox.ini` file is located at `azure-sdk-for-python/eng/tox/tox.ini` for reference. You do not need to make any changes to the tox file for tox to work with your project. Tox will create a directory (`.tox`) in the head of your branch. The first time you run tox commands it may take several moments, but subsequent runs will be quicker. To install tox run the following command from within your virtual environment.
-`(env) > pip install tox tox-monorepo`.
+The Python SDK uses the [tox project](https://tox.wiki/en/latest/) to automate releases, run tests, run linters, and build our documentation. The `tox.ini` file is located at `azure-sdk-for-python/eng/tox/tox.ini` for reference. You do not need to make any changes to the tox file for tox to work with your project. Tox will create a directory (`.tox`) in the head of your branch. The first time you run tox commands it may take several moments, but subsequent runs will be quicker. To install tox run the following command from within your virtual environment.
+`(env) > pip install tox<5`.
 
 To run a tox command from your directory use the following commands:
 ```cmd
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e sphinx
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e pylint
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e mypy
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e pyright
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e verifytypes
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e whl
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e sdist
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e samples
-(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e apistub
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e sphinx -c ../../../eng/tox/tox.ini --root . 
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e pylint -c ../../../eng/tox/tox.ini --root . 
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e mypy -c ../../../eng/tox/tox.ini --root . 
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e pyright -c ../../../eng/tox/tox.ini --root . 
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e verifytypes -c ../../../eng/tox/tox.ini --root . 
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e whl -c ../../../eng/tox/tox.ini --root . 
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e sdist -c ../../../eng/tox/tox.ini --root . 
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e samples -c ../../../eng/tox/tox.ini --root . 
+(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e apistub -c ../../../eng/tox/tox.ini --root . 
 ```
 A quick description of the five commands above:
 * sphinx: documentation generation using the inline comments written in our code