Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[engsys] Migrate to tox4 #30159

Merged
merged 42 commits into from
May 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
1910a58
Remove references to {envbindir}
kdestin Apr 9, 2023
3a4449d
refactor: Write paths relative to git root, instead of package dir
kdestin Apr 9, 2023
e4d6043
refactor: Rename {toxinidir} to {tox_root}
kdestin Apr 9, 2023
c453c0b
refactor: Flatten some commands that are no longer super long
kdestin Apr 9, 2023
04a8b78
refactor: Make a pytest section
kdestin Apr 9, 2023
7d3da58
refactor: Remove `changedir = {tox_root}`
kdestin Apr 9, 2023
48291cd
fix: {distdir} is deprecated
kdestin Apr 9, 2023
c0fd18b
fix: Remove --ignore-installed from pip command
kdestin Apr 9, 2023
df6b428
refactor: Add descriptions to all tox environments
kdestin Apr 10, 2023
0280e05
fix: Remove ineffective platform config
kdestin Apr 10, 2023
359c9db
chore: Bump `tox` to `>4.4.11` and no longer install `tox-monorepo`
kdestin Apr 30, 2023
796e553
docs: Update CONTRIBUTING.md to use tox 4 commands
kdestin Apr 30, 2023
d4966e7
docs: Make `azure-media-{videoanalyzer,analytics}-edge`'s DevTips.md …
kdestin Apr 30, 2023
47e62a0
docs: Remove remaining references to tox-monorepo
kdestin Apr 30, 2023
ff88741
doc: Use `--root` throughout documentation
kdestin Apr 30, 2023
4f99e5a
engsys: Make tox_harness set `{toxinidir}` when invoking tox
kdestin Apr 30, 2023
236e179
engsys: Set required tox version in config
kdestin Apr 30, 2023
c27d324
docs: Use the new url for tox docs
kdestin Apr 30, 2023
e9929cb
docs: Add a callout that TOX_* environment variables control defaults
kdestin Apr 30, 2023
f3b64dc
update packaging requirement everywhere it is used.
scbedd May 1, 2023
a18e287
fix cspell
kdestin May 1, 2023
4efb01e
docs: Use `tox run` syntax throughout documentation
kdestin May 2, 2023
4cbd85a
engsys: Use `tox run` syntax in tox_harness.py
kdestin May 2, 2023
e958de7
docs: Replace – (U+2013 : EN DASH) with - (ASCII code 45)
kdestin May 2, 2023
0ec9074
update pytest requirements + coverage.
scbedd May 3, 2023
472ee62
update virtualenv
scbedd May 3, 2023
deedaba
more conflicts resolved
scbedd May 3, 2023
535294d
bringing in urllib3 restriction, aligning ci_tools.txt with test_tool…
scbedd May 3, 2023
19f6523
check_call requires a list of type string. if we append ['blah', 'bla…
scbedd May 4, 2023
1c72fff
engsys: Tox uses `run` for serial execution and `run-parallel` for pa…
kdestin May 4, 2023
53e6068
azure-core: Use `@pytest_asyncio.fixture` to mark an asyncio fixture
kdestin May 10, 2023
8edb70b
pin coverage to newer version with no conflicts
scbedd May 10, 2023
32cdba9
identify issue with missing code coverage being caused by wrongly nam…
scbedd May 10, 2023
f298007
remove fix of dotfile!
scbedd May 11, 2023
a6ad545
apply black formatting to resolve failing analyze step
scbedd May 11, 2023
2c2dd28
tox installing tox. probably not a good thing :)
scbedd May 11, 2023
c1ffbc9
engsys: Remove `tox` from `azure-media-analytics-edge` dev_requirements
kdestin May 12, 2023
5d6243e
align regression version of ci/test_tools.txt
scbedd May 12, 2023
9f74919
new coverage format requires that the package sources be present when…
scbedd May 17, 2023
c278ce5
ensure that missing hidden folder can't break cleanup operation
scbedd May 17, 2023
c297c9d
ensure coverage isn't generated on whl_no_aio
scbedd May 17, 2023
d506e99
ensure that coverage is generated without wonky package paths
scbedd May 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .vscode/cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,12 @@
"pytestmarkparametrize"
]
},
{
"filename": "doc/dev/pylint_checking.md",
"words": [
"pylintrc"
]
},
{
"filename": "tools/azure-sdk-tools/devtools_testutils/proxy_startup.py",
"words": [
Expand Down
76 changes: 49 additions & 27 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ If you want to contribute to a file that is generated (header contains `Code gen

The Azure SDK team's Python CI leverages the tool `tox` to distribute tests to virtual environments, handle test dependency installation, and coordinate tooling reporting during PR/CI builds. This means that a dev working locally can reproduce _exactly_ what the build machine is doing.

[A Brief Overview of Tox](https://tox.readthedocs.io/en/latest/)
[A Brief Overview of Tox](https://tox.wiki/en/latest/)

#### A Monorepo and Tox in Harmony

Traditionally, the `tox.ini` file for a package sits _alongside the setup.py_ in source code. The `azure-sdk-for-python` necessarily does not adhere to this policy. There are over one-hundred packages contained here-in. That's a lot of `tox.ini` files to maintain!

Instead, the CI system leverages an tox plugin called `tox-monorepo`. This plugin allows `tox` to act as if the `tox.ini` is located in whatever directory you executed tox in!
Instead, the CI system leverages the `--root` argument which is new to `tox4`. The `--root` argument allows `tox` to act as if the `tox.ini` is located in whatever directory you specify!

#### Tox Environments

Expand All @@ -32,40 +32,62 @@ Internally `tox` leverages `virtualenv` to create each test environment's virtua

This means that once the `tox` workflow is in place, all tests will be executed _within a virtual environment._

To see the default environments from a specific `tox.ini` file, use the command `tox -l` in the same directory as the file itself.

> sdk-for-python/eng/tox> tox -l

```

whl
sdist

You can use the command `tox list` to list all the environments provided by a `tox.ini` file. You can either use that command in the
same directory as the file itself, or use the `--conf` argument to specify the path to it directly.


Sample output of `tox list`:

```
sdk-for-python/eng/tox> tox list
default environments:
whl -> Builds a wheel and runs tests
sdist -> Builds a source distribution and runs tests

additional environments:
pylint -> Lints a package with a pinned version of pylint
next-pylint -> Lints a package with pylint (version 2.15.8)
mypy -> Typechecks a package with mypy (version 1.0.0)
next-mypy -> Typechecks a package with the latest version of mypy
pyright -> Typechecks a package with pyright (version 1.1.287)
next-pyright -> Typechecks a package with the latest version of static type-checker pyright
verifytypes -> Verifies the "type completeness" of a package with pyright
whl_no_aio -> Builds a wheel without aio and runs tests
develop -> Tests a package
sphinx -> Builds a package's documentation with sphinx
depends -> Ensures all modules in a target package can be successfully imported
verifywhl -> Verify directories included in whl and contents in manifest file
verifysdist -> Verify directories included in sdist and contents in manifest file. Also ensures that py.typed configuration is correct within the setup.py
devtest -> Tests a package against dependencies installed from a dev index
latestdependency -> Tests a package against the released, upper-bound versions of its azure dependencies
mindependency -> Tests a package against the released, lower-bound versions of its azure dependencies
apistub -> Generate an api stub of a package ( for https://apiview.dev )
bandit -> Runs bandit, a tool to find common security issues, against a package
samples -> Runs a package's samples
breaking -> Runs the breaking changes checker against a package
```

Unfortunately, the command `tox -l` only returns the _default_ test builds. The common `tox.ini` file also supports `pylint` and `mypy` environments.

### Example Usage of the common Azure SDK For Python `tox.ini`

Basic usage of `tox` within this monorepo is:

1. `pip install tox<4 tox-monorepo`
2. `cd` to target package folder
3. run `tox -c path/to/tox.ini`
1. `pip install tox<5`
2. Run `tox run -e ENV_NAME -c path/to/tox.ini --root path/to/python_package`
* **Note**: You can use environment variables to provide defaults for tox config values
* With `TOX_CONFIG_FILE` set to the absolute path of `tox.ini`, you can avoid needing `-c path/to/tox.ini` in your tox invocations
* With `TOX_ROOT_DIR` set to the absolute path to your python package, you can avoid needing `--root path/to/python_package`

The common `tox.ini` location is `eng/tox/tox.ini` within the repository.

If at any time you want to blow away the tox created virtual environments and start over, simply append `-r` to any tox invocation!

#### Example `azure-core` mypy

1. `cd` to `sdk/core/azure-core`
2. Run `tox -e mypy -c ../../../eng/tox/tox.ini`
1. Run `tox run -e mypy -c ../../../eng/tox/tox.ini --root sdk/core/azure-core`

#### Example `azure-storage-blob` tests

1. `cd` to `sdk/storage/azure-storage-blob`
2. Execute `tox -c ../../../eng/tox/tox.ini`
2. Execute `tox run -c ../../../eng/tox/tox.ini --root sdk/storage/azure-storage-blob`

Note that we didn't provide an `environment` argument for this example. Reason here is that the _default_ environment selected by our common `tox.ini` file is one that runs `pytest`.

Expand All @@ -75,7 +97,7 @@ Used for test execution across the spectrum of all the platforms we want to supp
* Installs the wheel, runs tests using the wheel

```
\> tox -e whl -c <path to tox.ini>
\> tox run -e whl -c <path to tox.ini> --root <path to python package>

```

Expand All @@ -87,30 +109,30 @@ Used for the local dev loop.

```

\> tox -e sdist -c <path to tox.ini>
\> tox run -e sdist -c <path to tox.ini> --root <path to python package>

```

#### `pylint` environment
Pylint install and run.

```
\> tox -e pylint -c <path to tox.ini>
\> tox run -e pylint -c <path to tox.ini> --root <path to python package>
```


#### `mypy` environment
Mypy install and run.

```
\> tox -e mypy -c <path to tox.ini>
\> tox run -e mypy -c <path to tox.ini> --root <path to python package>
```

#### `sphinx` environment
Generate sphinx doc for this package.

```
\> tox -e sphinx -c <path to tox.ini>
\> tox run -e sphinx -c <path to tox.ini> --root <path to python package>
```

### Custom Pytest Arguments
Expand All @@ -120,7 +142,7 @@ Generate sphinx doc for this package.
[Tox Documentation on Positional Arguments](https://tox.wiki/en/latest/config.html#substitutions-for-positional-arguments-in-commands)

**Example: Invoke tox, breaking into the debugger on failure**
`tox -e whl -c ../../../eng/tox/tox.ini -- --pdb`
`tox run -e whl -c <path to tox.ini> --root <path to python package> -- --pdb`

### Performance Testing

Expand Down Expand Up @@ -150,7 +172,7 @@ a. cd to package root folder
b. run tox environment devtest

```
\> tox -e devtest -c <path to tox.ini>
\> tox run -e devtest -c <path to tox.ini> --root <path to python package>
```

This tox test( devtest) will fail if installed dependent packages are not dev build version.
Expand Down
7 changes: 3 additions & 4 deletions doc/dev/dev_setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,11 @@ or execute the various commands available in the toolbox.

4. Setup your development environment

Install the development requirements for a specific library (located in the `dev_requirements.txt` file at the root of the library), [Tox][tox], [Tox monorepo][tox_monorepo] and an editable install of your library. For example, to install requirements for `azure-ai-formrecognizer`:
Install the development requirements for a specific library (located in the `dev_requirements.txt` file at the root of the library), [Tox][tox] and an editable install of your library. For example, to install requirements for `azure-ai-formrecognizer`:
```
azure-sdk-for-python> cd sdk/formrecognizer/azure-ai-formrecognizer
azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install -r dev_requirements.txt
azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install tox tox-monorepo
azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install tox<5
azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install -e .
```

Expand All @@ -58,6 +58,5 @@ Follow the instructions in the [Perform one-time test proxy setup][proxy_setup]
[python_39]: https://www.microsoft.com/p/python-39/9p7qfqmjrfp7
[proxy_setup]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md#perform-one-time-test-proxy-setup
[tests]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md
[tox]: https://tox.readthedocs.io/en/latest/
[tox_monorepo]: https://pypi.org/project/tox-monorepo/
[tox]: https://tox.wiki/en/latest/
[virtual_environment]: https://docs.python.org/3/tutorial/venv.html
6 changes: 3 additions & 3 deletions doc/dev/pylint_checking.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ In the Azure SDK for Python repository, in addition to the standard pylint libra

One way to run pylint is to run at the package level with tox:

.../azure-sdk-for-python/sdk/eventgrid/azure-eventgrid>tox -e pylint -c ../../../eng/tox/tox.ini
.../azure-sdk-for-python/sdk/eventgrid/azure-eventgrid>tox run -e pylint -c ../../../eng/tox/tox.ini --root .

If you don't want to use tox, you can also install and run pylint on its own:

Expand Down Expand Up @@ -58,9 +58,9 @@ In addition to being a part of the CI, the custom pylint checkers are also integ

There is now a new step on the CI pipeline called `Run Pylint Next`. This is merely a duplicate of the `Run Pylint` step with the exception that `Run Pylint Next` uses the latest version of pylint and the latest version of the custom pylint checkers.

This next-pylint enviornment can also be run locally through tox:
This next-pylint environment can also be run locally through tox:

tox -e next-pylint -c ../../../eng/tox/tox.ini
tox run -e next-pylint -c ../../../eng/tox/tox.ini --root <path to python package>

The errors generated by the `Run Pylint Next` step will not break your weekly test pipelines, but make sure to fix the warnings so that your client library is up to date for the next pylint release.

Expand Down
4 changes: 2 additions & 2 deletions doc/dev/sample_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,13 @@ The given `START`/`END` keywords can be used in a [sphinx literalinclude][sphinx
[Literalinclude example][literalinclude]

The rendered code snippets are sensitive to the indentation in the sample file. Adjust the `dedent` accordingly to ensure the sample is captured accurately and not accidentally trimmed.
You can preview how published reference documentation will look by running [tox][tox]: `tox -e sphinx -c ../../../eng/tox/tox.ini`.
You can preview how published reference documentation will look by running [tox][tox]: `tox run -e sphinx -c ../../../eng/tox/tox.ini --root <path to python package>`.

## Test run samples in CI live tests
Per the [Python guidelines][snippet_guidelines], sample code and snippets should be test run in CI to ensure they remain functional. Samples should be run in the package's live test pipeline which is scheduled to run daily.
To ensure samples do get tested as part of regular CI runs, add these [lines][live_tests] to the package's tests.yml.

You can test this CI step locally first, by utilizing [tox][tox] and running `tox -e samples -c ../../../eng/tox/tox.ini` at the package-level.
You can test this CI step locally first, by utilizing [tox][tox] and running `tox run -e samples -c ../../../eng/tox/tox.ini --root <path to python package>`.

The `Test Samples` step in CI will rely on the resources provisioned and environment variables used for running the package's tests.

Expand Down
10 changes: 5 additions & 5 deletions doc/dev/static_type_checking.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,17 +175,17 @@ version of the type checker ships. All client libraries in the Python SDK repo a

The easiest way to install and run the type checkers locally is
with [tox](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md#tox). This reproduces the exact type checking
environment run in CI and brings in the third party stub packages necessary. To begin, first install `tox` and `tox-monorepo`:
environment run in CI and brings in the third party stub packages necessary. To begin, first install `tox`:

`pip install tox tox-monorepo`
`pip install tox<5`

### Run mypy

mypy is currently pinned to version [1.0.0](https://pypi.org/project/mypy/1.0.0/).

To run mypy on your library, run the tox mypy env at the package level:

`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e mypy -c ../../../eng/tox/tox.ini`
`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e mypy -c ../../../eng/tox/tox.ini --root .`

If you don't want to use `tox` you can also install and run mypy on its own:

Expand Down Expand Up @@ -217,7 +217,7 @@ Note that pyright requires that node is installed. The command-line [wrapper pac

To run pyright on your library, run the tox pyright env at the package level:

`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e pyright -c ../../../eng/tox/tox.ini`
`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e pyright -c ../../../eng/tox/tox.ini --root .`

If you don't want to use `tox` you can also install and run pyright on its own:

Expand Down Expand Up @@ -249,7 +249,7 @@ from the code in the PR vs. the latest release on PyPi.

To run verifytypes on your library, run the tox verifytypes env at the package level:

`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox -e verifytypes -c ../../../eng/tox/tox.ini`
`.../azure-sdk-for-python/sdk/textanalytics/azure-ai-textanalytics>tox run -e verifytypes -c ../../../eng/tox/tox.ini --root .`

If you don't want to use `tox` you can also install and run pyright/verifytypes on its own:

Expand Down
22 changes: 11 additions & 11 deletions doc/dev/tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,20 +123,20 @@ If you have print statements in your tests for debugging you can add the `-s` fl

## Tox

The Python SDK uses the [tox project](https://tox.readthedocs.io/en/latest/) to automate releases, run tests, run linters, and build our documentation. The `tox.ini` file is located at `azure-sdk-for-python/eng/tox/tox.ini` for reference. You do not need to make any changes to the tox file for tox to work with your project. Tox will create a directory (`.tox`) in the head of your branch. The first time you run tox commands it may take several moments, but subsequent runs will be quicker. To install tox run the following command from within your virtual environment.
`(env) > pip install tox tox-monorepo`.
The Python SDK uses the [tox project](https://tox.wiki/en/latest/) to automate releases, run tests, run linters, and build our documentation. The `tox.ini` file is located at `azure-sdk-for-python/eng/tox/tox.ini` for reference. You do not need to make any changes to the tox file for tox to work with your project. Tox will create a directory (`.tox`) in the head of your branch. The first time you run tox commands it may take several moments, but subsequent runs will be quicker. To install tox run the following command from within your virtual environment.
`(env) > pip install tox<5`.

To run a tox command from your directory use the following commands:
```cmd
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e sphinx
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e pylint
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e mypy
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e pyright
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e verifytypes
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e whl
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e sdist
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e samples
(env) azure-sdk-for-python\sdk\my-service\my-package> tox -c ../../../eng/tox/tox.ini -e apistub
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e sphinx -c ../../../eng/tox/tox.ini --root .
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e pylint -c ../../../eng/tox/tox.ini --root .
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e mypy -c ../../../eng/tox/tox.ini --root .
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e pyright -c ../../../eng/tox/tox.ini --root .
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e verifytypes -c ../../../eng/tox/tox.ini --root .
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e whl -c ../../../eng/tox/tox.ini --root .
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e sdist -c ../../../eng/tox/tox.ini --root .
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e samples -c ../../../eng/tox/tox.ini --root .
(env) azure-sdk-for-python\sdk\my-service\my-package> tox run -e apistub -c ../../../eng/tox/tox.ini --root .
```
A quick description of the five commands above:
* sphinx: documentation generation using the inline comments written in our code
Expand Down
Loading