Tox Workflow (Azure#6624)

* Swapping CI to Leverage Tox Workflow
florentx · Aug 27, 2019 · 21e442f · 21e442f
1 parent 1741c19
commit 21e442f
Show file tree

Hide file tree

Showing 34 changed files with 1,273 additions and 186 deletions.
diff --git a/.gitignore b/.gitignore
@@ -29,6 +29,16 @@ build/
 # Test results
 TestResults/
 
+# tox generated artifacts
+test-junit-*.xml
+pylint-*.out.txt
+coverage-*.xml
+stderr.txt
+stdout.txt
+
+# tox environment folders
+.tox/
+
 # Credentials
 credentials_real.json
 testsettings_local.json

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -3,6 +3,114 @@
 If you would like to become an active contributor to this project please
 follow the instructions provided in [Microsoft Azure Projects Contribution Guidelines](http://azure.github.io/guidelines/).
 
+## Building and Testing
+
+The Azure SDK team's Python CI leverages the tool `tox` to distribute tests to virtual environments, handle test dependency installation, and coordinate tooling reporting during PR/CI builds. This means that a dev working locally can reproduce _exactly_ what the build machine is doing. 
+
+[A Brief Overview of Tox](https://tox.readthedocs.io/en/latest/)
+
+#### A Monorepo and Tox in Harmony
+
+Traditionally, the `tox.ini` file for a package sits _alongside the setup.py_ in source code. The `azure-sdk-for-python` necessarily does not adhere to this policy. There are over one-hundred packages contained here-in. That's a lot of `tox.ini` files to maintain!
+
+Instead, the CI system leverages an tox plugin called `tox-monorepo`. This plugin allows `tox` to act as if the `tox.ini` is located in whatever directory you executed tox in!
+
+#### Tox Environments
+
+A given `tox.ini` works on the concept of `test environments`. A given test environment is a combination of:
+
+1. An identifier (or identifiers)
+2. A targeted Python version 
+    1. `tox` will default to base python executing the `tox` command if no Python environment is specified
+3. (optionally) an OS platform
+
+Internally `tox` leverages `virtualenv` to create each test environment's virtual environment. 
+
+This means that once the `tox` workflow is in place, all tests will be executed _within a virtual environment._
+
+To see the default environments from a specific `tox.ini` file, use the command `tox -l` in the same directory as the file itself.
+
+> sdk-for-python/eng/tox> tox -l
+
+```
+
+whl
+sdist
+
+```
+
+Unfortunately, the command `tox -l` only returns the _default_ test builds. The common `tox.ini` file also supports `lint` and `mypy` environments.
+
+### Example Usage of the common Azure SDK For Python `tox.ini` 
+
+Basic usage of `tox` within this monorepo is:
+
+1. `pip install tox tox-monorepo`
+2. `cd` to target package folder
+3. run `tox -c path/to/tox.ini`
+
+The common `tox.ini` location is `eng/tox/tox.ini` within the repository.
+
+If at any time you want to blow away the tox created virtual environments and start over, simply append `-r` to any tox invocation! 
+
+#### Example `azure-core` mypy
+
+1. `cd` to `sdk/core/azure-core`
+2. Run `tox -e mypy -c ../../../eng/tox/tox.ini`
+
+#### Example `azure-storage-blob` tests
+
+1. `cd` to `sdk/storage/azure-storage-blob`
+2. Execute `tox -c ../../../eng/tox/tox.ini`
+
+Note that we didn't provide an `environment` argument for this example. Reason here is that the _default_ environment selected by our common `tox.ini` file is one that runs `pytest`.
+
+#### `*wheel_tests` environments
+Used for test execution across the spectrum of all the platforms we want to support. Maintained at a `platform specific` level just in case we run into platform-specific bugs.
+
+* Installs the wheel, runs tests using the wheel
+
+```
+\> tox -e whl -c <path to tox.ini>
+
+```
+
+#### `sdist` environment
+Used for the local dev loop.
+
+* Installs package in editable mode
+* Runs tests using the editable mode installation, not the wheel
+
+```
+
+\> tox -e sdist -c <path to tox.ini>
+
+```
+
+#### `lint` environment
+Pylint install and run.
+
+```
+\> tox -e lint -c <path to tox.ini>
+```
+
+
+#### `mypy` environment
+Mypy install and run.
+
+```
+\> tox -e mypy -c <path to tox.ini>
+```
+
+### Custom Pytest Arguments
+
+`tox` supports custom arguments, and the defined pytest environments within the common `tox.ini` also allow these. Essentially, separate the arguments you want passed to `pytest` by a `--` in your tox invocation.
+
+[Tox Documentation on Positional Arguments](https://tox.readthedocs.io/en/latest/example/general.html#interactively-passing-positional-arguments)
+
+**Example: Invoke tox, breaking into the debugger on failure**
+`tox -e whl -c ../../../eng/tox/tox.ini -- --pdb`
+
 ## Code of Conduct
 This project's code of conduct can be found in the
 [CODE_OF_CONDUCT.md file](https://github.com/Azure/azure-sdk-for-python/blob/master/CODE_OF_CONDUCT.md)

diff --git a/build_package.py b/build_package.py
@@ -19,7 +19,7 @@ def create_package(name, dest_folder=DEFAULT_DEST_FOLDER):
     absdirs = [os.path.dirname(package) for package in (glob.glob('{}/setup.py'.format(name)) + glob.glob('sdk/*/{}/setup.py'.format(name)))]
 
     absdirpath = os.path.abspath(absdirs[0])
-    check_call(['python', 'setup.py', 'bdist_wheel', '-d', dest_folder], cwd=absdirpath)
+    check_call(['python', 'setup.py', 'bdist_wheel', '--universal', '-d', dest_folder], cwd=absdirpath)
     check_call(['python', 'setup.py', "sdist", "--format", "zip", '-d', dest_folder], cwd=absdirpath)
 
 

diff --git a/doc/dev/engineering_assumptions.md b/doc/dev/engineering_assumptions.md
@@ -0,0 +1,17 @@
+## Engineering System Assumptions, Gotchas, and Minutiae
+
+1. All wheels are generated with `--universal` flag passed to `bdist_wheel`. We do not currently allow non-universal packages to be shipped out of this repository.
+
+## Build
+
+Build CI for `azure-sdk-for-python` essentially builds and tests packages in one of two methodologies.
+
+### Individual Packages
+1. Leverage `tox` to create wheel, install, and execute tests against newly installed wheel
+2. Tests each package in isolation (outside of dev_requirements.txt dependencies + necessary `pylint` and `mypy`)
+
+### Global Method
+
+1. Install on packages (and their dev_requirements!) in one go.
+2. Run `pytest <folder1>, pytest <folder2>` where folders correspond to package folders
+    1. While all packages are installed alongside each other, each test run is individual to the package. This has the benefit of not allowing `packageA`'s `conftest.py` to mess with `packageB`'s environment.'
diff --git a/eng/pipelines/client.yml b/eng/pipelines/client.yml
@@ -8,12 +8,12 @@ variables:
     value: 'not cosmosEmulator'
 
 jobs:
-  - template: templates/jobs/archetype-sdk-client.yml
+  - template: templates/jobs/archetype-sdk-nightly.yml
     parameters:
       ServiceDirectory: ''
       TestMarkArgument: $(TestMarkArgument)
       BuildTargetingString: $(BuildTargetingString)
-      BeforeBuildJobPublishBuildArtifacts:
+      BeforePublishSteps:
         - script: 'echo "##vso[build.addbuildtag]$(ReleaseTag)"'
           displayName: 'Tag The Build'
           condition: eq(variables['CandidateForRelease'], True)
diff --git a/eng/pipelines/templates/jobs/archetype-sdk-client.yml b/eng/pipelines/templates/jobs/archetype-sdk-client.yml
@@ -1,6 +1,6 @@
 parameters:
   ServiceDirectory: ''
-  BeforeBuildJobPublishBuildArtifacts: []
+  BeforePublishSteps: []
   TestMarkArgument: ''
   BuildTargetingString: 'azure-*'
 
@@ -13,37 +13,11 @@ jobs:
       vmImage: 'ubuntu-16.04'
 
     steps:
-      - script: |
-          echo "##vso[build.addbuildtag]Scheduled"
-        displayName: 'Tag scheduled builds'
-        condition: and(eq(variables['Build.SourceBranchName'],'master'),eq(variables['Build.Reason'],'Schedule'))
-
-      - task: UsePythonVersion@0
-        displayName: 'Use Python $(PythonVersion)'
-        inputs:
-          versionSpec: $(PythonVersion)
-
-      - script: |
-          pip install wheel setuptools pathlib twine readme-renderer[md]
-        displayName: 'Prep Environment'
-
-      - task: PythonScript@0
-        displayName: 'Generate Packages'
-        inputs:
-          scriptPath: 'scripts/devops_tasks/build_packages.py'
-          arguments: '-d "$(Build.ArtifactStagingDirectory)" "${{ parameters.BuildTargetingString }}" --service=${{parameters.ServiceDirectory}}'
-
-      - script: |
-          twine check $(Build.ArtifactStagingDirectory)/*
-        displayName: 'Verify Readme'
-
-      - ${{ parameters.BeforeBuildJobPublishBuildArtifacts }}
-
-      - task: PublishBuildArtifacts@1
-        condition: succeededOrFailed()
-        displayName: 'Publish Artifacts'
-        inputs:
-          artifactName: packages
+    - template: ../steps/build-artifacts.yml
+      parameters: 
+        ServiceDirectory: ${{ parameters.ServiceDirectory }}
+        BuildTargetingString: ${{ parameters.BuildTargetingString }}
+        BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
 
   - job: 'Analyze'
     variables:
@@ -55,53 +29,22 @@ jobs:
     pool:
       vmImage: 'ubuntu-16.04'
 
-    steps:
-      - task: UsePythonVersion@0
-        displayName: 'Use Python $(PythonVersion)'
-        inputs:
-          versionSpec: '$(PythonVersion)'
-
-      - script: |
-          pip install setuptools wheel Jinja2
-          pip install doc-warden==0.3.0
-          ward scan -d $(Build.SourcesDirectory) -c $(Build.SourcesDirectory)/.docsettings.yml
-        displayName: 'Verify Readmes'
-
-      - task: PythonScript@0
-        displayName: 'Analyze dependencies'
-        inputs:
-          scriptPath: 'scripts/analyze_deps.py'
-          arguments: '--verbose --out "$(Build.ArtifactStagingDirectory)/dependencies.html"'
-
-      - task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0
-        # ComponentGovernance is currently unable to run on pull requests of public projects. Running on non-PR
-        # builds should be sufficient.
-        condition: and(succeededOrFailed(), ne(variables['Build.Reason'],'PullRequest'))
-        displayName: 'Component Detection'
-
-      - task: PublishBuildArtifacts@1
-        condition: succeededOrFailed()
-        displayName: 'Publish Report Artifacts'
-        inputs:
-          artifactName: reports
+      steps:
+      - template: ../steps/analyze.yml
 
   - job: 'Generic_Tests'
     variables:
     - template: ../variables/globals.yml
 
     dependsOn:
-      - 'Build'
+       - 'Build'
 
     strategy:
       matrix:
         Linux_Python27:
           OSName: 'Linux'
           OSVmImage: 'ubuntu-16.04'
           PythonVersion: '2.7'
-        Linux_Python34:
-          OSName: 'Linux'
-          OSVmImage: 'ubuntu-16.04'
-          PythonVersion: '3.4'
         Linux_Python35:
           OSName: 'Linux'
           OSVmImage: 'ubuntu-16.04'
@@ -127,28 +70,33 @@ jobs:
       vmImage: '$(OSVmImage)'
 
     steps:
-      - template: ../steps/build-test.yml
-        parameters:
-          ServiceDirectory: ${{ parameters.ServiceDirectory }}
-          TestMarkArgument: ${{ parameters.TestMarkArgument }}
-          OSName: $(OSName)
-          PythonVersion: $(PythonVersion)
-          BuildTargetingString: ${{ parameters.BuildTargetingString }}
-
-      - template: ../steps/publish-coverage.yml
-
-
-# Run PyPy tests without coverage
+    - template: ../steps/build-test.yml
+      parameters:
+        ServiceDirectory: ${{ parameters.ServiceDirectory }}
+        TestMarkArgument: ${{ parameters.TestMarkArgument }}
+        AdditionalTestArgs: '--wheel_dir="$(Build.ArtifactStagingDirectory)"'
+        OSName: $(OSName)
+        PythonVersion: $(PythonVersion)
+        BuildTargetingString: ${{ parameters.BuildTargetingString }}
+        ToxTestEnv: 'whl,sdist'
+        BeforeTestSteps: 
+          - task: DownloadPipelineArtifact@0
+            inputs:
+              artifactName: 'artifacts' 
+              targetPath: $(Build.ArtifactStagingDirectory)
+    - template: ../steps/publish-coverage.yml
+
+  # Run PyPy tests without coverage
   - job: 'Test_PyPy'
 
     variables:
-      - template: ../variables/globals.yml
-      - name: OSName
-        value: 'Linux'
-      - name: PythonVersion
-        value: 'pypy3'
-      - name: OSVmImage
-        value: ubuntu-16.04
+    - template: ../variables/globals.yml
+    - name: OSName
+      value: 'Linux'
+    - name: PythonVersion
+      value: 'pypy3'
+    - name: OSVmImage
+      value: ubuntu-16.04
 
     dependsOn:
       - 'Build'
@@ -157,11 +105,17 @@ jobs:
       vmImage: $(OSVmImage)
 
     steps:
-      - template: ../steps/build-test.yml
-        parameters:
-          AdditionalTestArgs: '--disablecov'
-          ServiceDirectory: ${{ parameters.ServiceDirectory }}
-          TestMarkArgument: ${{ parameters.TestMarkArgument }}
-          OSName: $(OSName)
-          PythonVersion: $(PythonVersion)
-          BuildTargetingString: ${{ parameters.BuildTargetingString }}
+    - template: ../steps/build-test.yml
+      parameters:
+        AdditionalTestArgs: '--disablecov --wheel_dir="$(Build.ArtifactStagingDirectory)"'
+        ServiceDirectory: ${{ parameters.ServiceDirectory }}
+        TestMarkArgument: ${{ parameters.TestMarkArgument }}
+        OSName: $(OSName)
+        PythonVersion: $(PythonVersion)
+        BuildTargetingString: ${{ parameters.BuildTargetingString }}
+        ToxTestEnv: 'whl,sdist'
+        BeforeTestSteps: 
+          - task: DownloadPipelineArtifact@0
+            inputs:
+              artifactName: 'artifacts' 
+              targetPath: $(Build.ArtifactStagingDirectory)