test: Remove 'src' from pytest test testpaths to allow for non-editab…

…le install in CI (#1467) * Remove 'src' from the pytest testpaths in pyproject.toml * Remove '--editable' pip install option from CI * Use 'pytest' over 'python -m pytest' to avoid adding the current directory to sys.path - c.f. https://docs.pytest.org/en/latest/how-to/usage.html#calling-pytest-through-python-m-pytest * Split doctest test coverage into separate run and report * For backend consistency, specify a backend in README examples - Also truncate prints of CLs values to finite precision * Split pytest addopts across multiple lines in pyproject.toml for readability
scikit-hep · Oct 19, 2021 · 8a3dc6b · 8a3dc6b
1 parent be5ae65
commit 8a3dc6b
Show file tree

Hide file tree

Showing 9 changed files with 54 additions and 34 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -40,14 +40,14 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip install --upgrade --editable .[test]
+        python -m pip install --upgrade .[test]
 
     - name: List installed Python packages
       run: python -m pip list
 
     - name: Test with pytest
       run: |
-        python -m pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
+        pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
 
     - name: Report core project coverage with Codecov
       if: github.event_name != 'schedule' && matrix.python-version == 3.9 && matrix.os == 'ubuntu-latest'
@@ -58,7 +58,7 @@ jobs:
 
     - name: Test Contrib module with pytest
       run: |
-        python -m pytest -r sx tests/contrib --mpl --mpl-baseline-path tests/contrib/baseline
+        pytest -r sx tests/contrib --mpl --mpl-baseline-path tests/contrib/baseline
 
     - name: Report contrib coverage with Codecov
       if: github.event_name != 'schedule' && matrix.python-version == 3.9 && matrix.os == 'ubuntu-latest'
@@ -67,7 +67,17 @@ jobs:
         files: ./coverage.xml
         flags: contrib
 
+    - name: Test docstring examples with doctest
+      run: pytest -r sx src/ README.rst
+
+    - name: Report doctest coverage with Codecov
+      if: github.event_name != 'schedule' && matrix.python-version == 3.9 && matrix.os == 'ubuntu-latest'
+      uses: codecov/codecov-action@v2
+      with:
+        files: ./coverage.xml
+        flags: doctest
+
     - name: Run benchmarks
       if: github.event_name == 'schedule' && matrix.python-version == 3.9
       run: |
-        python -m pytest -r sx --benchmark-sort=mean tests/benchmarks/test_benchmark.py
+        pytest -r sx --benchmark-sort=mean tests/benchmarks/test_benchmark.py
diff --git a/.github/workflows/dependencies-head.yml b/.github/workflows/dependencies-head.yml
@@ -28,14 +28,14 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip --no-cache-dir --quiet install --upgrade --editable .[test]
+        python -m pip --no-cache-dir --quiet install --upgrade .[test]
         python -m pip uninstall --yes scipy
         python -m pip install --upgrade --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy
         python -m pip list
 
     - name: Test with pytest
       run: |
-        python -m pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
+        pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
 
   iminuit:
 
@@ -54,14 +54,14 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip --no-cache-dir --quiet install --upgrade --editable .[test]
+        python -m pip --no-cache-dir --quiet install --upgrade .[test]
         python -m pip uninstall --yes iminuit
         python -m pip install --upgrade cython
         python -m pip install --upgrade git+git://github.com/scikit-hep/iminuit.git
         python -m pip list
     - name: Test with pytest
       run: |
-        python -m pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
+        pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
 
   uproot4:
 
@@ -80,13 +80,13 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip --no-cache-dir --quiet install --upgrade --editable .[test]
+        python -m pip --no-cache-dir --quiet install --upgrade .[test]
         python -m pip uninstall --yes uproot
         python -m pip install --upgrade git+git://github.com/scikit-hep/uproot4.git
         python -m pip list
     - name: Test with pytest
       run: |
-        python -m pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
+        pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
 
   pytest:
 
@@ -105,10 +105,10 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip --no-cache-dir --quiet install --upgrade --editable .[test]
+        python -m pip --no-cache-dir --quiet install --upgrade .[test]
         python -m pip uninstall --yes pytest
         python -m pip install --upgrade git+git://github.com/pytest-dev/pytest.git
         python -m pip list
     - name: Test with pytest
       run: |
-        python -m pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
+        pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -26,7 +26,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip --quiet --use-feature=in-tree-build install --upgrade .[docs,test]
+        python -m pip --quiet install --upgrade .[docs,test]
         python -m pip install yq
         python -m pip list
         sudo apt-get update

diff --git a/.github/workflows/lower-bound-requirements.yml b/.github/workflows/lower-bound-requirements.yml
@@ -28,11 +28,11 @@ jobs:
       run: |
         python -m pip install --upgrade pip setuptools wheel
         python -m pip --no-cache-dir --quiet install --requirement lower-bound-requirements.txt
-        python -m pip --no-cache-dir --quiet install --editable .[test]
+        python -m pip --no-cache-dir --quiet install .[test]
         python -m pip install --requirement lower-bound-requirements.txt
         python -m pip list
 
     - name: Test with pytest
       run: |
         # Run on tests/ to skip doctests of src given examples are for latest APIs
-        python -m pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py tests/
+        pytest -r sx --ignore tests/benchmarks/ --ignore tests/contrib --ignore tests/test_notebooks.py tests/
diff --git a/.github/workflows/notebooks.yml b/.github/workflows/notebooks.yml
@@ -23,8 +23,8 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip setuptools wheel
-        python -m pip --quiet install --upgrade --editable .[complete]
+        python -m pip --quiet install --upgrade .[complete]
         python -m pip list
     - name: Test example notebooks
       run: |
-        python -m pytest -r sx tests/test_notebooks.py
+        pytest -r sx tests/test_notebooks.py
diff --git a/.github/workflows/release_tests.yml b/.github/workflows/release_tests.yml
@@ -39,7 +39,7 @@ jobs:
 
     - name: Canary test public API
       run: |
-        python -m pytest -r sx tests/test_public_api.py
+        pytest -r sx tests/test_public_api.py
 
     - name: Verify requirements in codemeta.json
       run: |

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -54,13 +54,13 @@ We recommend first reading the "[Developing](https://scikit-hep.org/pyhf/develop
 
 You can install the development environment (which includes a number of extra) libraries and all others needed to run the tests via `pip`:
 
-```bash
+```console
 python -m pip install --upgrade --editable .[complete]
 ```
 
 To make the PR process much smoother we also strongly recommend that you setup the Git pre-commit hook for [Black](https://github.com/psf/black) by running
 
-```bash
+```console
 pre-commit install
 ```
 
@@ -70,15 +70,15 @@ This will run `black` over your code each time you attempt to make a commit and
 
 You can run the unit tests (which should be fast!) via the following command.
 
-```bash
-python -m pytest --ignore=tests/test_notebooks.py
+```console
+pytest --ignore=tests/test_notebooks.py
 ```
 
 Note: This ignores the notebook tests (which are run via [papermill](https://github.com/nteract/papermill) which run somewhat slow.
 Make sure to run the complete suite before submitting a PR
 
-```bash
-python -m pytest
+```console
+pytest
 ```
 
 ## Making a pull request
@@ -89,6 +89,6 @@ We try to follow [Conventional Commit](https://www.conventionalcommits.org/) for
 
 New baseline visuals can be generated using this command:
 
-```bash
-python -m pytest tests/contrib/test_viz.py --mpl-generate-path=tests/contrib/baseline
+```console
+pytest tests/contrib/test_viz.py --mpl-generate-path=tests/contrib/baseline
 ```
diff --git a/README.rst b/README.rst
@@ -40,6 +40,7 @@ This is how you use the ``pyhf`` Python API to build a statistical model and run
 .. code:: pycon
 
    >>> import pyhf
+   >>> pyhf.set_backend("numpy")
    >>> model = pyhf.simplemodels.uncorrelated_background(
    ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
    ... )
@@ -48,24 +49,25 @@ This is how you use the ``pyhf`` Python API to build a statistical model and run
    >>> CLs_obs, CLs_exp = pyhf.infer.hypotest(
    ...     test_mu, data, model, test_stat="qtilde", return_expected=True
    ... )
-   >>> print(f"Observed: {CLs_obs}, Expected: {CLs_exp}")
-   Observed: 0.05251497423736956, Expected: 0.06445320535890459
+   >>> print(f"Observed: {CLs_obs:.9f}, Expected: {CLs_exp:.9f}")
+   Observed: 0.052514974, Expected: 0.064453205
 
 Alternatively the statistical model and observational data can be read from its serialized JSON representation (see next section).
 
 .. code:: pycon
 
    >>> import pyhf
    >>> import requests
+   >>> pyhf.set_backend("numpy")
    >>> wspace = pyhf.Workspace(requests.get("https://git.io/JJYDE").json())
    >>> model = wspace.model()
    >>> data = wspace.data(model)
    >>> test_mu = 1.0
    >>> CLs_obs, CLs_exp = pyhf.infer.hypotest(
    ...     test_mu, data, model, test_stat="qtilde", return_expected=True
    ... )
-   >>> print(f"Observed: {CLs_obs}, Expected: {CLs_exp}")
-   Observed: 0.3599840922126626, Expected: 0.3599840922126626
+   >>> print(f"Observed: {CLs_obs:.9f}, Expected: {CLs_exp:.9f}")
+   Observed: 0.359984092, Expected: 0.359984092
 
 
 Finally, you can also use the command line interface that ``pyhf`` provides

diff --git a/pyproject.toml b/pyproject.toml
@@ -41,11 +41,19 @@ ignore = [
 
 [tool.pytest.ini_options]
 minversion = "6.0"
-addopts = "--ignore=setup.py --ignore=validation/ --ignore=binder/ --ignore=docs/ --cov=pyhf --cov-report=term-missing --cov-config=.coveragerc --cov-report xml --doctest-modules --doctest-glob='*.rst'"
-testpaths = [
-    "src",
-    "tests",
+addopts = [
+    "--ignore=setup.py",
+    "--ignore=validation/",
+    "--ignore=binder/",
+    "--ignore=docs/",
+    "--cov=pyhf",
+    "--cov-report=term-missing",
+    "--cov-config=.coveragerc",
+    "--cov-report=xml",
+    "--doctest-modules",
+    "--doctest-glob='*.rst'"
 ]
+testpaths = "tests"
 markers = [
     "fail_jax",
     "fail_numpy",