Skip to content

Commit

Permalink
Update test pipelines. (IDSIA#799)
Browse files Browse the repository at this point in the history 
* Update test pipelines.

* Update dependencies.

* Update dev-requirements.

* Fix tests.

* Fix versions.

* Fix blake tests.

* Fix yaml deprecation warning.

* Fix tf test.

* Fix S3Observer test.

* Fix tinydb tests.

* Minor fixes.

* Minor fixes.

* Fix pre-commit-config.

* Fix pre-commit-config.

* Fix spelling mistake.

* Fix traceback filtering.

* Add FilteredTracebackException.

* Add missing traceback stack.

* Fix FilteredTracebackException __init__.

* Minor fix.

* Remove old version checks.

* Filter exc_value __cause__ and __context__ tracebacks.

Add test for chained errors.

* Make exc_value filterting recursive.

* Fix Traceback for Paython 3.6.

* Fix Traceback for Paython 3.6.

* Fix Traceback for Paython 3.6.

* Fix Traceback for Paython 3.6.

* Remove TracebackTypeP36.

* Add TracebackTypeP36 and override class.

* Fix version check.

* Change FilteredTracebackException fpr py3.6.

* Fix traceback filtering for py3.6.

* Fix traceback filtering for py3.6.

* Fix traceback filtering for py3.6.

* Fix traceback filtering for py3.6.

* Drop python 3.6.

* Add numpy 1.20 test.

* Fix.

* Update pytest version and update README.

* Update README.

* Update black.

* Reformat file storage observer test.

* Reformat test_exceptions.

* Update README.

* Add doc msg to FilteredTracebackException.
  • Loading branch information
@jnphilipp
jnphilipp authored Apr 27, 2021
1 parent df84414 commit 90b38f8
Show file tree
Hide file tree
Showing 19 changed files with 266 additions and 183 deletions.
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
fail_fast: true
repos:
- repo: https://github.com/psf/black
rev: stable
rev: 21.4b0
hooks:
- id: black
language_version: python3
Expand Down
14 changes: 12 additions & 2 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,16 @@ can automatically run the tests for various python versions like this:

tox

Update pyptest version
++++++++++++++++++++++

If you update or change the pytest version, the following files need to be changed:

- ``dev-requirements.txt``
- ``tox.ini``
- ``test/test_utils.py``
- ``setup.py``

Contributing
------------
If you find a bug, have a feature request or want to discuss something general you are welcome to open an
Expand Down Expand Up @@ -229,8 +239,8 @@ This project is released under the terms of the `MIT license <http://opensource.

Citing Sacred
-------------
`K. Greff, A. Klein, M. Chovanec, F. Hutter, and J. Schmidhuber, ‘The Sacred Infrastructure for Computational Research’,
in Proceedings of the 15th Python in Science Conference (SciPy 2017), Austin, Texas, 2017, pp. 49–56
`K. Greff, A. Klein, M. Chovanec, F. Hutter, and J. Schmidhuber, ‘The Sacred Infrastructure for Computational Research’,
in Proceedings of the 15th Python in Science Conference (SciPy 2017), Austin, Texas, 2017, pp. 49–56
<http://conference.scipy.org/proceedings/scipy2017/klaus_greff.html>`_.


Expand Down
124 changes: 68 additions & 56 deletions azure-pipelines.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,85 +3,97 @@ jobs:
- job: test
strategy:
matrix:
Windows py35:
IMAGE_NAME: 'vs2017-win2016'
PYTHON_VERSION: '3.5'
TOX_CMD: 'py35'
Windows py36:
IMAGE_NAME: 'vs2017-win2016'
PYTHON_VERSION: '3.6'
TOX_CMD: 'py36'
Windows py37:
IMAGE_NAME: 'vs2017-win2016'
IMAGE_NAME: 'windows-latest'
PYTHON_VERSION: '3.7'
TOX_CMD: 'py37'
OSX py35:
IMAGE_NAME: 'macos-10.15'
PYTHON_VERSION: '3.5'
TOX_CMD: 'py35'
OSX py36:
IMAGE_NAME: 'macos-10.15'
PYTHON_VERSION: '3.6'
TOX_CMD: 'py36'
Windows py38:
IMAGE_NAME: 'windows-latest'
PYTHON_VERSION: '3.8'
TOX_CMD: 'py38'
Windows py39:
IMAGE_NAME: 'windows-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'py39'
OSX py37:
IMAGE_NAME: 'macos-10.15'
IMAGE_NAME: 'macOS-latest'
PYTHON_VERSION: '3.7'
TOX_CMD: 'py37'
Linux py35:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.5'
TOX_CMD: 'py35'
Linux py36:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.6'
TOX_CMD: 'py36'
OSX py38:
IMAGE_NAME: 'macOS-latest'
PYTHON_VERSION: '3.8'
TOX_CMD: 'py38'
OSX py39:
IMAGE_NAME: 'macOS-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'py39'
Linux py37:
IMAGE_NAME: 'ubuntu-16.04'
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.7'
TOX_CMD: 'py37'
Linux py38:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.8'
TOX_CMD: 'py38'
Linux py39:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'py39'
Linux numpy_117:
IMAGE_NAME: 'ubuntu-16.04'
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.7'
TOX_CMD: 'numpy-117'
Linux numpy_118:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.7'
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.8'
TOX_CMD: 'numpy-118'
Linux numpy_119:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.7'
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'numpy-119'
Linux tensorflow_112:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.6'
TOX_CMD: 'tensorflow-112'
Linux tensorflow_113:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.6'
TOX_CMD: 'tensorflow-113'
Linux tensorflow-114:
IMAGE_NAME: 'ubuntu-16.04'
Linux numpy_120:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'numpy-120'
Linux tensorflow_115:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.7'
TOX_CMD: 'tensorflow-114'
Linux tensorflow_2:
IMAGE_NAME: 'ubuntu-16.04'
TOX_CMD: 'tensorflow-115'
Linux tensorflow_20:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.7'
TOX_CMD: 'tensorflow-2'
Linux setup:
IMAGE_NAME: 'ubuntu-16.04'
TOX_CMD: 'tensorflow-20'
Linux tensorflow-21:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.7'
TOX_CMD: 'tensorflow-21'
Linux tensorflow_22:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.8'
TOX_CMD: 'tensorflow-22'
Linux tensorflow_23:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.8'
TOX_CMD: 'tensorflow-23'
Linux tensorflow_24:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.8'
TOX_CMD: 'tensorflow-24'
Linux setup:
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'setup'
Linux flake8:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.7'
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'flake8'
Linux black:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.6'
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'black'
Linux coverage:
IMAGE_NAME: 'ubuntu-16.04'
PYTHON_VERSION: '3.7'
IMAGE_NAME: 'ubuntu-latest'
PYTHON_VERSION: '3.9'
TOX_CMD: 'coverage'
pool:
vmImage: $(IMAGE_NAME)
Expand All @@ -93,9 +105,9 @@ jobs:
- bash: tox -e $TOX_CMD
- job: test_pre_commit
pool:
vmImage: 'ubuntu-16.04'
vmImage: 'ubuntu-latest'
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.6'
versionSpec: '3.9'
- bash: bash ./tests/check_pre_commit.sh
67 changes: 33 additions & 34 deletions dev-requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,34 +1,33 @@
colorama==0.4.1
docopt==0.6.2
gitdb2==2.0.0
GitPython==2.1.1
hashfs==0.7.0
jsonpickle==1.2
Mako==1.0.6
MarkupSafe==0.23
mock==2.0.0
mongomock==3.18.0
munch==2.0.4
packaging>=18.0
pandas==0.24.2
pbr==2.0.0
# tests/test_utils.py depends on that pytest version is exactly 4.3.0
pytest==4.3.0
python-dateutil==2.6.0
pytz==2016.10
PyYAML>=4.2b1
scandir==1.4
sentinels==1.0.0
smmap2==2.0.1
SQLAlchemy>=1.3.0
tinydb>=3.6
tinydb-serialization>=2.0.0
wrapt==1.10.8
scikit-learn==0.20.3
pymongo==3.9.0
py-cpuinfo==4.0
boto3>=1.9.0
moto==1.3.13
google-compute-engine>=2.8.0
google-cloud-storage==1.20.0
pre-commit==1.18.0
pytest==6.2.3 # tests/test_utils.py depends on that pytest version is exactly 6.2.3
colorama
docopt
gitdb2
GitPython
hashfs
jsonpickle
Mako
MarkupSafe
mock
mongomock
munch
packaging
pandas
pbr
python-dateutil
pytz
PyYAML
scandir
sentinels
smmap2
SQLAlchemy
tinydb
tinydb-serialization
wrapt
scikit-learn
pymongo
py-cpuinfo
boto3
moto
google-compute-engine>
google-cloud-storage
pre-commit
4 changes: 2 additions & 2 deletions pyproject.toml
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[tool.black]
target-version = ['py35']
target-version = ['py37', 'py38', 'py39']
include = '\.pyi?$'
exclude = '''
(
Expand All @@ -16,4 +16,4 @@ exclude = '''
| dist
)/
)
'''
'''
7 changes: 4 additions & 3 deletions sacred/observers/mongo.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ def __init__(
priority: int = DEFAULT_MONGO_PRIORITY,
client: Optional["pymongo.MongoClient"] = None,
failure_dir: Optional[PathType] = None,
**kwargs
**kwargs,
):
"""Initializer for MongoObserver.
Expand Down Expand Up @@ -623,14 +623,15 @@ def __init__(
overwrite: Optional[Union[int, str]] = None,
priority: int = DEFAULT_MONGO_PRIORITY,
client: Optional["pymongo.MongoClient"] = None,
**kwargs
**kwargs,
):
"""Initializer for MongoObserver.
Parameters
----------
interval
The interval in seconds at which the background thread is woken up to process new events.
The interval in seconds at which the background thread is woken up to
process new events.
retry_interval
The interval in seconds to wait if an event failed to be processed.
url
Expand Down
2 changes: 1 addition & 1 deletion sacred/observers/telegram_obs.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ def __init__(
chat_id,
silent_completion=False,
priority=DEFAULT_TELEGRAM_PRIORITY,
**kwargs
**kwargs,
):
self.silent_completion = silent_completion
self.chat_id = chat_id
Expand Down
7 changes: 1 addition & 6 deletions sacred/stdout_capturing.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -117,12 +117,7 @@ def tee_output_python():
@contextmanager
def tee_output_fd():
"""Duplicate stdout and stderr to a file on the file descriptor level."""
get_temp_file = (
lambda: NamedTemporaryFile(mode="w+", newline="")
if sys.version_info >= (3,)
else NamedTemporaryFile(mode="w+")
)
with get_temp_file() as target:
with NamedTemporaryFile(mode="w+", newline="") as target:
original_stdout_fd = 1
original_stderr_fd = 2
target_fd = target.fileno()
Expand Down
Loading

0 comments on commit 90b38f8

Please sign in to comment.