A simple demo package to practice creating python packages.
Inspired in this article: https://mathspp.com/blog/how-to-create-a-python-package-in-2022
The code implements a simple Rock, Paper, Scissors text-based game, loosely inspired in the one Al Sweigart wrote for Chapter 2 of Automate the boring stuff with Python.
Installation:
$ pip install mhered-test-pkg
Usage:
$ rps
Alternatively
$ python3 -m mhered_test_pkg
Check that the name is available in PyPI
Install poetry
(I started from here then modified the instructions):
$ curl -sSL https://install.python-poetry.org/ | python3 -
Create a local folder for the project and initialize poetry
inside
$ cd ~
$ mkdir mhered-test-pkg
$ cd mhered-test-pkg
$ poetry new .
Created package mhered_test_pkg in .
$ tree
.
├── mhered_test_pkg
│  └── __init__.py
├── pyproject.toml
├── README.rst
└── tests
├── __init__.py
└── test_mhered_test_pkg.py
Note: I renamed README.rst
to README.md
to work in Markdown.
$ mv README.rst README.md
Run poetry install
to create a file poetry.lock
with the dependencies:
$ poetry install
Create an empty github repo: mhered-test-pkg and follow the instructions to set it as the remote and push a first commit with the local file structure:
$ git init
$ git add *
$ git commit -m "First commit"
$ git branch -M main
$ git remote add origin https://github.com/mhered/mhered-test-pkg.git
$ git push -u origin main
Add pre-commit
as a development dependency, then commit updates:
$ poetry add -D pre-commit
$ git add poetry.lock pyproject.toml
$ git commit -m "Add pre-commit devt dependency."
Create a file .pre-commit-config.yaml
in the root:
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.0.1
hooks:
- id: check-toml
- id: check-yaml
- id: end-of-file-fixer
- id: mixed-line-ending
- repo: https://github.com/psf/black
rev: 22.3.0
hooks:
- id: black
- repo: https://github.com/PyCQA/isort
rev: 5.10.1
hooks:
- id: isort
args: ["--profile", "black"]
Activate the poetry
virtual environment to be able to use pre-commit
then install the hooks and run them once:
$ poetry shell
$ pre-commit install
$ pre-commit run --all-files
Note: pre-commit
is not found unless run from inside the shell - or you can use poetry run pre-commit
Commit the changes (including the updates to README.md
):
$ git add *
$ git commit -m "Run all pre-commits."
$ git push
Note: If some test fails the commit is cancelled and the files are automatically modified, so you need to add again the modified files and repeat the git commit - which is fine. However there is a strange behavior if the file was open: it seems to revert to an older version?
Add a license from the github repo then pull changes to local.
This will add LICENSE.md
Declare the test repository https://test.pypi.org in poetry
and name it testpypi
:
$ poetry config repositories.testpypi https://test.pypi.org/legacy/
Create an account on TestPyPI, go to Account Settings to get an API token and then configure poetry
to use it:
$ poetry config http-basic.testpypi __token__ pypi-YOUR-TESTPYPI-API-TOKEN
Note: Be careful not to expose your API token, e.g. I wrote it to a secrets.md
file then used .gitignore
so as not to commit and publish it publicly.
Build and upload the package:
$ poetry build
$ poetry publish -r testpypi
With this our package is live in TestPyPI: https://test.pypi.org/project/mhered-test-pkg/
Note: Build creates the dist/
folder that should be added to .gitignore
$ echo dist/ >> .gitignore
$ git add .
$ git commit -m "Publish to TestPyPI"
$ git push
For this example I wrote a simple Rock, Paper, Scissors game inspired and slightly refactored from the example proposed by Al Sweigart in his great book Automate the boring stuff with Python. The code goes in mhered-test-pkg/__init__.py
.
Add scriv for changelog management, as a development dependency with the [toml]
extra :
$ poetry add -D scriv[toml]
Configure scriv
to use Markdown and add version numbering in the title by adding the following lines to the pyproject.toml
file, refer to scriv's readthedocs:
[tool.scriv]
format = "md"
version = "literal: pyproject.toml: tool.poetry.version"
Then create the default directory for changelog fragments changelog.d/
, and add an empty .gitkeep
file so that git tracks the empty folder.
$ mkdir changelog.d
$ touch changelog.d/.gitkeep
$ git add pyproject.toml poetry.lock changelog.d/.gitkeep
$ git commit -m "Add scriv as devt dependency."
Create a new .md
fragment file in the changelog.d
folder:
$ scriv create
Edit it to add a description of the changes:
### Added
- A first simple implementation of Rock Paper Scissors
Update README.md
and commit everything:
$ git add README.md changelog.d/* __init__.py
$ git commit -m "Simple Rock Paper Scissors game"
Create a PyPI account and API token, and configure poetry
to use it:
$ poetry config pypi-token.pipy pypi-YOUR-PYPI-API-TOKEN
Build and publish:
$ poetry publish --build
Install, import and uninstall the package (outside of the shell) to check it works. Note: the hyphens (-
) in the package name turn into underscores (_
) in the module name.
$ pip install mhered-test-pkg
Defaulting to user installation because normal site-packages is not writeable
Collecting mhered-test-pkg
Using cached mhered_test_pkg-0.1.0-py3-none-any.whl (2.5 kB)
Installing collected packages: mhered-test-pkg
Successfully installed mhered-test-pkg-0.1.0
$ python3 -m mhered_test_pkg
ROCK, PAPER, SCISSORS
0 Wins, 0 Losses, 0 Ties
Enter your move: (r)ock (p)aper (s)cissors or (q)uit
r
ROCK versus... SCISSORS
You win!
1 Wins, 0 Losses, 0 Ties
Enter your move: (r)ock (p)aper (s)cissors or (q)uit
q
Bye!
$ pip uninstall mhered-test-pkg
Add a description, installation and usage instructions in the README.md
and declare it in pyproject.toml
:
readme = "README.md"
Make scriv
collect the previously created changelog fragment to a new CHANGELOG.md
file with:
$ scriv collect
Lets commit:
$ git add CHANGELOG.md README.md pyproject.toml
$ git commit -m "Prepare release 0.1.0"
Tag the commit, and push the tag to the remote (seen here):
$ git tag -a v0.1.0 -m "Initial version"
$ git push origin v0.1.0
I discovered I hadn't properly configured version numbering for scriv ( had forgotten to add version = "literal: pyproject.toml: tool.poetry.version"
under [tool.scriv]
in pyproject.toml
), so I did it now, and added a new release to test it. I bumped the version in pyproject.toml
to 0.1.1
with:
$ poetry version patch
Bumping version from 0.1.0 to 0.1.1
Next add a changelog fragment:
$ scriv create
and edit it to describe the change
### Fixed
- Configure `scriv` to get version number from `pyproject.toml`
Then I manually increased the version in mhered-test-pkg/__init__.py
__version__ = "0.1.1"
and added a test to check it is always in sync with tool.poetry.version
in pyproject.toml
(there seems to be no better way)
import toml
from pathlib import Path
import mhered_test_pkg
def test_versions_are_in_sync():
""" Checks if tool.poetry.version in pyproject.toml and
__version__ in mhered_test_pkg.__init__.py are in sync."""
path = Path(__file__).resolve().parents[2] / "pyproject.toml"
pyproject = toml.loads(open(str(path)).read())
pyproject_version = pyproject["tool"]["poetry"]["version"]
init_py_version = mhered_test_pkg.__version__
assert init_py_version == pyproject_version
Add a new changelog fragment and edit it to describe the change
$ scriv create --edit
## Added
- Test to check that versions defined in `pyproject.py` and `__init__.py` are in sync
Update the Changelog:
$ scriv collect
Commit and push, tag and push:
$ git add pyproject.toml mhered_test_pkg/__init__.py tests/test_mhered_test_pkg.py CHANGELOG.md README.md
$ git commit -m "Configure versions in scriv"
$ git push
$ git tag -a v0.1.1 -m "Configure versions in scriv"
$ git push origin v0.1.1
$ scriv github-release -v DEBUG
debug: Running command 'git tag'
debug: Command exited with 0 status. Output: 'v0.1.0\nv0.1.1\n'
debug: Running command ['git', 'config', '--get-regex', 'remote[.].*[.]url']
debug: Command exited with 0 status. Output: 'remote.origin.url https://github.com/mhered/mhered-test-pkg.git\n'
debug: Starting new HTTPS connection (1): api.github.com:443
debug: https://api.github.com:443 "GET /repos/mhered/mhered-test-pkg/releases HTTP/1.1" 200 600
warning: Version 0.1.1 has no tag. No release will be made.
warning: Version 0.1.0 has no tag. No release will be made.
It still does not work... to be continued.
There is something off with the tag v0.1.0
. Was linked to 8ad878f
which does not show here!
$ git log --oneline
5979855 (HEAD -> main, origin/main) Update README.md
fa7a843 (tag: v0.1.1) Configure versions in scriv
74b8fcb Prepare release 0.1.0
9519229 Add victory lap to README.md
c22f560 Move __init__.py to mhered_test_pkg/
066172e Minor updates to README.md
e438080 Simple Rock Paper Scissors game
2c93077 Add scriv as devt dependency
262bb9e Publish to TestPyPI
dd9cfd9 Add .gitignore to protect API tokens
81fa08d Minor change of README.md
4d8b935 Update License in README.md
e18df98 Create LICENSE.md
9400227 Update README.md
3d3a9d8 Run all pre-commits.
8de2b5a Add pre-commit devt dependency
9743aef First commit
lets repair the tag v0.1.0
:
$ git tag -d v0.1.0 # delete the old tag locally
Deleted tag 'v0.1.0' (was 8ad878f)
$ git push origin :refs/tags/v0.1.0 # delete the old tag remotely
To https://github.com/mhered/mhered-test-pkg.git
- [deleted] v0.1.0
$ git tag -a v0.1.0 74b8fcb # make a new tag locally
$ git push origin v0.1.0 # push the new local tag to the remote
Voilá:
$ git log --oneline
5979855 (HEAD -> main, origin/main) Update README.md
fa7a843 (tag: v0.1.1) Configure versions in scriv
74b8fcb (tag: v0.1.0) Prepare release 0.1.0
9519229 Add victory lap to README.md
c22f560 Move __init__.py to mhered_test_pkg/
066172e Minor updates to README.md
e438080 Simple Rock Paper Scissors game
2c93077 Add scriv as devt dependency
262bb9e Publish to TestPyPI
dd9cfd9 Add .gitignore to protect API tokens
81fa08d Minor change of README.md
4d8b935 Update License in README.md
e18df98 Create LICENSE.md
9400227 Update README.md
3d3a9d8 Run all pre-commits.
8de2b5a Add pre-commit devt dependency
9743aef First commit
I added a couple of tests, refactored a bit the code and committed the changes.
Then, to create a release there are a few steps:
- bump the version
- edit
__init__.py
manually to sync version number - run the tests
- create a fragment - with
--edit
option to launch the editor directly - collect all fragments to
CHANGELOG.md
- commit changes to
pyproject.toml README.md CHANGELOG.md mhered_test_pkg/__init__.py
- create tag
- push commit and tag
$ git commit -a -m "Add tests"
$ poetry version patch
$ atom mhered_test_pkg/__init__.py
$ pytest
============================= test session starts =============================
platform linux -- Python 3.8.10, pytest-5.4.3, py-1.11.0, pluggy-0.13.1
rootdir: /home/mhered/mhered-test-pkg
collected 4 items
tests/test_mhered_test_pkg.py .... [100%]
============================== 4 passed in 0.02s ==============================
$ scriv create --edit
$ scriv collect
$ git commit -m "Prepare release 0.1.2"
$ git push
$ git tag -a 0.1.2 -m "Add tests"
$ git push origin 0.1.2
- try to create a release with
scriv github-release
This time $ scriv github-release -v DEBUG
gives an error message so I rename tag 0.1.2
to v0.1.2
:
$ git tag v0.1.2 0.1.2^{}
$ git tag -d 0.1.2
$ git push origin :refs/tags/0.1.2
$ git log --oneline
aa3e44a (HEAD -> main, tag: v0.1.2, origin/main) Prepare release 0.1.2
b03efa8 Add tests
d688927 Passes basic tests
5979855 Update README.md
fa7a843 (tag: v0.1.1) Configure versions in scriv
74b8fcb (tag: v0.1.0) Prepare release 0.1.0
9519229 Add victory lap to README.md
...
Back to normal:
$ scriv github-release -v DEBUG
debug: Running command 'git tag'
debug: Command exited with 0 status. Output: 'v0.1.0\nv0.1.1\nv0.1.2\n'
debug: Running command ['git', 'config', '--get-regex', 'remote[.].*[.]url']
debug: Command exited with 0 status. Output: 'remote.origin.url https://github.com/mhered/mhered-test-pkg.git\n'
debug: Starting new HTTPS connection (1): api.github.com:443
debug: https://api.github.com:443 "GET /repos/mhered/mhered-test-pkg/releases HTTP/1.1" 200 717
warning: Version 0.1.2 has no tag. No release will be made.
warning: Version 0.1.1 has no tag. No release will be made.
warning: Version 0.1.0 has no tag. No release will be made.
Apparently this is because I need to add my PAT as environment variable GITHUB_TOKEN
, see scriv docs. I tried though, and it does not work... I create the release manually in github.
tox
automates testing, linting, formatting, test coverage, documentation, etc.
We add tox as a development dependency:
$ poetry add -D tox
Configuration is done in a tox.ini
file in toml
format, which can be initiated in its simplest form running $ tox-quickstart
and answering a few questions.
# tox (https://tox.readthedocs.io/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.
[tox]
envlist = py37
[testenv]
deps =
pytest
commands =
pytest
However this simplest version did not work. I had to do a few iteratons:
- Added
isolated_build = True
to work withpoetry
- Added
toml
to dependencies otherwisepytest
cannot import it. It seems thattox
creates its own virtual environments and you need to add all dependencies again... - Added formatting with
black
, linting withflake8
,pylint
,mccabe
, sorting of imports with isort, and testing withpytest
. mccabe is aflake8
plugin to check the code's McCabe complexity (should be <10). black --check
does not modify the files, only exits with an error if the check is not passed. What is the point?
Result:
# tox (https://tox.readthedocs.io/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.
[tox]
isolated_build = True
envlist = py38
[testenv]
deps =
toml
black
flake8
isort
mccabe
pylint
pytest
commands =
black --check mhered_test_pkg
isort --check mhered_test_pkg
flake8 mhered_test_pkg --max-complexity 10
pylint mhered_test_pkg
pytest .
Execute with:
$ tox
Question: why use tox
when I can use a poetry
script or even better a pre-commit
hook?
Installation and basic execution of coverage:
$ poetry add -D coverage
$ coverage run -m pytest
$ coverage report
Name Stmts Miss Cover
---------------------------------------------------
mhered_test_pkg/__init__.py 49 38 22%
tests/__init__.py 0 0 100%
tests/test_mhered_test_pkg.py 15 0 100%
---------------------------------------------------
TOTAL 64 38 41%
We can execute the checks in a more nuanced way including all files in the package and all branching paths, and we can also generate nicer HTML reports such as this one as follows:
$ coverage run --source=mhered_test_pkg --branch -m pytest .
$ coverage html
To add coverage
to tox
modify tox.ini
to add the relevant lines:
# tox (https://tox.readthedocs.io/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.
[tox]
isolated_build = True
envlist = py38
[testenv]
deps =
toml
black
flake8
isort
mccabe
pylint
pytest
coverage # development dependency
commands =
black --check mhered_test_pkg
isort --check mhered_test_pkg
flake8 mhered_test_pkg --max-complexity 10
pylint mhered_test_pkg
coverage run --source=mhered_test_pkg --branch -m pytest . # execute
coverage report -m --fail-under 90 # report & fail below 90%
Added tests to increase coverage up to 97% - i.e. all lines of code covered except the case "__name__" == "__main__:"
because tests import as module.
In the process I learned about monkeypatching user input, using iterators to simulate a sequence of inputs, or testing for sys exit, see references in ./tests/test_mhered_test_pkg.py
Usual ritual to create a release. As some of the steps modify files that need to be committed this process ends up being iterative. I try to keep it clean using profusely $ git commit --amend
along the way:
$ scriv create --edit # describe changes in a fragment
$ poetry version patch # bump version
$ atom mhered_test_pkg/__init__.py # sync _version__
$ scriv collect # update CHANGELOG.md
$ git add .
$ git commit
$ git tag -a v0.1.3 -m "97% test coverage"
$ git push
$ git log --oneline
91c4aa1 (HEAD -> main, tag: v0.1.3, origin/main) 0.1.3 automated with tox and 97% coverage
d6670c4 Automating with tox
028028a Update README.md
aa3e44a (tag: v0.1.2) Prepare release 0.1.2
b03efa8 Add tests
d688927 Passes basic tests
5979855 Update README.md
fa7a843 (tag: v0.1.1) Configure versions in scriv
74b8fcb (tag: v0.1.0) Prepare release 0.1.0
...
$ git push origin v0.1.3
GitHub Actions allow automating workflows of actions that are triggered by certain events e.g. a commit pushed to the repo, a pull request or a release. They are defined in YAML files that live in the directory .github/workflows
.
The CI runner spawns the full environment in a Github server, including setting up the OS withruns-on:
, installing and activating python with uses: actions/setup-python@v2
... with:
... python-version:
..."3.8"
, installing dependencies (via tox
or poetry
commands) and downloading our repository with uses: actions/checkout@v2
A nice intro tutorial in Youtube: https://www.youtube.com/watch?v=R8_veQiYBjI
# .github/workflows/CI.yaml
name: mhered-test-pkg CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10"]
steps:
- name: Checkout sources
uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox tox-gh-actions
- name: Run tox
run: tox
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
with:
fail_ci_if_error: true
Everything seems to be working including Codecov integration except PR comments.
I wrote a new PyPI_publish.yaml
file with the following steps:
- Checkout the repo
- Set up Python 3.8
- Install Poetry and dependencies
- Configure Poetry with a PyPI token
- Build and publish the package
A more straighforward solution using a pre-made GH action from PyPA to upload both to Test PyPI and PyPI is here: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/
Note: add the Test PyPI and PyPI credentials created earlier as repository secrets in Github as PYPI_TOKEN
and TEST_PYPI_TOKEN
: Settings --> Secrets --> Actions --> Add new repository secret.
# .github/workflows/PyPI_publish.yaml
name: publish mhered-test-pkg to PyPI
on:
release:
types: [published]
branches: [ main ]
workflow_dispatch:
jobs:
build-and-publish:
runs-on: ubuntu-latest
steps:
- name: Checkout sources
uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: "3.8"
- name: Install poetry and dependencies
run: |
python -m pip install --upgrade pip
python -m pip install poetry
- name: Configure poetry
env:
pypi_token: ${{ secrets.PYPI_TOKEN }}
run: poetry config pypi-token.pypi $pypi_token
- name: Build and publish
run: poetry publish --build
New release:
$ poetry version patch
$ atom mhered_test_pkg/__init__.py
$ pytest
$ scriv create --edit
$ scriv collect
$ git add .
$ git commit -m "Add CI/CD and Codecov - release 0.1.4"
$ git push
$ git tag -a v0.1.4 -m "Add GH actions for CI/CD and Codecov integration"
$ git push origin v0.1.4
Did not work initially. I found three typos in PyPI_publish.yaml
, should be:
secrets.PYPI_TOKEN
instead ofsecrets.PyPI_TOKEN
run: poetry config pypi-token.pypi $pypi_token
instead ofpypi_token.pypi
run: poetry publish --build
instead ofuses:
I also added minor updates to README.md
and I created a few releases in the process.
$ poetry version patch
$ atom mhered_test_pkg/__init__.py
$ pytest
$ scriv create --edit
$ scriv collect
$ git add .
$ git commit -m "Commit Message - release 0.1.X"
$ git push
$ git tag -a v0.1.X -m "Release Message"
$ git push origin v0.1.X
Finally succeeded with v0.1.7
Piece of cake following the instructions in https://shields.io/
Ad adapt everywhere:
- in tests change the import statement:
from src.mhered_test_pkg import ...
- in
tox.ini
replacemhered_test_pkg
bysrc
...
commands =
black --check src
isort --check src
flake8 src --max-complexity 10
pylint src
coverage run --source=src --branch -m pytest .
coverage report -m --fail-under 60
coverage xml
...
In development, as an alternative to calling:
$ python3 ./src/mhered_test_pkg/__init__.py
- We can create a
__main__.py
file:
""" __main__ entry point """
from mhered_test_pkg import rock_paper_scissors
if __name__ == "__main__":
rock_paper_scissors()
to allow calling the package as a module:
$ python3 -m src.mhered_test_pkg
And/or
- edit
pyproject.py
to add the following line:
[tool.poetry.scripts]
rps = "mhered_test_pkg.__init__:rock_paper_scissors
to create a shortcut to execute the game writing:
$ rps
Note that when the app is distributed the methods that work are different! - cfr. the ones described at the beginning of the README.md
Time to make a new release 0.1.8...
I noticed some issues due to discrepancies between the linting in pre-commit and the one in tox. For instance isort and black seem to be doing incompatible changes in one of the improts in tests.
To add all the linting to pre-commit, we edit .pre-commit-config.yaml
:
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.0.1
hooks:
- id: check-toml
- id: check-yaml
- id: end-of-file-fixer
- id: mixed-line-ending
- repo: https://github.com/psf/black
rev: 22.3.0
hooks:
- id: black
- repo: https://github.com/PyCQA/isort
rev: 5.10.1
hooks:
- id: isort
args: ["--profile", "black"]
- repo: https://github.com/PyCQA/flake8
rev: 4.0.1
hooks:
- id: flake8
additional_dependencies: [mccabe]
args: ["--max-line-length", "88", "--max-complexity", "10"]
- repo: https://github.com/PyCQA/pylint/
rev: v2.14.5
hooks:
- id: pylint
exclude: tests/ # Prevent files in tests/ to be passed in to pylint.
Modify tox.ini
to create dedicated environments for linting and coverage and execute them only with python 3.8 to avoid running them multiple times. The linting one simple calls pre-commit:
# tox (https://tox.readthedocs.io/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.
[tox]
isolated_build = True
envlist =
py38,
py39,
py310,
linting,
coverage,
[testenv]
deps =
toml
pytest
changedir = {envtmpdir}
commands =
pytest {toxinidir}
[testenv:linting]
deps = pre-commit
commands = pre-commit run --all-files --show-diff-on-failure
[testenv:coverage]
deps =
toml
pytest
coverage
commands =
coverage run --source=src --branch -m pytest {toxinidir}
coverage report -m --fail-under 90
coverage xml -o {toxinidir}/coverage.xml
[gh-actions]
python =
3.8: py38, linting, coverage
3.9: py39
3.10: py310
Finally modify the GH action CI.yaml
to upload to codecov only when running Python 3.8. In this case we use a premade GH action instead of running manually the commands:
...
- name: Upload coverage to Codecov
# Only generate the coverage report in Python 3.8
if: "matrix.python-version == '3.8'"
uses: codecov/codecov-action@v2
with:
fail_ci_if_error: true
And with this we are ready to publish the final release 0.1.10.