Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite the release workflow and refactor existing GHA workflows #40

Merged
merged 3 commits into from
Jul 29, 2023

Conversation

diazona
Copy link
Owner

@diazona diazona commented Jul 24, 2023

In this PR I've removed the old release workflow (including the release environment in tox.ini) and added a completely new Github Actions workflow for handling a release. As part of that, I refactored the existing workflow into multiple files, including reusable workflows test.yml and docs.yml which get called in multiple ways from the primary workflows.

Release step sequencing

The process I implemented here works as follows:

  1. We start by making a Github release ("Github release" refers to the announcement page on Github that we make using this form), with a corresponding tag, in the Github web interface (or by using the Github API)
  2. The workflow automatically runs and tests the code from a checkout of the main Git branch
  3. It also tests the documentation from a checkout of the main Git branch
  4. It builds the package for distribution
  5. After getting a maintainer's approval through the Github UI (or API), it publishes the package to Test PyPI
  6. Then after getting approval again, it publishes to PyPI

I'm not sure if that's the best way to sequence the steps involved in making a release, though. (Note I'm using "release" by itself to mean a tagged and published version, in contrast with "Github release") For one thing, the tests are not run against the actual distributable package, so there's a slight chance that a checkout of the repo would pass tests while the same tests run against the installable wheel would fail. (This is very unlikely, but it could happen if the package uses a hard-coded relative file or something like that.) This sequencing also carries the risk that the package might fail tests after creating the tag, and then we're faced with having a tagged release that doesn't work and can't be posted to PyPI, which kind of forces us to skip a version number.

There are some alternative sequences we could consider, such as this:

  1. We start by making a Github release, with a corresponding tag, in the Github web interface (or by using the Github API)
  2. The workflow automatically runs and builds the package for distribution
  3. It tests the code from the package (not from the checkout)
  4. It tests the documentation from a checkout of the main branch (because the documentation source isn't included in the wheel)
  5. With approval, it publishes the package to Test PyPI
  6. Again with approval, it publishes to PyPI

This would take care of the risk that tests pass against the checked-out source code but fail against the installed package. But it turns out to be somewhat tricky to set up tox to run tests against the installed package (although doable with some hacky workarounds in our case, at least for now).

Or we could do this:

  1. Manually trigger a workflow
  2. The workflow tests the code from a checkout of the main Git branch
  3. It also tests the documentation from the main Git branch
  4. It creates the tag to be used for the release
  5. It builds the package for distribution
  6. With approval, it publishes the package to Test PyPI
  7. With approval, it publishes to PyPI
  8. The workflow prompts us to create the Github release, or perhaps creates a draft Github release which we can then fill in with release notes and publish

That would take care of the risk of the tests failing when we've already created a tag, but it still has the problem where what we're testing is not exactly the same as what gets installed. And it's also awkward to have the semi-manual step of creating the Github release at the end, which basically needs Github Actions to prompt us to create a release but wait for us to actually fill in the release description. It would be nice to have the release notes prepared in advance.

I think the ideal sequence would be this:

  1. We manually draft the Github release including the release notes and the name of the tag to be created, but don't make it public yet
  2. The workflow runs and checks out the code in a local view
  3. It creates the tag in that local view but doesn't push it up to Github
  4. It builds the package for distribution
  5. It tests the code from the package
  6. It tests the documentation in the local view
  7. Now that tests have passed, it pushes the tag from the local view to the Github repo
  8. With approval, it publishes to Test PyPI
  9. Then it publishes to PyPI
  10. Finally it marks the previously drafted Github release as public

But it's fairly complicated to make all those steps happen in exactly that order, as well as to arrange for tests to be run against the built package. I decided to leave that as a future goal, and for now just commit something that we can use to make a release so that we're not blocked in making this package available to the public.

@diazona diazona added the enhancement New feature or request label Jul 24, 2023
@diazona diazona added this to the Initial release milestone Jul 24, 2023
@diazona diazona self-assigned this Jul 24, 2023
@diazona
Copy link
Owner Author

diazona commented Jul 24, 2023

@sjlongland Feel free to take a look at this if you want. It's theoretically done (or, done enough for now), but given how complex these changes are, I'm going to want to wait a day or two and look over it with fresh eyes before I'm ready for it to be merged.

diazona added 3 commits July 25, 2023 12:05
I want to force all releases to go through Github for traceability, so
I'm removing the release environment from tox.ini to make it difficult
for someone to create a release locally. And since the existing release
job in Github Actions uses the tox environment, I'm removing that job as
well. In an upcoming commit I'm going to add a brand new GHA workflow to
produce a release.
In this commit I'm breaking out the parts of the main GHA workflow that
build the documentation and run tests. That will allow us to use
the reusable docs and test workflows as part of the release process,
which I'm going to add in the next commit.
This commit adds a completely new Github Actions workflow for handling
a release. Or technically, we have to make a release in the Github web
interface (or by using the Github API); the workflow just handles
everything that needs to happen after that, namely building and testing
the package and uploading it to (test and regular) PyPI. The flow is
thus:

- create a tag and Github release (manual)
- test the code from the main Git branch
- test the documentation from the main Git branch
- build the package for distribution
- publish to Test PyPI
- publish to PyPI

I'm not sure if that's the best way to sequence the steps involved in
making a release, though. For one thing, the tests are not run against
the actual distributable package, so there's a slight chance that
a checkout of the repo would pass tests while the same tests run against
the installable wheel would fail. (This is very unlikely, but it could
happen if the package uses a hard-coded relative file or something like
that.) This sequencing also carries the risk that the package might fail
tests *after* creating the tag, and then we're faced with having
a tagged release that doesn't work and can't be posted to PyPI.

There are some alternative sequences we could consider, such as:

- create a tag and Github release (manual)
- build the package for distribution
- test the code from the package
- test the documentation from the main Git branch
- publish to Test PyPI
- publish to PyPI

This would take care of the risk that tests pass against the checked-out
source code but fail against the installed package. But it turns out to
be somewhat tricky to set up tox to run tests against the installed
package (although doable with some hacky workarounds in our case, at
least for now).

Or we could do this:

- test the code from the main Git branch
- test the documentation from the main Git branch
- create the tag
- build the package for distribution
- publish to Test PyPI
- publish to PyPI
- create the release (semi-manual)

That would take care of the risk of the tests failing when we've already
created a tag, but it still has the problem where what we're testing is
not exactly the same as what gets installed. And it's also awkward to
have the semi-manual step of creating the Github release at the end,
which basically needs Github Actions to prompt us to create a release
but wait for us to actually fill in the release description. (Of course
we could probably get away with having the release be created
automatically and then just manually editing the release notes
afterwards, but it would be nice to have the release notes prepared in
advance.)

I think the ideal sequence would be this:

- draft the release including the release notes and the name of the tag
  to be created, but don't make it public yet (manual)
- check out the code in a local view
- create the tag in that local view
- build the package for distribution
- test the code from the package
- test the documentation in the local view
- push the tag from the local view to the Github repo
- publish to Test PyPI
- publish to PyPI
- publish the previously drafted release

But it's fairly complicated to make all those steps happen in exactly
that order, as well as to arrange for tests to be run against the built
package. I decided to leave that as a future goal, and for now just
commit something that we can use to make a release so that we're not
blocked in making this package available to the public.
@diazona diazona force-pushed the release-workflow/1/dev branch from 49e1277 to 3961149 Compare July 25, 2023 19:22
@diazona diazona marked this pull request as ready for review July 25, 2023 21:32
@diazona diazona requested a review from sjlongland July 25, 2023 21:32
@diazona diazona assigned sjlongland and unassigned diazona Jul 25, 2023
@diazona
Copy link
Owner Author

diazona commented Jul 25, 2023

I think it's ready now 👍 (unless I missed something)

We won't be able to test the release workflow without actually creating a release, but it could be a pre-release that never gets pushed to PyPI.

@sjlongland sjlongland merged commit a34aa11 into main Jul 29, 2023
@sjlongland sjlongland deleted the release-workflow/1/dev branch July 29, 2023 01:22
@diazona diazona linked an issue Aug 2, 2023 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rewrite release logic
2 participants