Migrate core CI pipeline to GitHub Actions #1865

negz · 2020-10-21T06:08:54Z

Description of your changes

This PR migrates Crossplane's CI/CD system from Jenkins to GitHub Actions. Its logic is largely implemented by the https://github.com/upbound/build/ make library, and thus unchanged from the Jenkins pipeline. The big differences in this implementation are:

We use the Docker Buildx and Setup Go actions - not the Upbound cross container - to ensure repeatable builds. This allows us to avoid the ~5 minute cross container build time penalty.
We run the key parts of the build (lint, unit test, integration test, build artifacts) in parallel on distinct workers.

The net result of these changes is that we can build and verify a pull request in ~6.5 minutes, where Jenkins has been averaging ~20 minutes lately.

In addition to being faster, migrating to GitHub Actions means:

No particular entity (i.e. Upbound) needs to own and operate the CI/CD infrastructure.
We don't need to pay for Jenkins and its runners anymore. Actions is free for OSS.
Setting up CI for new repos is easier. Any repo that includes .github/workflows/ci.yml will have CI/CD enabled.
GitHub Actions is arguably friendlier, more idiomatic, and more tightly integrated for GitHub users than Jenkins.
We always use a cached Go binary, and often used cached Go dependencies. Failures downloading these has been a common cause of failed builds on Jenkins.

Take a look at https://github.com/negz/crossplane/actions/runs/319197909 to see this pipeline in action.

I have:

Read and followed Crossplane's contribution process.
Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

negz · 2020-10-21T06:16:48Z

.github/workflows/ci.yml

+          BRANCH_NAME: master
+          CHANNEL: master
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}


Note that I haven't tested the publish and promote steps, except to verify that they don't run on my fork. The idea here is that:

Publish will run on any build that has valid AWS secrets set.

Promote will run only on the master branch as long as valid AWS secrets are set.

Note that unlike Jenkins, "any build" here can include forks. In practice forks should run this workflow, but skip the publish and promote stages because they don't have the AWS credential secrets set. If a fork did somehow acquire and set these credentials (in its GitHub Settings) it would publish and promote artifacts, but this seems unlikely.

@negz I believe that we currently do publish artifacts for to https://releases.crossplane.io/build/ for any successful build on a PR. I don't think we necessarily have to do that though, especially since it looks like we now upload to GH directly (which is super cool!)

Right - this implementation will also publish (but not promote) artifacts to Docker Hub and S3 on every build, as long as the appropriate secrets are set to do so.

.github/workflows/ci.yml

negz · 2020-10-21T07:08:21Z

.github/workflows/ci.yml

+on:
+  push:
+    branches:
+      - master


Should we include the release branches here @hasheddan and @jbw976?

I believe so 👍

negz · 2020-10-21T07:48:55Z

.github/workflows/ci.yml

+        if: env.DOCKER_USR != ''
+        with:
+          username: ${{ secrets.DOCKER_USR }}
+          password: ${{ secrets.DOCKER_PSW }}


I suggest we set these and the AWS secrets at the org level, to make setting up new repos easy - https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-an-organization

That sounds good to me, just want to make sure that we don't allow someone to submit a PR that modifies the action and they can do whatever they want using the secret 👍

hasheddan

This is awesome to see @negz! Thanks so much for investing time in this :)

It looks like we are building multiple times here in order to be able to run our different steps concurrently. Had you considered doing a serial build, then splitting out to concurrent running of unit tests / e2e tests / etc.? Is that even possible with GH actions?

hasheddan · 2020-10-21T11:55:28Z

.github/workflows/ci.yml

+        if: env.DOCKER_USR != ''
+        with:
+          username: ${{ secrets.DOCKER_USR }}
+          password: ${{ secrets.DOCKER_PSW }}


That sounds good to me, just want to make sure that we don't allow someone to submit a PR that modifies the action and they can do whatever they want using the secret 👍

.github/workflows/ci.yml

Jenkinsfile

.github/workflows/ci.yml

hasheddan · 2020-10-21T12:02:24Z

.github/workflows/ci.yml

+on:
+  push:
+    branches:
+      - master


I believe so 👍

.github/workflows/ci.yml

khos2ow

This looks pretty amazing, if I may suggest some further changes.

.github/workflows/ci.yml

negz · 2020-10-21T20:12:55Z

Had you considered doing a serial build, then splitting out to concurrent running of unit tests / e2e tests / etc.? Is that even possible with GH actions?

It may be possible, but I suspect it would get tricky. We'd need to model the fan out as separate jobs, which would run on distinct VMs each with a pristine build environment. So we'd need to use the cache or artifacts to pass our Go build cache from the initial build onto the jobs that run the unit tests, e2e tests, etc. I don't think the complexity is worth it, given that it I don't believe it would actually speed up builds.

This pipeline has been replaced by GitHub Actions. The promote and tag pipelines will be replaced in a future commit. Signed-off-by: Nic Cope <negz@rk0n.org>

This commit is a moderately faithful port of the existing Jenkinsfile to GitHub Actions. Notably, it eschews the cross container to avoid the ~5 min penalty of having to build it and runs much of the build in parallel. Signed-off-by: Nic Cope <negz@rk0n.org>

hasheddan · 2020-10-21T21:01:30Z

I don't think the complexity is worth it, given that it I don't believe it would actually speed up builds.

True, we would still be constrained by the longest running action. That being said, it would be nice to not needlessly build three times. Definitely can be a future optimization though 👍

hasheddan

LGTM

negz requested review from jbw976 and hasheddan October 21, 2020 06:08

negz commented Oct 21, 2020

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

negz force-pushed the action branch from b7ce9a5 to efea3b8 Compare October 21, 2020 06:30

negz commented Oct 21, 2020

View reviewed changes

negz force-pushed the action branch from aa3f595 to 84dcefd Compare October 21, 2020 07:29

negz commented Oct 21, 2020

View reviewed changes

hasheddan reviewed Oct 21, 2020

View reviewed changes

khos2ow reviewed Oct 21, 2020

View reviewed changes

negz force-pushed the action branch from 30a175a to 2c242c1 Compare October 21, 2020 20:30

negz added 2 commits October 21, 2020 13:56

Remove the primary Jenkinsfile

50151ae

This pipeline has been replaced by GitHub Actions. The promote and tag pipelines will be replaced in a future commit. Signed-off-by: Nic Cope <negz@rk0n.org>

negz force-pushed the action branch from 2c242c1 to be07dd7 Compare October 21, 2020 20:57

hasheddan approved these changes Oct 21, 2020

View reviewed changes

negz merged commit d6d40ef into crossplane:master Oct 21, 2020

negz deleted the action branch October 21, 2020 21:42

This was referenced Oct 22, 2020

Migrate tag and promote Jenkins jobs to GitHub Actions #1873

Closed

Shut down jenkinsci.upbound.io #1881

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate core CI pipeline to GitHub Actions #1865

Migrate core CI pipeline to GitHub Actions #1865

negz commented Oct 21, 2020 •

edited

Loading

negz Oct 21, 2020

hasheddan Oct 21, 2020

negz Oct 21, 2020

negz Oct 21, 2020

hasheddan Oct 21, 2020

negz Oct 21, 2020

hasheddan Oct 21, 2020

hasheddan left a comment

hasheddan Oct 21, 2020

hasheddan Oct 21, 2020

khos2ow left a comment

negz commented Oct 21, 2020

hasheddan commented Oct 21, 2020

hasheddan left a comment

Migrate core CI pipeline to GitHub Actions #1865

Migrate core CI pipeline to GitHub Actions #1865

Conversation

negz commented Oct 21, 2020 • edited Loading

Description of your changes

How has this code been tested

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hasheddan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khos2ow left a comment

Choose a reason for hiding this comment

negz commented Oct 21, 2020

hasheddan commented Oct 21, 2020

hasheddan left a comment

Choose a reason for hiding this comment

negz commented Oct 21, 2020 •

edited

Loading