Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: rewrite the release pipeline to make it clean #2038

Merged
merged 2 commits into from
Jul 31, 2023

Conversation

zyy17
Copy link
Collaborator

@zyy17 zyy17 commented Jul 25, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

It's almost the rewrite of release.yml.

Why need to rewrite release.yml

The original release.yml has the following disadvantages:

  • All the logic is coupled in one big yaml file, and hard to add the new artifact release;
  • Lots of redundant logic, and always copy and paste the same logic;
  • Lack of reproducible builds: the original release.yml uses cross-compiling to build the artifacts instead of Dockerfile;
  • Don't support AWS EC2 instances to use larger instances;

The new release.yml

actions/

The new release.yml use composite action to decouple the logic in .github/actions/

  • Start/Stop EC2 runners

    • .github/actions/start-runner
    • .github/actions/stop-runner
  • Binary building

    • .github/actions/build-greptime-binary: Use make build-greptime-by-buildx command to build greptime and upload it;
    • .github/actions/build-linux-artifacts: Use .github/actions/build-greptime-binary to build Linux greptime by different base images and features.
    • .github/actions/build-macos-artifacts: Still use cross-compiling to build macOS artifacts(most of logic is ported from the original implementation);
  • Image building

    • .github/actions/build-images/: Use .github/actions/build-greptime-image to build and push docker images that are based on the different base images;
    • .github/actions/build-greptime-image: Build multiple platform docker image by buildx;
    • .github/actions/build-dev-builder-image: Build and push dev-builder image;
  • Releasing and distribution

    • .github/actions/release-artifacts: Create GitHub release notes and release artifacts;
    • .github/actions/upload-artifacts: Upload greptime binary and checksum to GitHub Artifacts and S3;

release.yml

The new release.yml split multiple small and clear jobs to handle releasing:

image

  • At first, we need to use the allocate-runners job to create a version(.github/scripts/create-version.sh) tag and allocate runners;

  • After all the resources are ready, the pipeline begins to build multiple binary artifacts;

  • After all the binary artifacts are ready, the pipeline begins to build and push docker images;

  • At last, the release pipeline will create the GitHub Relase;

The new manual inputs panel

The new manual inputs panel maybe looks complicated but integrates multiple building choices:

image

The Actions variables and secrets

All the variables and secrets are maintained by Greptime.

Variables

Name Usage
DEFAULT_SKIP_TEST The boolean variable that decides whether to run integration tests or not.
DEFAULT_AMD64_RUNNER The x86_64 runner that builds amd64 artifacts. It can be ubuntu-latest, ec2-c6i.4xlarge-amd64 etc.
DEFAULT_ARM64_RUNNER The aarch64 runner that builds amd64 artifacts. It can be ec2-c6g.xlarge-arm64, ec2-c6g.2xlarge-arm64 etc. For now, GitHub Actions don't support the aarch64 runner.
DEFAULT_MACOS_RUNNER The macOS runner that builds macOS artifacts.
EC2_RUNNER_LINUX_AMD64_IMAGE_ID The image id of the amd64 runner.
EC2_RUNNER_LINUX_ARM64_IMAGE_ID The image id of the arm64 runner.
EC2_RUNNER_REGION The aws region of the allocated ec2 instances.
EC2_RUNNER_SUBNET_ID The subnet id of ec2 runner.
EC2_RUNNER_SECURITY_GROUP_ID The security group id of ec2 runner.
AWS_RELEASE_BUCKET The release S3 bucket of greptime binary.
AWS_RELEASE_BUCKET_REGION The aws region of the release S3 bucket.
IMAGE_NAMESPACE The image namespace, can be greptimedb.
ACR_IMAGE_REGISTRY The ACR image registry, can be greptime-registry.cn-hangzhou.cr.aliyuncs.com.

Secrets

Name Usage
AWS_ACCESS_KEY_ID The AWS account is used for allocating ec2 instances.
AWS_SECRET_ACCESS_KEY The AWS account is used for allocating ec2 instances.
AWS_CN_ACCESS_KEY_ID The AWS-CN account is used for operating S3.
AWS_CN_SECRET_ACCESS_KEY The AWS-CN account that is used for operating S3.
DOCKERHUB_USERNAME The DockerHub username.
DOCKERHUB_TOKEN The DockerHub token.
ALICLOUD_USERNAME The ACR username.
ALICLOUD_PASSWORD The ACR password.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

@zyy17 zyy17 changed the title feat: rewrite release pipeline to make it clean feat: rewrite the release pipeline to make it clean Jul 25, 2023
@zyy17 zyy17 force-pushed the feat/new-release-pipeline branch from bb46bd5 to 80d335a Compare July 25, 2023 15:35
@zyy17 zyy17 force-pushed the feat/new-release-pipeline branch from 80d335a to 40c2162 Compare July 25, 2023 15:47
@zyy17 zyy17 marked this pull request as ready for review July 25, 2023 16:00
@codecov
Copy link

codecov bot commented Jul 25, 2023

Codecov Report

Merging #2038 (40c2162) into develop (662879f) will decrease coverage by 0.32%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #2038      +/-   ##
===========================================
- Coverage    85.23%   84.91%   -0.32%     
===========================================
  Files          664      664              
  Lines       104637   104637              
===========================================
- Hits         89183    88852     -331     
- Misses       15454    15785     +331     

Copy link
Contributor

@evenyag evenyag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L 😵 TM

@MichaelScofield MichaelScofield added this pull request to the merge queue Jul 31, 2023
Merged via the queue into GreptimeTeam:develop with commit 4583247 Jul 31, 2023
@zyy17 zyy17 deleted the feat/new-release-pipeline branch August 1, 2023 12:37
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* refactor: modify cache path of Dockerfile

* feat: rewrite the release pipeline to make it clean
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants