Skip to content

Commit

Permalink
merged
Browse files Browse the repository at this point in the history
  • Loading branch information
@milldr
milldr committed Sep 13, 2024
2 parents e101456 + c16dc6c commit 7e9fef2
Showing 26 changed files with 1,097 additions and 725 deletions.
5 changes: 0 additions & 5 deletions .github/actions/build-website/action.yml
View file
Original file line number Diff line number Diff line change
@@ -30,11 +30,6 @@ runs:
role-to-assume: ${{ inputs.iam_role_arn }}
role-session-name: ${{ inputs.iam_role_session_name }}

- name: Checkout Repository
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Setup Node
uses: actions/setup-node@v4
with:
2 changes: 2 additions & 0 deletions .github/workflows/website-deploy-preview.yml
View file
Original file line number Diff line number Diff line change
@@ -42,6 +42,8 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0
# This workflows runs on pull_request_target, so we need to checkout the PR branch
ref: ${{ github.event.pull_request.head.ref }}

- name: Build Website
uses: ./.github/actions/build-website
16 changes: 12 additions & 4 deletions docs/best-practices/developer/semver.mdx
View file
Original file line number Diff line number Diff line change
@@ -10,15 +10,23 @@ tags:

We practice [Semantic Versioning](https://semver.org/) for all projects (e.g. GitHub Tags/Releases, Helm Charts, Terraform Modules, Docker Images). Using this versioning standard helps to reduce the entropy related to [Dependency Hell](https://en.wikipedia.org/wiki/Dependency_hell).

![Image credit: [Gopher Academy](https://blog.gopheracademy.com/advent-2015/semver/)](/assets/aa35c54-semver.png)Image credit: [Gopher Academy](https://blog.gopheracademy.com/advent-2015/semver/)
<figure>
<img src="/assets/aa35c54-semver.png" style={{ background: '#fff' }} />
<figcaption>Image credit: [Gopher Academy](https://blog.gopheracademy.com/advent-2015/semver/)</figcaption>
</figure>

## Semantics

Generally, all of our versions follow this convention: `X.Y.Z` (e.g. `1.2.3`). Sometimes, we'll use this format: `X.Y.Z-branch` when we need to disambiguate between versions existing in multiple branches.

- **Major Releases** - These are releases when `X` changes. These releases will typically have major changes in the interface. These releases may not be backward-compatible.
- **Minor Releases** - These are releases when `Y` changes. These releases bundle new features, but the interface should be largely the same. Minor releases should be backward-compatible.
- **Patch Releases** - These are releases when `Z` changes. These releases are typically bug fixes which do not introduce new features. These releases are backward-compatible.
<dl>
<dt>Major Releases</dt>
<dd>These are releases when `X` changes. These releases will typically have major changes in the interface. These releases may not be backward-compatible.</dd>
<dt>Minor Releases</dt>
<dd>These are releases when `Y` changes. These releases bundle new features, but the interface should be largely the same. Minor releases should be backward-compatible.</dd>
<dt>Patch Releases</dt>
<dd>These are releases when `Z` changes. These releases are typically bug fixes which do not introduce new features. These releases are backward-compatible.</dd>
</dl>

We use GitHub tags & releases for all versioning. All docker images follow the same convention.

6 changes: 3 additions & 3 deletions docs/layers/ecs/tutorials/1password-scim-bridge.mdx
View file
Original file line number Diff line number Diff line change
@@ -118,11 +118,11 @@ The implementation of this is fairly simple. We will generate credentials for th
</Step>

<Step>
### <StepNumber/> Validate the Integration
### <StepNumber/> Validate the Integration

After the task is deployed, validate the integration. Connect to the VPN (if deployed the ECS service is deployed with a private ALB), navigate to the SCIM bridge URL, and confirm the service is running.
The final step is to validate the integration. Connect to the VPN (if deployed the ECS service is deployed with a private ALB), navigate to the SCIM bridge URL, and confirm the service is running.

For example, go to `https://1pass-scim.platform.usw1.dev.plat.acme-svc.com/`
For example, go to `https://1pass-scim.platform.usw1.dev.plat.acme-svc.com/`
</Step>

<Step>
211 changes: 107 additions & 104 deletions docs/layers/github-actions/eks-github-actions-controller.mdx
View file
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@ sidebar_label: "Actions Runner Controller (EKS)"
import Intro from '@site/src/components/Intro';
import KeyPoints from '@site/src/components/KeyPoints';
import Note from '@site/src/components/Note';
import Steps from '@site/src/components/Steps';
import Step from '@site/src/components/Step';
import StepNumber from '@site/src/components/StepNumber';
import AtmosWorkflow from '@site/src/components/AtmosWorkflow';
@@ -41,149 +42,151 @@ In order to deploy Self-Hosted GitHub Runners on EKS, follow the steps outlined
- Then we need to decide on the SSM path for the GitHub secret (Application private key) and GitHub webhook secret.
</KeyPoints>

<Step>
### <StepNumber/> GitHub Application Private Key
<Steps>
<Step>
### <StepNumber/> GitHub Application Private Key

Since the secret is automatically scoped by AWS to the account and region where the secret is stored, we recommend the
secret be stored at `/github/acme/github_token`.
Since the secret is automatically scoped by AWS to the account and region where the secret is stored, we recommend the
secret be stored at `/github/acme/github_token`.

`stacks/catalog/eks/actions-runner-controller.yaml`:
`stacks/catalog/eks/actions-runner-controller.yaml`:

```yaml
ssm_github_secret_path: "/github_runners/controller_github_app_secret"
```
```yaml
ssm_github_secret_path: "/github_runners/controller_github_app_secret"
```
The preferred way to authenticate is by _creating_ and _installing_ a GitHub App. This is the recommended approach as it
allows for more much more restricted access than using a personal access token, at least until
[fine-grained personal access token permissions](https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/)
are generally available. Follow the instructions
[here](https://github.com/actions/actions-runner-controller/blob/master/docs/authenticating-to-the-github-api.md) to
create and install the GitHub App.
The preferred way to authenticate is by _creating_ and _installing_ a GitHub App. This is the recommended approach as it
allows for more much more restricted access than using a personal access token, at least until
[fine-grained personal access token permissions](https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/)
are generally available. Follow the instructions
[here](https://github.com/actions/actions-runner-controller/blob/master/docs/authenticating-to-the-github-api.md) to
create and install the GitHub App.
At the creation stage, you will be asked to generate a private key. This is the private key that will be used to
authenticate the Action Runner Controller. Download the file and store the contents in SSM using the following command,
adjusting the profile and file name. The profile should be the `admin` role in the account to which you are deploying
the runner controller. The file name should be the name of the private key file you downloaded.
At the creation stage, you will be asked to generate a private key. This is the private key that will be used to
authenticate the Action Runner Controller. Download the file and store the contents in SSM using the following command,
adjusting the profile and file name. The profile should be the `admin` role in the account to which you are deploying
the runner controller. The file name should be the name of the private key file you downloaded.

```
AWS_PROFILE=acme-core-use1-auto-admin chamber write github_runners controller_github_app_secret -- "$(cat APP_NAME.DATE.private-key.pem)"
```
```
AWS_PROFILE=acme-core-use1-auto-admin chamber write github_runners controller_github_app_secret -- "$(cat APP_NAME.DATE.private-key.pem)"
```

You can verify the file was correctly written to SSM by matching the private key fingerprint reported by GitHub with:
You can verify the file was correctly written to SSM by matching the private key fingerprint reported by GitHub with:

```
AWS_PROFILE=acme-core-use1-auto-admin chamber read -q github_runners controller_github_app_secret | openssl rsa -in - -pubout -outform DER | openssl sha256 -binary | openssl base64
```
```
AWS_PROFILE=acme-core-use1-auto-admin chamber read -q github_runners controller_github_app_secret | openssl rsa -in - -pubout -outform DER | openssl sha256 -binary | openssl base64
```

At this stage, record the Application ID and the private key fingerprint in your secrets manager (e.g. 1Password). You
will need the Application ID to configure the runner controller, and want the fingerprint to verify the private key.
At this stage, record the Application ID and the private key fingerprint in your secrets manager (e.g. 1Password). You
will need the Application ID to configure the runner controller, and want the fingerprint to verify the private key.

Proceed to install the GitHub App in the organization or repository you want to use the runner controller for, and
record the Installation ID (the final numeric part of the URL, as explained in the instructions linked above) in your
secrets manager. You will need the Installation ID to configure the runner controller.
Proceed to install the GitHub App in the organization or repository you want to use the runner controller for, and
record the Installation ID (the final numeric part of the URL, as explained in the instructions linked above) in your
secrets manager. You will need the Installation ID to configure the runner controller.

In your stack configuration, set the following variables, making sure to quote the values so they are treated as
strings, not numbers.
In your stack configuration, set the following variables, making sure to quote the values so they are treated as
strings, not numbers.

```
github_app_id: "12345"
github_app_installation_id: "12345"
```
</Step>
```
github_app_id: "12345"
github_app_installation_id: "12345"
```
</Step>

<Step>
### <StepNumber/> GitHub Webhook Secret Token
<Step>
### <StepNumber/> GitHub Webhook Secret Token

If using the Webhook Driven autoscaling (recommended), generate a random string to use as the Secret when creating the
webhook in GitHub.
If using the Webhook Driven autoscaling (recommended), generate a random string to use as the Secret when creating the
webhook in GitHub.

Generate the string using 1Password (no special characters, length 45) or by running
Generate the string using 1Password (no special characters, length 45) or by running

```bash
dd if=/dev/random bs=1 count=33 2>/dev/null | base64
```
```bash
dd if=/dev/random bs=1 count=33 2>/dev/null | base64
```

Store this key in AWS SSM under the same path specified by `ssm_github_webhook_secret_token_path`
Store this key in AWS SSM under the same path specified by `ssm_github_webhook_secret_token_path`

`stacks/catalog/eks/actions-runner-controller.yaml`:
`stacks/catalog/eks/actions-runner-controller.yaml`:

```yaml
ssm_github_webhook_secret_token_path: "/github_runners/github_webhook_secret_token"
```
</Step>
```yaml
ssm_github_webhook_secret_token_path: "/github_runners/github_webhook_secret_token"
```
</Step>
## Deploy
## Deploy
Automation has an unique set of components from the `plat` clusters and therefore has its own Atmos Workflow. Notably,
`auto` includes the `eks/actions-runner-controller` component, which is used to create the `self-hosted` runners for the
GitHub Repository or Organization
Automation has an unique set of components from the `plat` clusters and therefore has its own Atmos Workflow. Notably,
`auto` includes the `eks/actions-runner-controller` component, which is used to create the `self-hosted` runners for the
GitHub Repository or Organization

<Note title="Tip">
The first three steps before are all included in the following workflow:
<Note title="Tip">
The first three steps before are all included in the following workflow:

<AtmosWorkflow workflow="deploy/github-runners" fileName="github" />
</Note>
<AtmosWorkflow workflow="deploy/github-runners" fileName="github" />
</Note>

<Step>
### <StepNumber/> `iam-service-linked-roles` Component
<Step>
### <StepNumber/> `iam-service-linked-roles` Component

At this point we assume that the `iam-service-linked-roles` component is already deployed for `core-auto`. If not,
deploy this component now with the following command:
At this point we assume that the `iam-service-linked-roles` component is already deployed for `core-auto`. If not,
deploy this component now with the following command:

```bash
atmos terraform apply iam-service-linked-roles -s core-gbl-auto
```
</Step>
```bash
atmos terraform apply iam-service-linked-roles -s core-gbl-auto
```
</Step>

<Step>
### <StepNumber/> Deploy Automation Cluster and Resources
<Step>
### <StepNumber/> Deploy Automation Cluster and Resources

Deploy the cluster with the same commands as `plat` cluster deployments:
Deploy the cluster with the same commands as `plat` cluster deployments:

<AtmosWorkflow workflow="deploy/cluster" fileName="eks" stack="core-use1-auto"/>
<AtmosWorkflow workflow="deploy/resources" fileName="eks" stack="core-use1-auto"/>
<AtmosWorkflow workflow="deploy/cluster" fileName="eks" stack="core-use1-auto"/>
<AtmosWorkflow workflow="deploy/resources" fileName="eks" stack="core-use1-auto"/>

Validate the `core-auto` deployment using Echo Server. For example: https://echo.use1.auto.core.acme-svc.com/
</Step>
Validate the `core-auto` deployment using Echo Server. For example: https://echo.use1.auto.core.acme-svc.com/
</Step>

<Step>
### <StepNumber/> Deploy the Actions Runner Controller
<Step>
### <StepNumber/> Deploy the Actions Runner Controller

Finally, deploy the `actions-runner-controller` component with the following command:
Finally, deploy the `actions-runner-controller` component with the following command:

```bash
atmos terraform deploy eks/actions-runner-controller -s core-use1-auto
```
```bash
atmos terraform deploy eks/actions-runner-controller -s core-use1-auto
```

</Step>
</Step>

<Step>
### <StepNumber/> Using Webhook Driven Autoscaling (Click Ops)
<Step>
### <StepNumber/> Using Webhook Driven Autoscaling (Click Ops)

To use the Webhook Driven autoscaling, you must also install the GitHub organization-level webhook after deploying the
component (specifically, the webhook server). The URL for the webhook is determined by the `webhook.hostname_template`
and where it is deployed. Recommended URL is
`https://gha-webhook.[environment].[stage].[tenant].[service-discovery-domain]`, which for this organization would be
`https://gha-webhook.use1.auto.core.acme-svc.com`
To use the Webhook Driven autoscaling, you must also install the GitHub organization-level webhook after deploying the
component (specifically, the webhook server). The URL for the webhook is determined by the `webhook.hostname_template`
and where it is deployed. Recommended URL is
`https://gha-webhook.[environment].[stage].[tenant].[service-discovery-domain]`, which for this organization would be
`https://gha-webhook.use1.auto.core.acme-svc.com`

As a GitHub organization admin, go to
`https://github.com/organizations/acme/settings/hooks`, and then:
As a GitHub organization admin, go to
`https://github.com/organizations/acme/settings/hooks`, and then:

- Click "Add webhook" and create a new webhook with the following settings:
- Payload URL: copy from Terraform output `webhook_payload_url`
- Content type: `application/json`
- Secret: whatever you configured in the secret above
- Which events would you like to trigger this webhook:
- Select "Let me select individual events"
- Uncheck everything ("Pushes" is likely the only thing already selected)
- Check "Workflow jobs"
- Ensure that "Active" is checked (should be checked by default)
- Click "Add webhook" at the bottom of the settings page
- Click "Add webhook" and create a new webhook with the following settings:
- Payload URL: copy from Terraform output `webhook_payload_url`
- Content type: `application/json`
- Secret: whatever you configured in the secret above
- Which events would you like to trigger this webhook:
- Select "Let me select individual events"
- Uncheck everything ("Pushes" is likely the only thing already selected)
- Check "Workflow jobs"
- Ensure that "Active" is checked (should be checked by default)
- Click "Add webhook" at the bottom of the settings page

After the webhook is created, select "edit" for the webhook and go to the "Recent Deliveries" tab and verify that there
is a delivery (of a "ping" event) with a green check mark. If not, verify all the settings and consult the logs of the
`actions-runner-controller-github-webhook-server` pod.
</Step>
After the webhook is created, select "edit" for the webhook and go to the "Recent Deliveries" tab and verify that there
is a delivery (of a "ping" event) with a green check mark. If not, verify all the settings and consult the logs of the
`actions-runner-controller-github-webhook-server` pod.
</Step>
</Steps>

# Related Topics

Loading

0 comments on commit 7e9fef2

Please sign in to comment.