Skip to content

Commit

Permalink
chore: remodel the aws-cdk repo to support v2 development only (#24376)
Browse files Browse the repository at this point in the history 
Restructures the repository to reflect the current state of package publishing. Combines all of the stable packages within @aws-cdk/* namespace into aws-cdk-lib, and separates the unstable portions of the alpha modules into their own packages depending on `aws-cdk-lib` for their L1s. Changes how code generation works to run during the build of `aws-cdk-lib` and new cloudformation resources are added automatically when the spec is updated.

Removed all of the integration tests and put them into their own package to avoid cyclic dependencies. Integ tests for alpha packages remain with their package source.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
@madeline-k
madeline-k authored Mar 31, 2023
1 parent 5003cad commit a2c633f
Show file tree
Hide file tree
Showing 14,321 changed files with 263,845 additions and 345,706 deletions.
The diff you're trying to view is too large. We only load the first 3000 changed files.
120 changes: 28 additions & 92 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,35 +88,35 @@ The CDK uses [jsii](https://github.com/aws/jsii/) as its primary build system. j
typescript-compliant source code and produce polyglot libraries, such as, in Java, .NET, Python and Go.

The repo contains `packages/` directory that contains the CDK public modules. The source code for the IAM module in the
CDK can be found at the location `packages/@aws-cdk/aws-iam`.
CDK can be found at the location `packages/aws-cdk-lib/aws-iam`.
The repo also contains the `tools/` directory that holds custom build tooling (modeled as private npm packages)
specific to the CDK.

### Build

The full build of the CDK takes a long time to complete; 1-2 hours depending on the performance of the build machine.
However, most first time contributions will require changing only one CDK module, sometimes two. A full build of the
CDK is not required in these cases.

If you want to work on the `@aws-cdk/aws-ec2` module, the following command will build just the EC2 module and any
necessary dependencies.
The full build of all of the packages within the repository can take a few minutes, about 20 when all tests are run.
Most contributions only require working on a single package, usually `aws-cdk-lib`. To build this package for the first
time, you can execute the following to build it and it's dependencies.

```console
$ cd packages/@aws-cdk/aws-ec2
$ cd packages/aws-cdk-lib
$ ../../../scripts/buildup
```

Note: The `buildup` command is resumable. If your build fails, you can fix the issue and run `buildup --resume` to
resume.

At this point, you can run build and test the `aws-ec2` module by running
At this point, you can run build and test the `aws-cdk-lib` module by running

```console
$ cd packages/@aws-cdk/aws-ec2
$ cd packages/aws-cdk-lib
$ yarn build
$ yarn test
```

To cut down on iteration time as you develop, you can run `yarn watch` within the `aws-cdk-lib` directory to keep
some of the build state in memory and incrementally rebuild as you make changes.

However, if you wish to build the entire repository, the following command will achieve this.

```console
Expand All @@ -137,12 +137,12 @@ Packing involves generating CDK code in the various target languages and packagi
respective package managers. Once in a while, these will need to be generated either to test the experience of a new
feature, or reproduce a packaging failure.

To package a specific module, say the `@aws-cdk/aws-ec2` module:
To package a specific module, say the `aws-cdk-lib` module:

```console
$ cd <root-of-cdk-repo>
$ docker run --rm --net=host -it -v $PWD:$PWD -w $PWD jsii/superchain:1-buster-slim
docker$ cd packages/@aws-cdk/aws-ec2
docker$ cd packages/aws-cdk-lib
docker$ ../../../scripts/foreach.sh --up yarn run package
docker$ exit
```
Expand Down Expand Up @@ -317,12 +317,12 @@ CDK integration tests.
We've added a watch feature to the CDK that builds your code as you type it. Start this by running `yarn watch` for
each module that you are modifying.

For example, watch the EC2 and IAM modules in a second terminal session:
For example, watch the aws-cdk-lib and aws-cdk modules in a second terminal session:

```console
$ cd packages/@aws-cdk/aws-ec2
$ cd packages/aws-cdk-lib
$ yarn watch & # runs in the background
$ cd packages/@aws-cdk/aws-iam
$ cd packages/aws-cdk
$ yarn watch & # runs in the background
```

Expand Down Expand Up @@ -637,10 +637,10 @@ The README file contains code snippets written as typescript code. Code snippets
(such as `` ```ts ``) will be automatically extracted, compiled and translated to other languages when the
during the [pack](#pack) step. We call this feature 'rosetta'.

You can run rosetta on the EC2 module (or any other module) by running:
You can run rosetta on the aws-cdk-lib module (or any other module) by running:

```console
$ cd packages/@aws-cdk/aws-ec2
$ cd packages/aws-cdk-lib
$ yarn rosetta:extract --strict
```

Expand Down Expand Up @@ -696,14 +696,14 @@ cases where some of those do not apply - good judgement is to be applied):
- Types from the documented module should be **un-qualified**:

```ts
// An example in the @aws-cdk/core library, which defines Duration
// An example in the aws-cdk-lib library, which defines Duration
Duration.minutes(15);
```

- Types from other modules should be **qualified**:

```ts
// An example in the @aws-cdk/core library, using something from @aws-cdk/aws-s3
// An example in the aws-cdk-lib library, using something from aws-cdk-lib/aws-s3
const bucket = new s3.Bucket(this, 'Bucket');
// ...rest of the example...
```
Expand All @@ -712,7 +712,7 @@ cases where some of those do not apply - good judgement is to be applied):
necessary for compilation but unimportant to the example:

```ts
// An example about adding a stage to a pipeline in the @aws-cdk/pipelines library
// An example about adding a stage to a pipeline in the aws-cdk-lib/pipelines library
declare const pipeline: pipelines.CodePipeline;
declare const myStage: Stage;
pipeline.addStage(myStage);
Expand Down Expand Up @@ -766,7 +766,7 @@ Consequently, there are two useful scripts that are built on top of `foreach.sh`
All linters are executed automatically as part of the build script, `yarn build`.

They can also be executed independently of the build script. From the root of a specific package (e.g.
`packages/@aws-cdk/aws-ec2`), run the following command to execute all the linters on that package -
`packages/aws-cdk-lib`), run the following command to execute all the linters on that package -

```bash
yarn lint
Expand Down Expand Up @@ -878,88 +878,24 @@ $ cdk -a some.app.js synth | $awscdk/scripts/template-deps-to-dot | dot -Tpng >

You can use `find-cycles` to print a list of internal dependency cycles:

```shell
$ scripts/find-cycles.sh
Cycle: @aws-cdk/aws-iam => @aws-cdk/assert => aws-cdk => @aws-cdk/aws-s3 => @aws-cdk/aws-kms => @aws-cdk/aws-iam
Cycle: @aws-cdk/assert => aws-cdk => @aws-cdk/aws-s3 => @aws-cdk/aws-kms => @aws-cdk/assert
Cycle: @aws-cdk/aws-iam => @aws-cdk/assert => aws-cdk => @aws-cdk/aws-s3 => @aws-cdk/aws-iam
Cycle: @aws-cdk/assert => aws-cdk => @aws-cdk/aws-s3 => @aws-cdk/assert
Cycle: @aws-cdk/assert => aws-cdk => @aws-cdk/aws-cloudformation => @aws-cdk/assert
Cycle: @aws-cdk/aws-iam => @aws-cdk/assert => aws-cdk => @aws-cdk/util => @aws-cdk/aws-iam
Cycle: @aws-cdk/aws-sns => @aws-cdk/aws-lambda => @aws-cdk/aws-codecommit => @aws-cdk/aws-sns
Cycle: @aws-cdk/aws-sns => @aws-cdk/aws-lambda => @aws-cdk/aws-codecommit => @aws-cdk/aws-codepipeline => @aws-cdk/aws-sns
```

## Running CLI integration tests

The CLI package (`packages/aws-cdk`) has some integration tests that aren't
run as part of the regular build, since they have some particular requirements.
See the [CLI CONTRIBUTING.md file](packages/aws-cdk/CONTRIBUTING.md) for
more information on running those tests.

## Building aws-cdk-lib

In AWS CDK v2, all stable libraries are packaged into a single monolithic
package and published as `aws-cdk-lib`. In most cases, you can iterate on a
single module's directory as previously described in this document (e.g.
`packages/@aws-cdk/aws-s3`). In some cases, you might need to build
`aws-cdk-lib`:

```
# Generate all of the L1s first. If you have already done a full build in the repository, you can skip this.
cd <CDK repo root>/
./scripts/gen.sh
# Generate and build `aws-cdk-lib`
cd packages/aws-cdk-lib
yarn build
```

The commands above perform the following steps:
1. Run `yarn install` to install all dependencies
2. Generate `.generated.ts` files in each `packages/@aws-cdk/aws-<service>`
directory. These files contain TypeScript source code for all of the L1 (Cfn)
Constructs, and are generated from the [CloudFormation Resource
Specification](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html).
3. Copy the `.ts` source code files from each `packages/@aws-cdk/aws-<service>`
directory to the corresponding `packages/aws-cdk-lib/aws-<service>`
directory.
4. Compile `aws-cdk-lib`.

Running unit tests and integration tests still has to be performed in each
module's `packages/@aws-cdk` directory.

## Building and testing v2 -alpha packages

In AWS CDK v2, all experimental libraries are published separately with an
-alpha suffix. In most cases, you can iterate on a single module's directory as
already described in this document (e.g. `packages/@aws-cdk/aws-amplify`). If
you need to generate and iterate on the alpha package, here are the steps. The
main differences between the alpha package is naming of the package, and import
statements.

First, make sure the following packages are built:
- packages/@aws-cdk/assert
- packages/aws-cdk-lib
- tools/individual-pkg-gen

The following command will create all of the alpha packages by copying files
from their source directories under `packages/@aws-cdk/aws-<service>`, and it
will build and run unit tests for all of them. This is sometimes too much for a
developer machine or laptop.
Modules that are not stable are vended separately from `aws-cdk-lib`. These packages are found in the
`packages/@aws-cdk` directory and are marked `stability: 'experimental'` in their package.json files.
This means they will be given the `alpha` version from the `version.v2.json` when published and they
cannot be taken as dependencies by `aws-cdk-lib`

```
<CDK repo root>/scripts/transform.sh
```

To only copy and transform the source files, and then build and test one
alpha package at a time, use the following:
Experimental packages are used to develop new constructs and experiment with their APIs before marking
them as stable and including them within `aws-cdk-lib`. Once they are included in `aws-cdk-lib`, no
more breaking api changes can be made.

```
<CDK repo root>/scripts/transform.sh --skip-build
cd packages/individual-packages/aws-<service>
yarn build+test
```
## Changing Cloud Assembly Schema

If you plan on making changes to the `cloud-assembly-schema` package, make sure you familiarize yourself with
Expand Down
1 change: 0 additions & 1 deletion buildspec-pr.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ phases:
build:
commands:
- /bin/bash ./build.sh
- /bin/bash ./scripts/transform.sh
# After compilation, run Rosetta (using the cache if available).
# This will print errors, and fail the build if there are compilation errors in any packages marked as 'strict'.
- /bin/bash ./scripts/run-rosetta.sh
Expand Down
41 changes: 41 additions & 0 deletions buildspec-remodel.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
version: 0.2

env:
secrets-manager:
TOKEN: $TOKEN_SECRET_ARN
variables:
NODE_OPTIONS: --max_old_space_size=8192

phases:
install:
commands:
# CodeBuild always runs as root, allow npm to operate as such
- npm config set unsafe-perm true
- mkdir ${HOME}/aws-cdk && cd ${HOME}/aws-cdk
# Configure git and git credentials
- git config --global user.email "${COMMIT_EMAIL}"
- git config --global user.name "${COMMIT_USERNAME}"
- echo "Retrieving gh token from secretsmanager"
- git clone https://$TOKEN@github.com/aws/aws-cdk -b feat/repo-restructure .
- git checkout ${CODEBUILD_RESOLVED_SOURCE_VERSION}
# Install yarn if it wasn't already present in the image
- yarn --version || npm -g install yarn
- yarn install --frozen-lockfile
- /sbin/sysctl -w vm.max_map_count=2251954
build:
commands:
- cd ${HOME}/aws-cdk
- npx lerna run build --scope @aws-cdk/remodel --include-dependencies
- npx remodel ${PWD} --tmp-dir ${HOME}/remodel --no-clean --dry-run
post_build:
commands:
- cd ${HOME}/remodel
- git add -A
- git commit -m "Execute remodel"
- git remote add aws-cdk https://$TOKEN@github.com/aws/aws-cdk
- git fetch aws-cdk
- git push aws-cdk +feat/repo-restructure:feat/remodel
artifacts:
files:
- "**/*"
base-directory: ${HOME}/remodel
1 change: 0 additions & 1 deletion buildspec.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ phases:
- 'if ${BUMP_CANDIDATE:-false}; then /bin/bash ./scripts/bump-candidate.sh; fi'
- /bin/bash ./scripts/align-version.sh
- /bin/bash ./build.sh
- /bin/bash ./scripts/transform.sh
post_build:
commands:
# Short-circuit: Don't run pack if the above build failed.
Expand Down
13 changes: 13 additions & 0 deletions packages/@aws-cdk-testing/framework-integ/.eslintrc.js
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
const baseConfig = require('../../../tools/@aws-cdk/cdk-build-tools/config/eslintrc');
baseConfig.parserOptions.project = __dirname + '/tsconfig.json';
module.exports = {
...baseConfig,
ignorePatterns: [
...baseConfig.ignorePatterns,
'**/*.snapshot/**/*'
],
rules: {
...baseConfig.rules,
'import/order': 'off',
}
};
8 changes: 8 additions & 0 deletions packages/@aws-cdk-testing/framework-integ/.gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@

.LAST_BUILD
*.snk
junit.xml
!.eslintrc.js
.nyc_output
coverage
nyc.config.js
7 changes: 7 additions & 0 deletions packages/@aws-cdk-testing/framework-integ/.npmignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@

.LAST_BUILD
*.snk
junit.xml
.eslintrc.js
# exclude cdk artifacts
**/cdk.out
0 packages/@aws-cdk/alexa-ask/LICENSE → .../@aws-cdk-testing/framework-integ/LICENSE
View file
File renamed without changes.
16 changes: 16 additions & 0 deletions packages/@aws-cdk-testing/framework-integ/NOTICE
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
AWS Cloud Development Kit (AWS CDK)
Copyright 2018-2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Third party attributions of this package can be found in the THIRD_PARTY_LICENSES file
63 changes: 63 additions & 0 deletions packages/@aws-cdk-testing/framework-integ/package.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"name": "@aws-cdk-testing/framework-integ",
"description": "Integration tests for aws-cdk-lib",
"private": true,
"version": "0.0.0",
"scripts": {
"build": "cdk-build",
"watch": "cdk-watch",
"lint": "cdk-lint",
"pkglint": "pkglint -f",
"test": "cdk-test",
"package": "cdk-package",
"build+test": "yarn build && yarn test",
"build+extract": "yarn build",
"build+test+package": "yarn build+test && yarn package",
"build+test+extract": "yarn build+test"
},
"pkglint": {
"exclude": [
"package-info/repository",
"dependencies/cdk-point-dependencies"
]
},
"author": {
"name": "Amazon Web Services",
"url": "https://aws.amazon.com",
"organization": true
},
"license": "Apache-2.0",
"devDependencies": {
"@aws-cdk/cdk-build-tools": "0.0.0",
"delay": "5.0.0",
"@aws-cdk/pkglint": "0.0.0"
},
"dependencies": {
"@aws-cdk/lambda-layer-kubectl-v24": "^2.0.100",
"aws-cdk-lib": "0.0.0",
"aws-sdk": "^2.1317.0",
"aws-sdk-mock": "5.6.0",
"cdk8s": "^2.7.15",
"cdk8s-plus-24": "2.4.40",
"constructs": "^10.0.0",
"@aws-cdk/integ-tests-alpha": "0.0.0"
},
"repository": {
"url": "https://github.com/aws/aws-cdk.git",
"type": "git",
"directory": "packages/@aws-cdk-testing/framework-integ"
},
"keywords": [
"aws",
"cdk"
],
"homepage": "https://github.com/aws/aws-cdk",
"engines": {
"node": ">= 14.15.0"
},
"stability": "experimental",
"maturity": "experimental",
"publishConfig": {
"tag": "latest"
}
}
1 change: 1 addition & 0 deletions ...esting/framework-integ/test/aws-apigateway/test/authorizers/integ.cognito-authorizer.d.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export {};
Loading

0 comments on commit a2c633f

Please sign in to comment.