V4vpc (widdix#74)

* first version of the new vpc templates * improved test coverage * execute tests in parallel (two at a time) * finished docs
luuthanhminh · May 24, 2017 · ad4f023 · ad4f023
1 parent ab211d9
commit ad4f023
Show file tree

Hide file tree

Showing 29 changed files with 956 additions and 119 deletions.
diff --git a/docs/migrate-v4.md b/docs/migrate-v4.md
@@ -0,0 +1,11 @@
+# Migrate from v3 to v4
+
+There is a breaking change in the VPC layout going from v3 to v4. This change enables HA NAT Gateway/Instance (https://github.com/widdix/aws-cf-templates/issues/65). Updating your VPC can will cause connectivity interruptions in `SubnetBPrivate`, `SubnetCPrivate`, and `SubnetDPrivate` until you created new NAT Gateway/Instance for each `SubnetZone` in step 3b.
+
+> None of our templates launch workloads into private subnets that require Internet access. This could only be an issue if you use other workloads.
+
+1. Update VPC stacks with the matching updated template (`vpc/vpc-2azs.yaml`, `vpc/vpc-2azs-legacy.yaml`, `vpc/vpc-3azs.yaml`, `vpc/vpc-3azs-legacy.yaml`, `vpc/vpc-4azs.yaml`, `vpc/vpc-4azs-legacy.yaml`), leave the parameters as they are.
+2. Update VPC Endpoint stacks with the matching updated template (`vpc/vpc-endpoint-s3.yaml`), leave the parameters as they are.
+3. Update VPC NAT Gateway/Instance stacks
+  a. Update VPC NAT Gateway/Instance stacks with the matching updated template (`vpc/vpc-nat-gateway.yaml`, `vpc/vpc-nat-instance.yaml`), set `SubnetZone` parameter to `A`.
+  b. If you updated anything in a. create a new VPC NAT Gateway/Instance stack for each missing `SubnetZone` by setting the `SubnetZone` parameter to `B`, `C`, or `D`.
diff --git a/docs/security.md b/docs/security.md
@@ -107,7 +107,6 @@ If you want to use an external S3 bucket, the bucket needs to have the following
     }
   }]
 }
-
 ```
 
 Replace `$ExternalTrailBucket` with the name of your bucket, and add a row for every account you want to write from `$AccountId[*]`.
@@ -147,7 +146,7 @@ If you want to use an external S3 bucket, the bucket needs to have the following
       "Effect": "Allow",
       "Principal": {
         "Service": [
-         "config.amazonaws.com"    
+         "config.amazonaws.com"
         ]
       },
       "Action": "s3:PutObject",
@@ -156,15 +155,14 @@ If you want to use an external S3 bucket, the bucket needs to have the following
         "arn:aws:s3:::$ExternalConfigBucket/AWSLogs/$AccountId[0]/Config/*",
         "arn:aws:s3:::$ExternalConfigBucket/AWSLogs/$AccountId[2]/Config/*"
       ],
-      "Condition": { 
-        "StringEquals": { 
+      "Condition": {
+        "StringEquals": {
           "s3:x-amz-acl": "bucket-owner-full-control" 
         }
       }
     }
   ]
 }
-
 ```
 
 Replace `$ExternalTrailBucket` with the name of your bucket, and add a row for every account you want to write from `$AccountId[*]`.
diff --git a/docs/vpc.md b/docs/vpc.md
@@ -47,7 +47,9 @@ This template describes a VPC with four private and four public subnets.
 If you have an existing VPC you can wrap it into our required form using a legacy VPC wrapper: [![Launch Stack](./img/launch-stack.png)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=vpc-4azs&templateURL=https://s3-eu-west-1.amazonaws.com/widdix-aws-cf-templates-releases-eu-west-1/__VERSION__/vpc/vpc-4azs-legacy.yaml) 
 
 # NAT Gateway
-This template describes a NAT Gateway that forwards HTTP, HTTPS and NTP traffic from private subnets to the Internet.
+This template describes a NAT Gateway that forwards HTTP, HTTPS and NTP traffic from a single private subnet to the Internet. You need one stack per availability zone. Example: If you use the `vpc-2azs.yaml` template, you will need two Nat Gateway stack in `A` and `B`.
+
+> You need one Gateway in each `SubnetZone` (e.g. `A` and `B` in `vpc-2azs.yaml`).
 
 ![Architecture](./img/vpc-nat-gateway.png)
 
@@ -64,11 +66,10 @@ This template describes a NAT Gateway that forwards HTTP, HTTPS and NTP traffic
 ## Dependencies
 * `vpc/vpc-*azs.yaml` (**required**)
 
-## Limitations
-* The NAT Gateway is a single point of failure because it runs only in one Subnet (and therefore in one Availability Zone): https://github.com/widdix/aws-cf-templates/issues/65
-
 # NAT instance
-This template describes a **highly available** Network Address Translation (NAT) instance that forwards HTTP, HTTPS and NTP traffic from private subnets to the Internet.
+This template describes a **highly available** Network Address Translation (NAT) instance that forwards HTTP, HTTPS and NTP traffic from a single private subnet to the Internet. You need one stack per availability zone. Example: If you use the `vpc-2azs.yaml` template, you will need two Nat Gateway stack in `A` and `B`.
+
+> You need one Instance in each `SubnetZone` (e.g. `A` and `B` in `vpc-2azs.yaml`).
 
 ![Architecture](./img/vpc-nat-instance.png)
 ## Installation Guide
@@ -87,9 +88,6 @@ This template describes a **highly available** Network Address Translation (NAT)
 * `vpc/vpc-ssh-bastion.yaml` (recommended)
 * `operations/alert.yaml` (recommended)
 
-## Limitations
-* Only one EC2 instance is managed by the ASG. In case of an outage the instance will be replaced within 5 minutes.
-
 # SSH bastion host/instance
 This template describes a **highly available** SSH bastion host/instance. SSH Port 22 is open to the world. You can enable the default ec2-user access protected by the referenced EC2 KeyPair. You can also enable personalized SSH access by using the IAM users and their configured public keys. Use `ssh -A user@ip` to enable forwarding of the authentication agent connection when connection to the bastion host.
 **Users are not able to sudo on the bastion host/instance! That's very important for security. Why? SSH places a SSH_AUTH_SOCK file into the /tmp directoy only accessible by the user. If you have root you could use any of those files and jump to other machines as another user!**

diff --git a/operations/alert.yaml b/operations/alert.yaml
@@ -18,15 +18,19 @@ Parameters:
   Email:
     Description: 'Optional email address that will receive alerts'
     Type: String
+    Default: ''
   HttpEndpoint:
     Description: 'Optional HTTP endpoint that will receive alerts via POST requests'
     Type: String
+    Default: ''
   HttpsEndpoint:
     Description: 'Optional HTTPS endpoint that will receive alerts via POST requests  (can be marbot.io)'
     Type: String
+    Default: ''
   FallbackEmail:
     Description: 'Optional email address that will receive alerts if alerts can not be delivered'
     Type: String
+    Default: ''
 Conditions:
   HasEmail: !Not [!Equals [!Ref Email, '']]
   HasHttpEndpoint: !Not [!Equals [!Ref HttpEndpoint, '']]

diff --git a/test/pom.xml b/test/pom.xml
@@ -70,7 +70,7 @@
             <dependency>
                 <groupId>com.amazonaws</groupId>
                 <artifactId>aws-java-sdk-bom</artifactId>
-                <version>1.11.95</version>
+                <version>1.11.133</version>
                 <type>pom</type>
                 <scope>import</scope>
             </dependency>
@@ -82,12 +82,21 @@
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-compiler-plugin</artifactId>
-                <version>3.5.1</version>
+                <version>3.6.1</version>
                 <configuration>
                     <source>1.8</source>
                     <target>1.8</target>
                 </configuration>
             </plugin>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-surefire-plugin</artifactId>
+                <version>2.20</version>
+                <configuration>
+                    <parallel>methods</parallel>
+                    <threadCount>2</threadCount>
+                </configuration>
+            </plugin>
         </plugins>
     </build>
 </project>
diff --git a/test/src/test/java/de/widdix/awscftemplates/AAWSTest.java b/test/src/test/java/de/widdix/awscftemplates/AAWSTest.java
@@ -1,14 +1,21 @@
 package de.widdix.awscftemplates;
 
 import com.amazonaws.auth.*;
+import com.amazonaws.regions.DefaultAwsRegionProviderChain;
+import com.amazonaws.services.s3.model.Region;
 import com.amazonaws.services.ec2.AmazonEC2;
 import com.amazonaws.services.ec2.AmazonEC2ClientBuilder;
 import com.amazonaws.services.ec2.model.*;
+import com.amazonaws.services.ec2.model.Filter;
 import com.amazonaws.services.route53.AmazonRoute53ClientBuilder;
 import com.amazonaws.services.route53.model.*;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.amazonaws.services.s3.model.*;
 import com.amazonaws.services.securitytoken.AWSSecurityTokenService;
 import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClientBuilder;
 import com.amazonaws.services.route53.AmazonRoute53;
+import com.amazonaws.services.securitytoken.model.GetCallerIdentityRequest;
 
 import java.util.List;
 import java.util.UUID;
@@ -23,16 +30,22 @@ public abstract class AAWSTest extends ATest {
 
     private AmazonRoute53 route53;
 
+    private final AmazonS3 s3;
+
+    private final AWSSecurityTokenService sts;
+
     public AAWSTest() {
         super();
         if (Config.has(Config.Key.IAM_ROLE_ARN)) {
-            final AWSSecurityTokenService sts = AWSSecurityTokenServiceClientBuilder.standard().withCredentials(new DefaultAWSCredentialsProviderChain()).build();
-            this.credentialsProvider = new STSAssumeRoleSessionCredentialsProvider.Builder(Config.get(Config.Key.IAM_ROLE_ARN), IAM_SESSION_NAME).withStsClient(sts).build();
+            final AWSSecurityTokenService local = AWSSecurityTokenServiceClientBuilder.standard().withCredentials(new DefaultAWSCredentialsProviderChain()).build();
+            this.credentialsProvider = new STSAssumeRoleSessionCredentialsProvider.Builder(Config.get(Config.Key.IAM_ROLE_ARN), IAM_SESSION_NAME).withStsClient(local).build();
         } else {
             this.credentialsProvider = new DefaultAWSCredentialsProviderChain();
         }
         this.ec2 = AmazonEC2ClientBuilder.standard().withCredentials(this.credentialsProvider).build();
         this.route53 = AmazonRoute53ClientBuilder.standard().withCredentials(this.credentialsProvider).build();
+        this.s3 = AmazonS3ClientBuilder.standard().withCredentials(this.credentialsProvider).build();
+        this.sts = AWSSecurityTokenServiceClientBuilder.standard().withCredentials(this.credentialsProvider).build();
     }
 
     protected final KeyPair createKey(final String keyName) {
@@ -96,6 +109,37 @@ protected final void deleteDomain(final String prefix) {
         }
     }
 
+    protected final void createBucket(final String name, final String policy) {
+        this.s3.createBucket(new CreateBucketRequest(name, Region.fromValue(this.getRegion())));
+        this.s3.setBucketPolicy(name, policy);
+    }
+
+    protected final void emptyBucket(final String name) {
+        ObjectListing objectListing = s3.listObjects(name);
+        while (true) {
+            objectListing.getObjectSummaries().forEach((summary) -> s3.deleteObject(name, summary.getKey()));
+            if (objectListing.isTruncated()) {
+                objectListing = s3.listNextBatchOfObjects(objectListing);
+            } else {
+                break;
+            }
+        }
+        VersionListing versionListing = s3.listVersions(new ListVersionsRequest().withBucketName(name));
+        while (true) {
+            versionListing.getVersionSummaries().forEach((vs) -> s3.deleteVersion(name, vs.getKey(), vs.getVersionId()));
+            if (versionListing.isTruncated()) {
+                versionListing = s3.listNextBatchOfVersions(versionListing);
+            } else {
+                break;
+            }
+        }
+    }
+
+    protected final void deleteBucket(final String name) {
+        this.emptyBucket(name);
+        this.s3.deleteBucket(new DeleteBucketRequest(name));
+    }
+
     protected final Vpc getDefaultVPC() {
         final DescribeVpcsResult res = this.ec2.describeVpcs(new DescribeVpcsRequest().withFilters(new Filter().withName("isDefault").withValues("true")));
         return res.getVpcs().get(0);
@@ -115,6 +159,14 @@ protected final SecurityGroup getDefaultSecurityGroup() {
         return res.getSecurityGroups().get(0);
     }
 
+    protected final String getRegion() {
+        return new DefaultAwsRegionProviderChain().getRegion();
+    }
+
+    protected final String getAccount() {
+        return this.sts.getCallerIdentity(new GetCallerIdentityRequest()).getAccount();
+    }
+
     protected final String random8String() {
         final String uuid = UUID.randomUUID().toString().replace("-", "").toLowerCase();
         final int beginIndex = (int) (Math.random() * (uuid.length() - 7));

diff --git a/test/src/test/java/de/widdix/awscftemplates/operations/TestAlert.java b/test/src/test/java/de/widdix/awscftemplates/operations/TestAlert.java
@@ -0,0 +1,21 @@
+package de.widdix.awscftemplates.operations;
+
+import de.widdix.awscftemplates.ACloudFormationTest;
+import org.junit.Test;
+
+public class TestAlert extends ACloudFormationTest {
+
+    @Test
+    public void test() {
+        final String stackName = "alert-" + this.random8String();
+        try {
+            this.createStack(stackName,
+                    "operations/alert.yaml"
+            );
+            // TODO how can we check if this stack works?
+        } finally {
+            this.deleteStack(stackName);
+        }
+    }
+
+}
diff --git a/test/src/test/java/de/widdix/awscftemplates/security/TestAccountPasswordPolicy.java b/test/src/test/java/de/widdix/awscftemplates/security/TestAccountPasswordPolicy.java
@@ -0,0 +1,21 @@
+package de.widdix.awscftemplates.security;
+
+import de.widdix.awscftemplates.ACloudFormationTest;
+import org.junit.Test;
+
+public class TestAccountPasswordPolicy extends ACloudFormationTest {
+
+    @Test
+    public void test() {
+        final String stackName = "account-password-policy-" + this.random8String();
+        try {
+            this.createStack(stackName,
+                    "security/account-password-policy.yaml"
+            );
+            // TODO how can we check if this stack works?
+        } finally {
+            this.deleteStack(stackName);
+        }
+    }
+
+}
diff --git a/test/src/test/java/de/widdix/awscftemplates/security/TestCloudtrail.java b/test/src/test/java/de/widdix/awscftemplates/security/TestCloudtrail.java
@@ -0,0 +1,56 @@
+package de.widdix.awscftemplates.security;
+
+import com.amazonaws.services.cloudformation.model.Parameter;
+import de.widdix.awscftemplates.ACloudFormationTest;
+import org.junit.Test;
+
+public class TestCloudtrail extends ACloudFormationTest {
+
+    @Test
+    public void test() {
+        final String stackName = "cloudtrail-" + this.random8String();
+        final String bucketName = "cloudtrail-" + this.random8String();
+        final String bucketPolicy = "{\n" +
+                "  \"Version\": \"2012-10-17\",\n" +
+                "  \"Statement\": [{\n" +
+                "    \"Sid\": \"AWSCloudTrailAclCheck\",\n" +
+                "    \"Effect\": \"Allow\",\n" +
+                "    \"Principal\": {\n" +
+                "      \"Service\": \"cloudtrail.amazonaws.com\"\n" +
+                "    },\n" +
+                "    \"Action\": \"s3:GetBucketAcl\",\n" +
+                "    \"Resource\": \"arn:aws:s3:::"+ bucketName + "\"\n" +
+                "  }, {\n" +
+                "    \"Sid\": \"AWSCloudTrailWrite\",\n" +
+                "    \"Effect\": \"Allow\",\n" +
+                "    \"Principal\": {\n" +
+                "      \"Service\": \"cloudtrail.amazonaws.com\"\n" +
+                "    },\n" +
+                "    \"Action\": \"s3:PutObject\",\n" +
+                "    \"Resource\": [\n" +
+                "      \"arn:aws:s3:::"+ bucketName + "/AWSLogs/" + this.getAccount() + "/*\"\n" +
+                "    ],\n" +
+                "    \"Condition\": {\n" +
+                "      \"StringEquals\": {\n" +
+                "        \"s3:x-amz-acl\": \"bucket-owner-full-control\"\n" +
+                "      }\n" +
+                "    }\n" +
+                "  }]\n" +
+                "}";
+        try {
+            this.createBucket(bucketName, bucketPolicy);
+            try {
+                this.createStack(stackName,
+                        "security/cloudtrail.yaml",
+                        new Parameter().withParameterKey("ExternalTrailBucket").withParameterValue(bucketName)
+                );
+                // TODO how can we check if this stack works?
+            } finally {
+                this.deleteStack(stackName);
+            }
+        } finally {
+            this.deleteBucket(bucketName);
+        }
+    }
+
+}