prepare v8 release (widdix#214)

* prepare v8 * [Improvement] ecs/cluster - Migrate from ECS optimized Amazon Linux to Amazon Linux 2
luuthanhminh · Nov 12, 2018 · 7cda356 · 7cda356
1 parent f51193e
commit 7cda356
Show file tree

Hide file tree

Showing 18 changed files with 158 additions and 107 deletions.
diff --git a/docs/migrate-v5.md b/docs/migrate-v5.md
@@ -17,3 +17,7 @@ If you want to migrate stacks based on `opertions/backup-dynamodb.yaml`, you hav
 The auto scaling rules have changed and new parameters where added (all with default values). The cluster now scales based on a new metric: `SchedulableContainers`. Scaling based on SchedulableContainers is described in detail here: http://garbe.io/blog/2017/04/12/a-better-solution-to-ecs-autoscaling/
 
 If you want to migrate stacks based on `ecs/cluster.yaml`, you have to check your ECS task definitions and look for the highest memory and cpu reservation and set the new parameters `ContainerMaxMemory` and `ContainerMaxCPU` to those values.
+
+# Migrate from v3 to v4
+
+[Learn more](./migrate-v4/)
diff --git a/docs/migrate-v6.md b/docs/migrate-v6.md
@@ -9,3 +9,11 @@ There is one smaller breaking change that affects mostly all templates: The opti
 [New Relic Servers](https://docs.newrelic.com/docs/servers) and Legacy Alerts are going away on or before May 15, 2018. That's why we removed them from the templates.
 
 If you want to migrate stacks, you can no longer provide the `NewRelicLicenseKey` parameter.
+
+# Migrate from v4 to v5
+
+[Learn more](./migrate-v5/)
+
+# Migrate from v3 to v4
+
+[Learn more](./migrate-v4/)
diff --git a/docs/migrate-v7.md b/docs/migrate-v7.md
@@ -56,6 +56,20 @@ The new static website makes use of Lambda@Edge.
 5. `SystemsManagerAccess` will be enabled by default.
 6. The parameter `BlogVersion` was removed to make updates possible.
 
-# Deprecation warnings
+## Deprecation warnings
 
 * vpc/vpc-*azs-legacy: Paramater `ClassB` and output `ClassB` will be removed in the next version (v8).
+
+
+# Migrate from v5 to v6
+
+[Learn more](./migrate-v6/)
+
+# Migrate from v4 to v5
+
+[Learn more](./migrate-v5/)
+
+# Migrate from v3 to v4
+
+[Learn more](./migrate-v4/)
+
diff --git a/docs/migrate-v8.md b/docs/migrate-v8.md
@@ -0,0 +1,30 @@
+<iframe src="https://ghbtns.com/github-btn.html?user=widdix&repo=aws-cf-templates&type=star&count=true&size=large" frameborder="0" scrolling="0" width="160px" height="30px"></iframe>
+
+> **New**: Manage Free Templates for AWS CloudFormation with the [widdix CLI](./cli/)
+
+# Migrate from v7 to v8
+
+## ecs/cluster
+
+1. Amazon Linux (ECS optimized) is updated to Amazon Linux 2 (ECS optimized).
+2. `SystemsManagerAccess` will be enabled by default.
+
+## Deprecation warnings
+
+* `operations/backup-dynamodb` will be removed in the next version (v9). Use `operations/backup-dynamodb-native` instead.
+
+# Migrate from v6 to v7
+
+[Learn more](./migrate-v7/)
+
+# Migrate from v5 to v6
+
+[Learn more](./migrate-v6/)
+
+# Migrate from v4 to v5
+
+[Learn more](./migrate-v5/)
+
+# Migrate from v3 to v4
+
+[Learn more](./migrate-v4/)
diff --git a/docs/operations.md b/docs/operations.md
@@ -41,7 +41,7 @@ I add links to AWS Management Console that are relevant to an incident. Contextu
 # Backup DynamoDB (Data Pipeline & EMR)
 This template describes a Data Pipeline to backup a single DynamoDB table. The Data Pipeline will spin up a EMR cluster to do the backup.
 
-> Deprecated in v7, will be removed in v8, use `operations/backup-dynamodb-native.yaml` instead!
+> Deprecated in v7, will be removed in v9, use `operations/backup-dynamodb-native.yaml` instead!
 
 ## Installation Guide
 1. This templates depends on our [`vpc-*azs.yaml`](./vpc/) template. [![Launch Stack](./img/launch-stack.png)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=vpc-2azs&templateURL=https://s3-eu-west-1.amazonaws.com/widdix-aws-cf-templates-releases-eu-west-1/__VERSION__/vpc/vpc-2azs.yaml)

diff --git a/ecs/README.md b/ecs/README.md
@@ -9,5 +9,5 @@ To update the region map execute the following lines in your terminal:
 
 ```
 $ regions=$(aws ec2 describe-regions --query "Regions[].RegionName" --output text)
-$ for region in $regions; do ami=$(aws --region $region ec2 describe-images --filters "Name=name,Values=amzn-ami-2018.03.g-amazon-ecs-optimized" --query "Images[0].ImageId" --output "text"); printf "'$region':\n  ECSAMI: '$ami'\n"; done
+$ for region in $regions; do ami=$(aws --region $region ec2 describe-images --filters "Name=name,Values=amzn2-ami-ecs-hvm-2.0.20181017-x86_64-ebs" --query "Images[0].ImageId" --output "text"); printf "'$region':\n  ECSAMI: '$ami'\n"; done
 ```
diff --git a/ecs/cluster.yaml b/ecs/cluster.yaml
@@ -29,7 +29,7 @@ Metadata:
       Parameters:
       - KeyName
       - IAMUserSSHAccess
-      # TODO - SystemsManagerAccess
+      - SystemsManagerAccess
     - Label:
         default: 'Load Balancer Parameters'
       Parameters:
@@ -79,6 +79,13 @@ Parameters:
     AllowedValues:
     - true
     - false
+  SystemsManagerAccess:
+    Description: 'Enable AWS Systems Manager agent and authorization.'
+    Type: String
+    Default: true
+    AllowedValues:
+    - true
+    - false
   SubnetsReach:
     Description: 'Should the cluster have direct access to the Internet or do you prefer private subnets with NAT?'
     Type: String
@@ -152,40 +159,41 @@ Parameters:
     MinValue: 2
     ConstraintDescription: 'Must be >= 2'
 Mappings:
-  RegionMap: # TODO update to Amazon Linux 2 (don't forget to adjust awslogs config as well)
+  RegionMap:
     'ap-south-1':
-      ECSAMI: 'ami-0590d0dd683026eab'
+      ECSAMI: 'ami-00afc9e848e32b1ca'
     'eu-west-3':
-      ECSAMI: 'ami-06b685336aa497c15'
+      ECSAMI: 'ami-0b0b5fca56d03d022'
     'eu-west-2':
-      ECSAMI: 'ami-0209769f0c963e791'
+      ECSAMI: 'ami-0ba06c0257c11d483'
     'eu-west-1':
-      ECSAMI: 'ami-05b65c0f6a75c1c64'
+      ECSAMI: 'ami-0651de2fa6ccf6d26'
     'ap-northeast-2':
-      ECSAMI: 'ami-0d947b1901b27a37c'
+      ECSAMI: 'ami-0fcea96f3b2c67274'
     'ap-northeast-1':
-      ECSAMI: 'ami-08681de00a0aae54f'
+      ECSAMI: 'ami-05a1f057a1a8e1a50'
     'sa-east-1':
-      ECSAMI: 'ami-01bca91ecf4c1f494'
+      ECSAMI: 'ami-0f2187a18380957e4'
     'ca-central-1':
-      ECSAMI: 'ami-00d1bdbd447b5933a'
+      ECSAMI: 'ami-0605f253b9e04698c'
     'ap-southeast-1':
-      ECSAMI: 'ami-0a3f70f0255af1d29'
+      ECSAMI: 'ami-0d4d4a42a45fb8e4a'
     'ap-southeast-2':
-      ECSAMI: 'ami-05b48eda7f92aadbe'
+      ECSAMI: 'ami-0231cd5b17565f0b1'
     'eu-central-1':
-      ECSAMI: 'ami-0bb804e8cd910a664'
+      ECSAMI: 'ami-07c3a868617f8acdb'
     'us-east-1':
-      ECSAMI: 'ami-0b9a214f40c38d5eb'
+      ECSAMI: 'ami-0a6be20ed8ce1f055'
     'us-east-2':
-      ECSAMI: 'ami-09a64272e7fe706b6'
+      ECSAMI: 'ami-075d44ed0d20df780'
     'us-west-1':
-      ECSAMI: 'ami-0e7dd5fe55b87a5fe'
+      ECSAMI: 'ami-0828cd2f20543a2b3'
     'us-west-2':
-      ECSAMI: 'ami-00430184c7bb49914'
+      ECSAMI: 'ami-04a4fb062c609f55b'
 Conditions:
   HasKeyName: !Not [!Equals [!Ref KeyName, '']]
   HasIAMUserSSHAccess: !Equals [!Ref IAMUserSSHAccess, 'true']
+  HasSystemsManagerAccess: !Equals [!Ref SystemsManagerAccess, 'true']
   HasSSHBastionSecurityGroup: !Not [!Equals [!Ref ParentSSHBastionStack, '']]
   HasNotSSHBastionSecurityGroup: !Equals [!Ref ParentSSHBastionStack, '']
   HasAuthProxySecurityGroup: !Not [!Equals [!Ref ParentAuthProxyStack, '']]
@@ -220,6 +228,7 @@ Resources:
             Service: 'ec2.amazonaws.com'
           Action: 'sts:AssumeRole'
       Path: '/'
+      ManagedPolicyArns: !If [HasSystemsManagerAccess, ['arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM'], []] # TODO get rid of managed policy
       Policies:
       - PolicyName: ecs
         PolicyDocument:
@@ -535,30 +544,28 @@ Resources:
               content: !Sub |
                 [general]
                 state_file = /var/lib/awslogs/agent-state
-                [/var/log/messages]
-                datetime_format = %b %d %H:%M:%S
-                file = /var/log/messages
-                log_stream_name = {instance_id}/var/log/messages
-                log_group_name = ${LogGroup}
-                [/var/log/secure]
-                datetime_format = %b %d %H:%M:%S
-                file = /var/log/secure
-                log_stream_name = {instance_id}/var/log/secure
+                [/var/log/amazon/ssm/amazon-ssm-agent.log]
+                datetime_format = %Y-%m-%d %H:%M:%S
+                file = /var/log/amazon/ssm/amazon-ssm-agent.log
+                log_stream_name = {instance_id}/var/log/amazon/ssm/amazon-ssm-agent.log
                 log_group_name = ${LogGroup}
-                [/var/log/cron]
-                datetime_format = %b %d %H:%M:%S
-                file = /var/log/cron
-                log_stream_name = {instance_id}/var/log/cron
+                [/var/log/amazon/ssm/errors.log]
+                datetime_format = %Y-%m-%d %H:%M:%S
+                file = /var/log/amazon/ssm/errors.log
+                log_stream_name = {instance_id}/var/log/amazon/ssm/errors.log
                 log_group_name = ${LogGroup}
-                [/var/log/cloud-init.log]
-                datetime_format = %b %d %H:%M:%S
-                file = /var/log/cloud-init.log
-                log_stream_name = {instance_id}/var/log/cloud-init.log
+                [/var/log/audit/audit.log]
+                file = /var/log/audit/audit.log
+                log_stream_name = {instance_id}/var/log/audit/audit.log
                 log_group_name = ${LogGroup}
-                [/var/log/cfn-init.log]
+                [/var/log/awslogs.log]
                 datetime_format = %Y-%m-%d %H:%M:%S
-                file = /var/log/cfn-init.log
-                log_stream_name = {instance_id}/var/log/cfn-init.log
+                file = /var/log/awslogs.log
+                log_stream_name = {instance_id}/var/log/awslogs.log
+                log_group_name = ${LogGroup}
+                [/var/log/boot.log]
+                file = /var/log/boot.log
+                log_stream_name = {instance_id}/var/log/boot.log
                 log_group_name = ${LogGroup}
                 [/var/log/cfn-hup.log]
                 datetime_format = %Y-%m-%d %H:%M:%S
@@ -570,21 +577,65 @@ Resources:
                 file = /var/log/cfn-init-cmd.log
                 log_stream_name = {instance_id}/var/log/cfn-init-cmd.log
                 log_group_name = ${LogGroup}
+                [/var/log/cfn-init.log]
+                datetime_format = %Y-%m-%d %H:%M:%S
+                file = /var/log/cfn-init.log
+                log_stream_name = {instance_id}/var/log/cfn-init.log
+                log_group_name = ${LogGroup}
+                [/var/log/cfn-wire.log]
+                datetime_format = %Y-%m-%d %H:%M:%S
+                file = /var/log/cfn-wire.log
+                log_stream_name = {instance_id}/var/log/cfn-wire.log
+                log_group_name = ${LogGroup}
                 [/var/log/cloud-init-output.log]
                 file = /var/log/cloud-init-output.log
                 log_stream_name = {instance_id}/var/log/cloud-init-output.log
                 log_group_name = ${LogGroup}
+                [/var/log/cloud-init.log]
+                datetime_format = %b %d %H:%M:%S
+                file = /var/log/cloud-init.log
+                log_stream_name = {instance_id}/var/log/cloud-init.log
+                log_group_name = ${LogGroup}
+                [/var/log/cron]
+                datetime_format = %b %d %H:%M:%S
+                file = /var/log/cron
+                log_stream_name = {instance_id}/var/log/cron
+                log_group_name = ${LogGroup}
                 [/var/log/dmesg]
                 file = /var/log/dmesg
                 log_stream_name = {instance_id}/var/log/dmesg
                 log_group_name = ${LogGroup}
+                [/var/log/grubby_prune_debug]
+                file = /var/log/grubby_prune_debug
+                log_stream_name = {instance_id}/var/log/grubby_prune_debug
+                log_group_name = ${LogGroup}
+                [/var/log/maillog]
+                datetime_format = %b %d %H:%M:%S
+                file = /var/log/maillog
+                log_stream_name = {instance_id}/var/log/maillog
+                log_group_name = ${LogGroup}
+                [/var/log/messages]
+                datetime_format = %b %d %H:%M:%S
+                file = /var/log/messages
+                log_stream_name = {instance_id}/var/log/messages
+                log_group_name = ${LogGroup}
+                [/var/log/secure]
+                datetime_format = %b %d %H:%M:%S
+                file = /var/log/secure
+                log_stream_name = {instance_id}/var/log/secure
+                log_group_name = ${LogGroup}
+                [/var/log/yum.log]
+                datetime_format = %b %d %H:%M:%S
+                file = /var/log/yum.log
+                log_stream_name = {instance_id}/var/log/yum.log
+                log_group_name = ${LogGroup}
               mode: '000644'
               owner: root
               group: root
             '/etc/awslogs/config/ecs.conf':
               content: !Sub |
                 [/var/log/ecs/ecs-init.log]
-                file = /var/log/ecs/ecs-init.log.*
+                file = /var/log/ecs/ecs-init.log
                 log_group_name = /var/log/ecs/ecs-init.log
                 log_stream_name = {instance_id}/var/log/ecs/ecs-init.log
                 datetime_format = %Y-%m-%dT%H:%M:%SZ
@@ -598,7 +649,7 @@ Resources:
               group: root
           services:
             sysvinit:
-              awslogs:
+              awslogsd:
                 enabled: true
                 ensureRunning: true
                 packages:
@@ -609,9 +660,6 @@ Resources:
                 - '/etc/awslogs/awscli.conf'
                 - '/etc/awslogs/config/ecs.conf'
         ssh-access:
-          packages:
-            yum:
-              'aws-cli': []
           files:
             '/opt/authorized_keys_command.sh':
               content: |
@@ -679,6 +727,7 @@ Resources:
           packages:
             yum:
               ruby: []
+              amazon-ssm-agent: []
             rubygems:
               'aws-sdk': ['3.0.1']
               daemons: ['1.2.4']
@@ -873,6 +922,12 @@ Resources:
                 files:
                 - '/etc/cfn/cfn-hup.conf'
                 - '/etc/cfn/hooks.d/cfn-auto-reloader.conf'
+              amazon-ssm-agent:
+                enabled: !If [HasSystemsManagerAccess, true, false]
+                ensureRunning: !If [HasSystemsManagerAccess, true, false]
+                packages:
+                  yum:
+                  - amazon-ssm-agent
               lifecycle-poller:
                 enabled: true
                 ensureRunning: true

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -14,7 +14,4 @@ pages:
 - 'VPC': 'vpc.md'
 - 'WordPress': 'wordpress.md'
 - 'widdix CLI': 'cli.md'
-- 'Migrate to v7': 'migrate-v7.md'
-- 'Migrate to v6': 'migrate-v6.md'
-- 'Migrate to v5': 'migrate-v5.md'
-- 'Migrate to v4': 'migrate-v4.md'
+- 'Migrate to v8': 'migrate-v8.md'
diff --git a/operations/backup-dynamodb.yaml b/operations/backup-dynamodb.yaml
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 AWSTemplateFormatVersion: '2010-09-09'
-Description: 'Operations: DynamoDB backup with Data Pipeline (Deprecated in v7, will be removed in v8, use operations/backup-dynamodb-native.yaml instead), a cloudonaut.io template'
+Description: 'Operations: DynamoDB backup with Data Pipeline (Deprecated in v7, will be removed in v9, use operations/backup-dynamodb-native.yaml instead), a cloudonaut.io template'
 Metadata:
   'AWS::CloudFormation::Interface':
     ParameterGroups:

diff --git a/test/src/test/java/de/widdix/awscftemplates/vpc/TestVPC2AZsLegacy.java b/test/src/test/java/de/widdix/awscftemplates/vpc/TestVPC2AZsLegacy.java
@@ -22,7 +22,6 @@ public void test() {
                         "vpc/vpc-2azs-legacy.yaml",
                         new Parameter().withParameterKey("AZA").withParameterValue(vpcOutputs.get("AZA")),
                         new Parameter().withParameterKey("AZB").withParameterValue(vpcOutputs.get("AZB")),
-                        new Parameter().withParameterKey("ClassB").withParameterValue(vpcOutputs.get("ClassB")),
                         new Parameter().withParameterKey("CidrBlock").withParameterValue(vpcOutputs.get("CidrBlock")),
                         new Parameter().withParameterKey("VPC").withParameterValue(vpcOutputs.get("VPC")),
                         new Parameter().withParameterKey("SubnetAPublic").withParameterValue(vpcOutputs.get("SubnetAPublic")),

diff --git a/test/src/test/java/de/widdix/awscftemplates/vpc/TestVPC3AZsLegacy.java b/test/src/test/java/de/widdix/awscftemplates/vpc/TestVPC3AZsLegacy.java
@@ -23,7 +23,6 @@ public void test() {
                         new Parameter().withParameterKey("AZA").withParameterValue(vpcOutputs.get("AZA")),
                         new Parameter().withParameterKey("AZB").withParameterValue(vpcOutputs.get("AZB")),
                         new Parameter().withParameterKey("AZC").withParameterValue(vpcOutputs.get("AZC")),
-                        new Parameter().withParameterKey("ClassB").withParameterValue(vpcOutputs.get("ClassB")),
                         new Parameter().withParameterKey("CidrBlock").withParameterValue(vpcOutputs.get("CidrBlock")),
                         new Parameter().withParameterKey("VPC").withParameterValue(vpcOutputs.get("VPC")),
                         new Parameter().withParameterKey("SubnetAPublic").withParameterValue(vpcOutputs.get("SubnetAPublic")),

diff --git a/test/src/test/java/de/widdix/awscftemplates/vpc/TestVPC4AZsLegacy.java b/test/src/test/java/de/widdix/awscftemplates/vpc/TestVPC4AZsLegacy.java
@@ -24,7 +24,6 @@ public void test() {
                         new Parameter().withParameterKey("AZB").withParameterValue(vpcOutputs.get("AZB")),
                         new Parameter().withParameterKey("AZC").withParameterValue(vpcOutputs.get("AZC")),
                         new Parameter().withParameterKey("AZD").withParameterValue(vpcOutputs.get("AZD")),
-                        new Parameter().withParameterKey("ClassB").withParameterValue(vpcOutputs.get("ClassB")),
                         new Parameter().withParameterKey("CidrBlock").withParameterValue(vpcOutputs.get("CidrBlock")),
                         new Parameter().withParameterKey("VPC").withParameterValue(vpcOutputs.get("VPC")),
                         new Parameter().withParameterKey("SubnetAPublic").withParameterValue(vpcOutputs.get("SubnetAPublic")),