Skip to content

aws-ecs: appends invalid ecs user data #32469

New issue
New issue
Closed
Closed
aws-ecs: appends invalid ecs user data#32469
Labels
@aws-cdk/aws-ecsRelated to Amazon Elastic ContainerbugThis issue is a bug.effort/mediumMedium work item – several days of effortp1
@cheruvian

Description

@cheruvian
cheruvian
opened on Dec 11, 2024

Describe the bug

ECS appends the following to the asg userdata

autoScalingGroup.addUserData('sudo service iptables save');

This command is invalid on the most recent versions of the ECS optimized AMI

+ sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP
+ sudo service iptables save
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.
+ echo ECS_AWSVPC_BLOCK_IMDS=true

Expanding to look at the whole code block the comments seem to be out of place

            // Deny containers access to instance metadata service
            // Source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html
            autoScalingGroup.addUserData('sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP');
            autoScalingGroup.addUserData('sudo service iptables save');
            // The following is only for AwsVpc networking mode, but doesn't hurt for the other modes.
            autoScalingGroup.addUserData('echo ECS_AWSVPC_BLOCK_IMDS=true >> /etc/ecs/ecs.config');

aws-cdk/packages/aws-cdk-lib/aws-ecs/lib/cluster.ts

Lines 567 to 569 in 59e96a3

// Source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html
autoScalingGroup.addUserData('sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP');
autoScalingGroup.addUserData('sudo service iptables save');

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Should have valid commands in userdata. Is a service restart not needed?

Current Behavior

Command fails and suggests using systemctl

Reproduction Steps

Deploy an ECS cluster backed by ASG

Possible Solution

At a minimum reorder the comments, not sure what command it is intending to run, perhaps sudo netfilter-persistent save which isn't installed on the latest EBS optimized GPU AMI

            // Source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html
			// ??????

            // The following is only for AwsVpc networking mode, but doesn't hurt for the other modes.
            autoScalingGroup.addUserData('sudo iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.169.254/32 --jump DROP');
            autoScalingGroup.addUserData('sudo service iptables restart');

            // Deny containers access to instance metadata service
            autoScalingGroup.addUserData('echo ECS_AWSVPC_BLOCK_IMDS=true >> /etc/ecs/ecs.config');

Additional Information/Context

No response

CDK CLI Version

2.134.0 (build 265d769)

Framework Version

No response

Node.js Version

v20.10.0

OS

OSX

Language

TypeScript

Language Version

No response

Other information

No response

Activity

cheruvian
added
bugThis issue is a bug.
needs-triageThis issue or PR still needs to be triaged.
on Dec 11, 2024
github-actions
added
@aws-cdk/aws-ecsRelated to Amazon Elastic Container
on Dec 11, 2024
ashishdhingra
self-assigned this
on Dec 11, 2024
ashishdhingra
added
p2
investigatingThis issue is being investigated and/or work is in progress to resolve the issue.
and removed
needs-triageThis issue or PR still needs to be triaged.
on Dec 11, 2024
ashishdhingra

ashishdhingra commented on Dec 11, 2024

@ashishdhingra
ashishdhingra
on Dec 11, 2024 · edited by ashishdhingra
Contributor

The MachineImageType enum supports 2 values as of now - AMAZON_LINUX_2 and BOTTLEROCKET. May be we also should update this enum to add AMAZON_LINUX_2023 and then move current default case here under MachineImageType.AMAZON_LINUX_2 switch case and add default case for AL 2023!

Using AWS CLI command aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id --region us-east-1 (reference Retrieving Amazon ECS-optimized Linux AMI metadata), we could get the latest AMI for AL 2023 returns the following output:

{
    "Parameters": [
        {
            "Name": "/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id",
            "Type": "String",
            "Value": "ami-0a357ea20d7b79c2c",
            "Version": 61,
            "LastModifiedDate": "2024-11-18T08:07:53.585000-08:00",
            "ARN": "arn:aws:ssm:us-east-1::parameter/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id",
            "DataType": "text"
        }
    ],
    "InvalidParameters": []
}

Below CDK code could be used to reproduce the issue (assuming we could SSH to EC2 instance launched by ECS task):

import * as cdk from 'aws-cdk-lib';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as autoscaling from 'aws-cdk-lib/aws-autoscaling';

export class CdktestStackNew extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'EcsVpc');
    /*
    const ecsAmi = ec2.MachineImage.lookup({
      name: '/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id'
    });
    */

    const asg = new autoscaling.AutoScalingGroup(this, 'EcsAsg', { 
      vpc: vpc,
      instanceType: new ec2.InstanceType('t2.xlarge'),
      machineImage: ecs.EcsOptimizedImage.amazonLinux2023(),
      desiredCapacity: 3
    }); 

    const capacityProvider = new ecs.AsgCapacityProvider(this, 'EcsCapacityProvider', {
      autoScalingGroup: asg
    });

    const cluster = new ecs.Cluster(this, 'MyEcsCluster', {
      vpc: vpc
    });
    cluster.addAsgCapacityProvider(capacityProvider);
  }
}

@cheruvian Please validate the finding above. Also share if you were able to gather logs, perhaps from /var/log/cloud-init.log and /var/log/cloud-init-output.log, and share in this issue.

ashishdhingra
added
p1
effort/mediumMedium work item – several days of effort
and removed
p2
on Dec 11, 2024
ashishdhingra
removed their assignment
on Dec 11, 2024
ashishdhingra
removed
investigatingThis issue is being investigated and/or work is in progress to resolve the issue.
on Dec 11, 2024
phuhung273
mentioned this on Dec 12, 2024

7 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    @aws-cdk/aws-ecsRelated to Amazon Elastic ContainerbugThis issue is a bug.effort/mediumMedium work item – several days of effortp1

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      aws-ecs: appends invalid ecs user data · Issue #32469 · aws/aws-cdk