Skip to content
This repository has been archived by the owner on Sep 4, 2021. It is now read-only.

Enable Kubernetes native support for AWS ECR container registries when using kube-aws #518

Closed
@whereisaaron

Description

TL;DR ECR cross-domain support is coming when k8s 1.3.0 releases and whichever version of kube-aws supports 1.3.0, provided by kube-aws patch #384 and k8s patch #24369. In the meantime you can use imagePullSecrets with a scheduled script like this one:
https://github.com/whereisaaron/kubernetes-aws-scripts/blob/master/create-ecr-imagepullsecret.sh

Kubernetes provides native support for AWS ECR container registries, including cross-region ECR support. The particular ECR features are that kubelets automatically obtains and regularly updates ECR login credentials (AWS ECR login credentials only last 12 hours), and uses those credentials when you specify an ECR repository for your image. Relevant Kurbernetes references below.

http://kubernetes.io/docs/user-guide/images/#using-aws-ec2-container-registry
kubernetes/kubernetes#19447
kubernetes/kubernetes#23298

However, this excellent feature is only activated if the kublet is started with the '--cloud-provider=aws' option and it appears the controller nodes generated by kube-aws do not have that option enabled (using kube-aws 0.7.0), CORRECTION: worker nodes do have this option.

/etc/systemd/system/kubelet.service

[Service]
Environment=KUBELET_VERSION=v1.2.3_coreos.0
Environment=KUBELET_ACI=quay.io/coreos/hyperkube
Environment="RKT_OPTS=--volume dns,kind=host,source=/etc/resolv.conf --mount volume=dns,target=/etc/resolv.conf"
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--api-servers=http://localhost:8080 \
--network-plugin-dir=/etc/kubernetes/cni/net.d \
--network-plugin= \
--register-schedulable=false \
--allow-privileged=true \
--config=/etc/kubernetes/manifests \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local
Restart=always
RestartSec=10

The second requirement for this feature to work is the node AWS IAM roles also need ECR ReadOnly permissions added. The IAM 'AmazonEC2ContainerRegistryReadOnly' managed policy is exactly what is needed. Or the following actions could be added to the kube-aws generated role definition.

arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "ecr:GetAuthorizationToken",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:GetRepositoryPolicy",
            "ecr:DescribeRepositories",
            "ecr:ListImages",
            "ecr:BatchGetImage"
        ],
        "Resource": "*"
    }]
}

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions