Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PodSecurityPolicies for addons #55509

Merged
merged 4 commits into from
Nov 14, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: gce:podsecuritypolicy:calico
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/cluster-service: "true"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gce:podsecuritypolicy:privileged
subjects:
- kind: ServiceAccount
name: calico
namespace: kube-system
21 changes: 0 additions & 21 deletions cluster/addons/cluster-monitoring/google/heapster-controller.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,26 +58,12 @@ spec:
- /heapster
- --source=kubernetes.summary_api:''
- --sink=gcm
volumeMounts:
- name: ssl-certs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does heapster not need to talk to googleapis.com?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this change heapster uses the SSL certs that are built into the container, rather than those on the host. My understanding is that the certificates in the container are sufficient that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(To clarify - usually this practice of mounting the host certificates is only needed for FROM scratch containers. However, heapster copies in the host certs when it's built, so this isn't necessary).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how long are those valid for?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ openssl x509 -in heapster/92474ba266e5a063e88a4cc5f136d5bac305cb75538b4086a73c1ac27d834ce6/etc/ssl/certs/ca-certificates.crt  -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6828503384748696800 (0x5ec3b7a6437fa4e0)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=ACCVRAIZ1, OU=PKIACCV, O=ACCV, C=ES
        Validity
            Not Before: May  5 09:37:37 2011 GMT
            Not After : Dec 31 09:37:37 2030 GMT
        Subject: CN=ACCVRAIZ1, OU=PKIACCV, O=ACCV, C=ES
...

Looks like that shouldn't be a problem, as long as they aren't revoked.

That said, this is a a common requirement for containers. Maybe we need to build a solution that doesn't require a HostPath volume...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, ya. I agree.

mountPath: /etc/ssl/certs
readOnly: true
- name: usr-ca-certs
mountPath: /usr/share/ca-certificates
readOnly: true
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.0
name: eventer
command:
- /eventer
- --source=kubernetes:''
- --sink=gcl
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs
readOnly: true
- name: usr-ca-certs
mountPath: /usr/share/ca-certificates
readOnly: true
- image: gcr.io/google_containers/addon-resizer:1.7
name: heapster-nanny
resources:
Expand Down Expand Up @@ -136,13 +122,6 @@ spec:
- --container=eventer
- --poll-period=300000
- --estimator=exponential
volumes:
- name: ssl-certs
hostPath:
path: "/etc/ssl/certs"
- name: usr-ca-certs
hostPath:
path: "/usr/share/ca-certificates"
serviceAccountName: heapster
tolerations:
- key: "CriticalAddonsOnly"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,26 +59,12 @@ spec:
- --source=kubernetes.summary_api:''
- --sink=influxdb:http://monitoring-influxdb:8086
- --sink=gcm:?metrics=autoscaling
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs
readOnly: true
- name: usr-ca-certs
mountPath: /usr/share/ca-certificates
readOnly: true
- image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.0
name: eventer
command:
- /eventer
- --source=kubernetes:''
- --sink=gcl
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs
readOnly: true
- name: usr-ca-certs
mountPath: /usr/share/ca-certificates
readOnly: true
- image: gcr.io/google_containers/addon-resizer:1.7
name: heapster-nanny
resources:
Expand Down Expand Up @@ -137,13 +123,6 @@ spec:
- --container=eventer
- --poll-period=300000
- --estimator=exponential
volumes:
- name: ssl-certs
hostPath:
path: "/etc/ssl/certs"
- name: usr-ca-certs
hostPath:
path: "/usr/share/ca-certificates"
serviceAccountName: heapster
tolerations:
- key: "CriticalAddonsOnly"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,6 @@ spec:
- /heapster
- --source=kubernetes.summary_api:''
- --sink=stackdriver:?cluster_name={{ cluster_name }}&min_interval_sec=100&batch_export_timeout_sec=110
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs
readOnly: true
- name: usr-ca-certs
mountPath: /usr/share/ca-certificates
readOnly: true
# BEGIN_PROMETHEUS_TO_SD
- name: prom-to-sd
image: gcr.io/google-containers/prometheus-to-sd:v0.2.2
Expand Down Expand Up @@ -112,13 +105,6 @@ spec:
- --container=heapster
- --poll-period=300000
- --estimator=exponential
volumes:
- name: ssl-certs
hostPath:
path: "/etc/ssl/certs"
- name: usr-ca-certs
hostPath:
path: "/usr/share/ca-certificates"
serviceAccountName: heapster
tolerations:
- key: "CriticalAddonsOnly"
Expand Down
11 changes: 11 additions & 0 deletions cluster/addons/etcd-empty-dir-cleanup/etcd-empty-dir-cleanup.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,14 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: etcd-empty-dir-cleanup
namespace: kube-system
labels:
k8s-app: etcd-empty-dir-cleanup
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: Pod
metadata:
name: etcd-empty-dir-cleanup
Expand All @@ -8,6 +18,7 @@ metadata:
labels:
k8s-app: etcd-empty-dir-cleanup
spec:
serviceAccountName: etcd-empty-dir-cleanup
hostNetwork: true
dnsPolicy: Default
containers:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: gce:podsecuritypolicy:etcd-empty-dir-cleanup
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/cluster-service: "true"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: gce:podsecuritypolicy:etcd-empty-dir-cleanup
subjects:
- kind: ServiceAccount
name: etcd-empty-dir-cleanup
namespace: kube-system
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gce:podsecuritypolicy:etcd-empty-dir-cleanup
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
- extensions
resourceNames:
- gce.etcd-empty-dir-cleanup
resources:
- podsecuritypolicies
verbs:
- use
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: gce.etcd-empty-dir-cleanup
annotations:
kubernetes.io/description: 'Policy used by the etcd-empty-dir-cleanup addon.'
# TODO: etcd-empty-dir-cleanup should run with the default seccomp profile
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
# 'runtime/default' is already the default, but must be filled in on the
# pod to pass admission.
apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
labels:
kubernetes.io/cluster-service: 'true'
addonmanager.kubernetes.io/mode: Reconcile
spec:
privileged: false
volumes:
- 'secret'
hostNetwork: true
hostIPC: false
hostPID: false
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
readOnlyRootFilesystem: false
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: gce:podsecuritypolicy:elasticsearch-logging
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/cluster-service: "true"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gce:podsecuritypolicy:privileged
subjects:
- kind: ServiceAccount
name: elasticsearch-logging
namespace: kube-system
11 changes: 11 additions & 0 deletions cluster/addons/ip-masq-agent/ip-masq-agent.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: ip-masq-agent
namespace: kube-system
labels:
k8s-app: ip-masq-agent
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
# https://github.com/kubernetes-incubator/ip-masq-agent/blob/v2.0.0/README.md
apiVersion: extensions/v1beta1
kind: DaemonSet
Expand All @@ -14,6 +24,7 @@ spec:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: ip-masq-agent
hostNetwork: true
containers:
- name: ip-masq-agent
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: gce:podsecuritypolicy:ip-masq-agent
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/cluster-service: "true"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: gce:podsecuritypolicy:privileged
subjects:
- kind: ServiceAccount
name: ip-masq-agent
namespace: kube-system
31 changes: 26 additions & 5 deletions cluster/gce/gci/configure-helper.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1705,14 +1705,35 @@ function start-cluster-autoscaler {
fi
}

# A helper function for copying addon manifests and set dir/files
# permissions.
# A helper function for setting up addon manifests.
#
# $1: addon category under /etc/kubernetes
# $2: manifest source dir
# $3: (optional) auxilary manifest source dir
function setup-addon-manifests {
local -r src_dir="${KUBE_HOME}/kube-manifests/kubernetes/gci-trusty/$2"
local -r src_dir="${KUBE_HOME}/kube-manifests/kubernetes/gci-trusty"
local -r dst_dir="/etc/kubernetes/$1/$2"

copy-manifests "${src_dir}/$2" "${dst_dir}"

# If the PodSecurityPolicy admission controller is enabled,
# set up the corresponding addon policies.
if [[ "${ENABLE_POD_SECURITY_POLICY:-}" == "true" ]]; then
local -r psp_dir="${src_dir}/${3:-$2}/podsecuritypolicies"
if [[ -d "${psp_dir}" ]]; then
copy-manifests "${psp_dir}" "${dst_dir}"
fi
fi
}

# A helper function for copying manifests and setting dir/files
# permissions.
#
# $1: absolute source dir
# $2: absolute destination dir
function copy-manifests {
local -r src_dir="$1"
local -r dst_dir="$2"
if [[ ! -d "${dst_dir}" ]]; then
mkdir -p "${dst_dir}"
fi
Expand Down Expand Up @@ -1783,7 +1804,7 @@ function start-kube-addons {
fi

if [[ "${ENABLE_POD_SECURITY_POLICY:-}" == "true" ]]; then
setup-addon-manifests "addons" "podsecuritypolicies"
setup-addon-manifests "addons" "podsecuritypolicies"
fi

# Set up manifests of other addons.
Expand Down Expand Up @@ -1892,7 +1913,7 @@ EOF
fi
if [[ "${ENABLE_NODE_PROBLEM_DETECTOR:-}" == "standalone" ]]; then
# Setup role binding for standalone node problem detector.
setup-addon-manifests "addons" "node-problem-detector/standalone"
setup-addon-manifests "addons" "node-problem-detector/standalone" "node-problem-detector"
fi
if echo "${ADMISSION_CONTROL:-}" | grep -q "LimitRanger"; then
setup-addon-manifests "admission-controls" "limit-range"
Expand Down