Document how RBAC interacts with kube-system components #29177
Closed
Description
In getting my cluster set up with RBAC, I discovered that Kubernetes system components need to be explicitly allowed to access the API just like any other client. I had to add a ClusterRole and ClusterRoleBinding for the kubelet (i.e. the common name in the k8s nodes' client certificate.) Without them, the nodes could not register themselves and begin handling work.
It's also not clear how RBAC affects the default service accounts. I'm trying to deploy kube-dns with this manifest:
---
apiVersion: "v1"
kind: "Namespace"
metadata:
name: "kube-system"
---
apiVersion: "v1"
kind: "Service"
metadata:
name: "kube-dns"
namespace: "kube-system"
labels:
k8s-app: "kube-dns"
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeDNS"
spec:
selector:
k8s-app: "kube-dns"
clusterIP: "10.3.0.10"
ports:
- name: "dns"
port: 53
protocol: "UDP"
- name: "dns-tcp"
port: 53
protocol: "TCP"
---
apiVersion: "extensions/v1beta1"
kind: "Deployment"
metadata:
name: "kube-dns"
namespace: "kube-system"
labels:
k8s-app: "kube-dns"
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
matchLabels:
k8s-app: "kube-dns"
template:
metadata:
labels:
k8s-app: "kube-dns"
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: "kubedns"
image: "gcr.io/google_containers/kubedns-amd64:1.5"
resources:
limits:
cpu: "100m"
memory: "200Mi"
requests:
cpu: "100m"
memory: "100Mi"
livenessProbe:
httpGet:
path: "/healthz"
port: 8080
scheme: "HTTP"
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: "/readiness"
port: 8081
scheme: "HTTP"
initialDelaySeconds: 30
timeoutSeconds: 5
args:
- "--domain=cluster.local."
- "--dns-port=10053"
ports:
- containerPort: 10053
name: "dns-local"
protocol: "UDP"
- containerPort: 10053
name: "dns-tcp-local"
protocol: "TCP"
- name: "dnsmasq"
image: "gcr.io/google_containers/kube-dnsmasq-amd64:1.3"
args:
- "--cache-size=1000"
- "--no-resolv"
- "--server=127.0.0.1#10053"
ports:
- containerPort: 53
name: "dns"
protocol: "UDP"
- containerPort: 53
name: "dns-tcp"
protocol: "TCP"
- name: "healthz"
image: "gcr.io/google_containers/exechealthz-amd64:1.0"
resources:
limits:
cpu: "10m"
memory: "20Mi"
requests:
cpu: "10m"
memory: "20Mi"
args:
- "-cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null"
- "-port=8080"
- "-quiet"
ports:
- containerPort: 8080
protocol: "TCP"
dnsPolicy: "Default"and the pod fails to start, showing this in the kube-dns container's logs:
I0719 08:34:53.090618 1 server.go:91] Using https://10.3.0.1:443 for kubernetes master
I0719 08:34:53.103017 1 server.go:92] Using kubernetes API <nil>
I0719 08:34:53.104573 1 server.go:132] Starting SkyDNS server. Listening on port:10053
I0719 08:34:53.104682 1 server.go:139] skydns: metrics enabled on :/metrics
I0719 08:34:53.104708 1 dns.go:166] Waiting for service: default/kubernetes
I0719 08:34:53.105794 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0719 08:34:53.105830 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
E0719 08:34:53.441778 1 reflector.go:216] pkg/dns/dns.go:155: Failed to list *api.Service: the server has asked for the client to provide credentials (get services)
E0719 08:34:53.442089 1 reflector.go:216] pkg/dns/dns.go:154: Failed to list *api.Endpoints: the server has asked for the client to provide credentials (get endpoints)
I0719 08:34:53.443210 1 dns.go:172] Ignoring error while waiting for service default/kubernetes: the server has asked for the client to provide credentials (get services kubernetes). Sleeping 1s before retrying.
I0719 08:34:53.537722 1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0719 08:34:53.537854 1 dns.go:539] records:[], retval:[], path:[local cluster svc default kubernetes]
I0719 08:34:53.540060 1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0719 08:34:53.540077 1 dns.go:539] records:[], retval:[], path:[local cluster svc default kubernetes]
E0719 08:34:54.443840 1 reflector.go:216] pkg/dns/dns.go:155: Failed to list *api.Service: the server has asked for the client to provide credentials (get services)
I0719 08:34:54.536969 1 dns.go:172] Ignoring error while waiting for service default/kubernetes: the server has asked for the client to provide credentials (get services kubernetes). Sleeping 1s before retrying.
E0719 08:34:54.537063 1 reflector.go:216] pkg/dns/dns.go:154: Failed to list *api.Endpoints: the server has asked for the client to provide credentials (get endpoints)
with that error recurring infinitely.
I added this subject to my full access ClusterRoleBinding:
kind: "ServiceAccount"
name: "default"
namespace: "*"but that didn't fix it.
Documentation for how to bootstrap RBAC so that all Kubernetes's own components work should be added. If someone can add some details here, I can make a PR to the docs website myself.