kubeadm doesn't support ABAC correctly #60844

msnelling · 2018-03-06T12:48:47Z

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
I attempted to create a cluster using the kubeadm tool with ABAC authorisation enabled using the below configuration files.

~/config.yaml

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
networking:
  podSubnet: 10.244.0.0/16
authorizationModes:
- Node
- RBAC
- ABAC
apiServerExtraArgs:
  basic-auth-file: /etc/kubernetes/basic_auth.csv
  authorization-policy-file: /etc/kubernetes/abac_policy.json

~/basic_auth.csv

admin,admin,1,"system:masters"

~/abac_policy.json

{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"*", "nonResourcePath": "*", "readonly": true}}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"admin", "namespace": "*", "resource": "*", "apiGroup": "*"}}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"group":"system:serviceaccounts", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}}

The kubeadm command hangs with the following output...

[init] Using Kubernetes version: v1.9.3
[init] Using Authorization modes: [Node RBAC ABAC]
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [####### #######]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.

It looks like the kube-apiserver container is constantly restarted.

The log output from the kube-apiserver container gives

I0306 12:39:33.179719       1 server.go:121] Version: v1.9.3
invalid authentication config: open /etc/kubernetes/basic_auth.csv: no such file or directory

What you expected to happen:
I expected the kube-apiserver to start.

How to reproduce it (as minimally and precisely as possible):

mkdir -p /etc/kubernetes
cp ~/basic_auth.csv /etc/kubernetes
cp ~/abac_policy.json /etc/kubernetes
kubeadm init --config ~/config.yaml

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): v1.9.3
Cloud provider or hardware configuration: Bare metal
OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus)
Kernel (e.g. uname -a): Linux k8s-master 4.4.0-116-generic PreStart and PostStop event hooks #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Install tools:
Others:

The text was updated successfully, but these errors were encountered:

msnelling · 2018-03-06T15:07:18Z

It seems that maybe the /etc/kubernetes/basic_auth.csv and /etc/kubernetes/abac_policy.json files are not mounted inside the kube-apiserver container.
I tried using the following extra volumes stanza but I couldn't get it to work with files, it only seems to work with directories.

apiServerExtraVolumes:
- name: <value|string>
  hostPath: <value|string>
  mountPath: <value|string>

If I try mounting a directory containing the two files and specify the new locations in the config it still looks for them in the default location! See the generated /etc/kubernetes/manifests/kube-apiserver.yaml below, it has two entries for the --authorization-policy-file option, the second seemingly overrides the first.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --authorization-policy-file=/etc/kubernetes/users/abac_policy.json
    - --basic-auth-file=/etc/kubernetes/users/basic_auth.csv
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota
    - --enable-bootstrap-token-auth=true
    - --allow-privileged=true
    - --requestheader-group-headers=X-Remote-Group
    - --advertise-address=#######
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --insecure-port=0
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --secure-port=6443
    - --requestheader-username-headers=X-Remote-User
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --requestheader-allowed-names=front-proxy-client
    - --service-cluster-ip-range=10.96.0.0/12
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --authorization-policy-file=/etc/kubernetes/abac_policy.json
    - --authorization-mode=Node,RBAC,ABAC
    - --etcd-servers=http://127.0.0.1:2379
    image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.3
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: #######
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/kubernetes/users
      name: users
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/kubernetes/users
      type: DirectoryOrCreate
    name: users
status: {}

timothysc · 2018-04-13T16:59:55Z

@msnelling - We generally discourage ABAC enablement, but it does look like an over-ride issue.

Where possible please open issues against the https://github.com/kubernetes/kubeadm repo.

timothysc self-assigned this Apr 13, 2018

timothysc closed this as completed Apr 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm doesn't support ABAC correctly #60844

kubeadm doesn't support ABAC correctly #60844

msnelling commented Mar 6, 2018

msnelling commented Mar 6, 2018

timothysc commented Apr 13, 2018

kubeadm doesn't support ABAC correctly #60844

kubeadm doesn't support ABAC correctly #60844

Comments

msnelling commented Mar 6, 2018

msnelling commented Mar 6, 2018

timothysc commented Apr 13, 2018