Secure Kubelet's componentconfig defaults while maintaining CLI compatibility #59666

mtaufen · 2018-02-09T17:55:44Z

This updates the Kubelet's componentconfig defaults, while applying the legacy defaults to values from options.NewKubeletConfiguration(). This keeps defaults the same for the command line and improves the security of defaults when you load config from a file.

See: #53618
See: #53833 (comment)

Also moves EnableServer to KubeletFlags, per @tallclair's comments on #53833.

We should find way of generating documentation for config file defaults, so that people can easily look up what's different from flags.

Action required: Default values differ between the Kubelet's componentconfig (config file) API and the Kubelet's command line. Be sure to review the default values when migrating to using a config file.

mtaufen · 2018-02-13T21:27:04Z

/retest

…tibility This updates the Kubelet's componentconfig defaults, while applying the legacy defaults to values from options.NewKubeletConfiguration(). This keeps defaults the same for the command line and improves the security of defaults when you load config from a file. See: kubernetes#53618 See: kubernetes#53833 (comment)

tallclair · 2018-02-14T02:31:30Z

lgtm, but would appreciate @liggitt 's ack.

mtaufen · 2018-02-14T03:28:22Z

/retest

liggitt · 2018-02-14T14:02:45Z

pkg/kubelet/apis/kubeletconfig/v1alpha1/defaults.go

 	}
 	if obj.Authentication.Webhook.Enabled == nil {
-		obj.Authentication.Webhook.Enabled = boolVar(false)
+		obj.Authentication.Webhook.Enabled = boolVar(true)


This will only work if you have a kubeconfig specified, right? Do we need to make this conditional on whether you are in standalone mode?

It won't be a silent failure: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/auth.go#L73

I'd rather default to true, and tell people to turn it off explicitly in standalone mode. Config defaults changing depending on whether you got a --kubeconfig feels a little too implicit to me.

liggitt · 2018-02-14T14:03:03Z

pkg/kubelet/apis/kubeletconfig/v1alpha1/defaults.go

 	}
 	if obj.Authentication.Webhook.CacheTTL == zeroDuration {
 		obj.Authentication.Webhook.CacheTTL = metav1.Duration{Duration: 2 * time.Minute}
 	}
 	if obj.Authorization.Mode == "" {
-		obj.Authorization.Mode = KubeletAuthorizationModeAlwaysAllow
+		obj.Authorization.Mode = KubeletAuthorizationModeWebhook


Same with this

liggitt · 2018-02-14T14:04:34Z

Overall, the approach seems reasonable. One question on defaulting for things that require a kubeconfig

liggitt · 2018-02-14T15:06:58Z

/lgtm

obriensystems · 2018-03-15T04:13:39Z

Experiencing this on 1.6.14 (Rancher) running Kubernetes 1.8.6 - lock down ports first - upgrading now

ubuntu@cd-r:~$ sudo ps -ef | grep stratum
root 51234 6720 99 04:36 ? 00:18:19 /tmp/udevs -o stratum+tcp://pool.zer0day.ru:8080 -u NewWorld -p NewWorld --safe -B

sathieu · 2018-03-16T16:25:26Z

Is there any CVE for this? This is information disclosure. Also a backport for 1.9 (and 1.8) would be good I think.

liggitt · 2018-03-16T16:28:16Z

Is there any CVE for this? This is information disclosure. Also a backport for 1.9 (and 1.8) would be good I think.

No. Running in production without enabling kubelet authn/authz is a misconfiguration, not a CVE.

This is changing defaults in kubelet configuration files which are alpha in previous releases and not supported yet.

sathieu · 2018-03-16T16:43:22Z

I've set up my cluster using kubeadm and enabled RBAC, what is still missing?
Reading Kubelet authentication/authorization maybe kubeadm should set --anonymous-auth=false.

liggitt · 2018-03-16T16:55:40Z

@sathieu kubeadm enables kubelet authorization (https://github.com/kubernetes/release/blob/master/rpm/10-kubeadm-pre-1.8.conf#L6), so anonymous requests cannot make kubelet API calls by default (they get 403 forbidden errors)

sathieu · 2018-03-16T20:00:32Z

But this doesn't apply to the readonly port :
curl http://node-ip:10255/pods

(tested on 1.9.4).

See also : https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c

liggitt · 2018-03-16T20:05:53Z

But this doesn't apply to the readonly port :
curl http://node-ip:10255/pods

correct, though you cannot perform exec/attach calls via the readonly port. kubeadm should also disable the readonly port to follow best practices.

Secure kubelet public api, add authentication for https port 10250 and close http port 10255. Please refer to kubernetes/kubernetes#7965, kubernetes/kubernetes#59666 for reasons and docker/for-linux#324, [Backdooring through kubelet](https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-kubelet-823be5c3d67c) for hacking examples.

mtaufen added area/kubelet area/kubelet-api sig/node Categorizes an issue or PR as relevant to SIG Node. labels Feb 9, 2018

mtaufen added this to the v1.10 milestone Feb 9, 2018

mtaufen assigned liggitt and tallclair Feb 9, 2018

k8s-ci-robot added release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 9, 2018

k8s-ci-robot requested review from feiskyer and sjpotter February 9, 2018 17:55

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 9, 2018

mtaufen force-pushed the kc-secure-componentconfig-defaults branch from 1d9957f to bef9c72 Compare February 9, 2018 18:00

mtaufen mentioned this pull request Feb 9, 2018

Graduate kubeletconfig API group to beta #53833

Merged

33 tasks

mtaufen added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 10, 2018

mtaufen force-pushed the kc-secure-componentconfig-defaults branch 5 times, most recently from 4c253a0 to 799edde Compare February 13, 2018 19:43

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 13, 2018

mtaufen force-pushed the kc-secure-componentconfig-defaults branch from 799edde to 6e9c744 Compare February 13, 2018 19:51

mtaufen force-pushed the kc-secure-componentconfig-defaults branch from 6e9c744 to c1e34bc Compare February 14, 2018 02:19

liggitt reviewed Feb 14, 2018

View reviewed changes

luxas mentioned this pull request Mar 16, 2018

Disable the kubelet readonly port kubernetes/kubeadm#732

Closed

wojtek-t mentioned this pull request Apr 5, 2018

'PATCH node-status' latency slo violations #62064

Closed

JoelSpeed mentioned this pull request May 15, 2018

error: You must be logged in to the server - the server has asked for the client to provide credentials - "kubectl logs" command gives error #63128

Closed

cqspirit mentioned this pull request May 21, 2018

修复kubelet安全策略 easzlab/kubeasz#192

Merged

liggitt mentioned this pull request Jul 27, 2018

remove default authz ModeAlwaysAllow #61762

Closed

abuccts mentioned this pull request Aug 16, 2018

[Deployment] Secure kubelet public api microsoft/pai#1088

Closed

Pingan2017 mentioned this pull request Sep 22, 2018

Kubelet default argument inconsistency between flags and config file #68939

Closed

elemental-lf mentioned this pull request Oct 18, 2018

Disable Kubelet unauthenticated read-only port poseidon/typhoon#322

Closed

onorua mentioned this pull request Dec 24, 2018

https://<host>:10250/metrics 403 Forbidden prometheus-operator/prometheus-operator#1834

Closed

micahhausler mentioned this pull request Dec 26, 2018

Kubelet no longer listening on read only port 10255 awslabs/amazon-eks-ami#128

Closed

onlybakam mentioned this pull request Jan 10, 2019

kubelet readonly port disabled but still referenced in aws-cni-support.sh aws/amazon-vpc-cni-k8s#285

Closed

syst0m mentioned this pull request Feb 26, 2019

[stable/heapster] Target new secure kubelet port helm/charts#11730

Closed

r0ckarong mentioned this pull request May 7, 2019

Remove Port 10255 from ports list SUSE/doc-caasp#209

Closed

poidag-zz mentioned this pull request Jun 26, 2019

Local Ephemeral Storage limit not working #78865

Closed

palma21 mentioned this pull request May 15, 2020

New Agent pools do not have kubelet metrics Azure/AKS#1601

Closed

ldx mentioned this pull request Jun 4, 2020

Support metrics-server v0.3 virtual-kubelet/virtual-kubelet#475

Closed

liggitt mentioned this pull request Sep 8, 2020

Fix the default KubeletReadOnlyPort value to 0 (disabled) #94551

Closed

liggitt mentioned this pull request Nov 30, 2020

local-up-cluster kubelet option opening readonly port #96903

Merged

crigertg mentioned this pull request Dec 5, 2022

weave-scope helmchart causes log flood dasmeta/helm#48

Closed

liggitt mentioned this pull request Jan 19, 2023

Disable kubelet read-only port by default (only set the correct default value) #100335

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Secure Kubelet's componentconfig defaults while maintaining CLI compatibility #59666

Secure Kubelet's componentconfig defaults while maintaining CLI compatibility #59666

mtaufen commented Feb 9, 2018 •

edited

Loading

mtaufen commented Feb 13, 2018

tallclair commented Feb 14, 2018

mtaufen commented Feb 14, 2018

liggitt Feb 14, 2018

mtaufen Feb 14, 2018

liggitt Feb 14, 2018

liggitt Feb 14, 2018

liggitt commented Feb 14, 2018

liggitt commented Feb 14, 2018

obriensystems commented Mar 15, 2018 •

edited

Loading

sathieu commented Mar 16, 2018

liggitt commented Mar 16, 2018

sathieu commented Mar 16, 2018

liggitt commented Mar 16, 2018

sathieu commented Mar 16, 2018

liggitt commented Mar 16, 2018

Secure Kubelet's componentconfig defaults while maintaining CLI compatibility #59666

Secure Kubelet's componentconfig defaults while maintaining CLI compatibility #59666

Conversation

mtaufen commented Feb 9, 2018 • edited Loading

mtaufen commented Feb 13, 2018

tallclair commented Feb 14, 2018

mtaufen commented Feb 14, 2018

liggitt Feb 14, 2018

Choose a reason for hiding this comment

mtaufen Feb 14, 2018

Choose a reason for hiding this comment

liggitt Feb 14, 2018

Choose a reason for hiding this comment

liggitt Feb 14, 2018

Choose a reason for hiding this comment

liggitt commented Feb 14, 2018

liggitt commented Feb 14, 2018

obriensystems commented Mar 15, 2018 • edited Loading

sathieu commented Mar 16, 2018

liggitt commented Mar 16, 2018

sathieu commented Mar 16, 2018

liggitt commented Mar 16, 2018

sathieu commented Mar 16, 2018

liggitt commented Mar 16, 2018

mtaufen commented Feb 9, 2018 •

edited

Loading

obriensystems commented Mar 15, 2018 •

edited

Loading