Set pids limit at pod level #57973

dims · 2018-01-08T18:07:29Z

What this PR does / why we need it:

Add a new Alpha Feature to set a maximum number of pids per Pod.
This is to allow the use case where cluster administrators wish
to limit the pids consumed per pod (example when running a CI system).

By default, we do not set any maximum limit, If an administrator wants
to enable this, they should enable SupportPodPidsLimit=true in the
--feature-gates= parameter to kubelet and specify the limit using the
--pod-max-pids parameter.

The limit set is the total count of all processes running in all
containers in the pod.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #43783

Special notes for your reviewer:

Release note:

New alpha feature to limit the number of processes running in a pod. Cluster administrators will be able to place limits by using the new kubelet command line parameter --pod-max-pids. Note that since this is a alpha feature they will need to enable the "SupportPodPidsLimit" feature.

dims · 2018-01-08T18:09:33Z

/assign @tallclair
/assign @thockin
/assign @dchen1107

cc @BenTheElder

dims · 2018-01-08T19:09:37Z

/test pull-kubernetes-unit

dims · 2018-01-08T19:12:56Z

/test pull-kubernetes-cross

tallclair

Thanks, looks much cleaner.

tallclair · 2018-01-08T19:25:35Z

pkg/features/kube_features.go

@@ -214,6 +214,12 @@ const (
 	//
 	// Implement IPVS-based in-cluster service load balancing
 	SupportIPVSProxyMode utilfeature.Feature = "SupportIPVSProxyMode"
+
+	// owner: @dims
+	// beta: v1.10


Duh, cut-n-paste error :) fixing

tallclair · 2018-01-08T19:27:01Z

pkg/kubelet/apis/kubeletconfig/v1alpha1/types.go

@@ -197,6 +197,8 @@ type KubeletConfiguration struct {
 	// The CIDR to use for pod IP addresses, only used in standalone mode.
 	// In cluster mode, this is obtained from the master.
 	PodCIDR string `json:"podCIDR"`
+	// PodPidsLimit is the maximum number of pids in any pod.
+	PodPidsLimit int64


I think this should be a pointer, and also it needs a json tag.

tallclair · 2018-01-08T19:29:21Z

pkg/kubelet/cm/cgroup_manager_linux.go

@@ -463,6 +470,10 @@ func (m *cgroupManagerImpl) Create(cgroupConfig *CgroupConfig) error {
 		Resources: resources,
 	}

+	if cgroupConfig.ResourceParameters.PodPidsLimit != nil {


I think you should check the featuregate here?

tallclair · 2018-01-08T19:29:31Z

pkg/kubelet/cm/cgroup_manager_linux.go

@@ -430,6 +433,10 @@ func (m *cgroupManagerImpl) Update(cgroupConfig *CgroupConfig) error {
 		Paths:     cgroupPaths,
 	}

+	if cgroupConfig.ResourceParameters.PodPidsLimit != nil {


Check the feature gate here?

dims · 2018-01-09T11:25:22Z

/assign @brendandburns

k8s-ci-robot · 2018-01-09T11:25:23Z

@dims: GitHub didn't allow me to assign the following users: brendanburns.

Note that only kubernetes members can be assigned.

In response to this:

/assign @brendanburns

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

derekwaynecarr · 2018-01-09T13:55:06Z

/cc @sjenning

I did a quick pass and this looks generally good. Will take a deeper look shortly.

dchen1107 · 2018-01-09T14:36:21Z

Looks good to me, thanks.

cc/ @mtaufen

yujuhong · 2018-01-09T16:25:43Z

Would like to take a look of this later.

dims · 2018-01-11T13:08:53Z

@yujuhong @mtaufen @sjenning can you please take a look when you get a chance?

cc @kubernetes/sig-node-pr-reviews

sjenning · 2018-01-11T22:28:19Z

cmd/kubelet/app/options/options.go

@@ -453,6 +453,8 @@ func AddKubeletConfigFlags(fs *pflag.FlagSet, c *kubeletconfig.KubeletConfigurat
 	fs.Int32Var(&c.MaxPods, "max-pods", c.MaxPods, "Number of Pods that can run on this Kubelet.")

 	fs.StringVar(&c.PodCIDR, "pod-cidr", c.PodCIDR, "The CIDR to use for pod IP addresses, only used in standalone mode.  In cluster mode, this is obtained from the master.")
+	fs.Int64Var(c.PodPidsLimit, "pod-max-pids", *c.PodPidsLimit, "<Warning: Alpha feature> Tune a pod's pids limit. Set -1 for unlimited.")


nit reword: "Set the maximum number of processes per pod"

sjenning · 2018-01-11T22:29:36Z

/lgtm

dims · 2018-01-22T16:09:56Z

@derekwaynecarr - yep will file one today (at least a WIP). Thanks!

dims · 2018-01-23T18:58:07Z

@derekwaynecarr please see kubernetes/enhancements#546

@brendandburns @dchen1107 @lavalamp @smarterclayton @thockin - Need your approval per list in pkg/OWNERS. PTAL

dims · 2018-01-23T22:38:07Z

/test all

BenTheElder · 2018-01-23T23:34:34Z

/retest Looks like AWS quota

…

On Tue, Jan 23, 2018, 15:21 k8s-ci-robot ***@***.***> wrote: @dims <https://github.com/dims>: The following tests *failed*, say /retest to rerun them all: Test name Commit Details Rerun command pull-kubernetes-e2e-kubeadm-gce-canary 3df1ce5 <3df1ce5> link <https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/57973/pull-kubernetes-e2e-kubeadm-gce-canary/31/> /test pull-kubernetes-e2e-kubeadm-gce-canary pull-kubernetes-e2e-kops-aws 3df1ce5 <3df1ce5> link <https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/57973/pull-kubernetes-e2e-kops-aws/70690/> /test pull-kubernetes-e2e-kops-aws Full PR test history <https://k8s-gubernator.appspot.com/pr/57973>. Your PR dashboard <https://k8s-gubernator.appspot.com/pr/dims>. Please help us cut down on flakes by linking to <https://git.k8s.io/community/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests> an open issue <https://github.com/kubernetes/kubernetes/issues?q=is:issue+is:open> when you hit one in your PR. Instructions for interacting with me using PR comments are available here <https://git.k8s.io/community/contributors/devel/pull-requests.md>. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra <https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:> repository. I understand the commands that are listed here <https://go.k8s.io/bot-commands>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#57973 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4Bq1SmvOrHQ1a1UOWph5G1dej0qTztks5tNmd2gaJpZM4RWvsF> .

dims · 2018-01-24T00:30:39Z

/test pull-kubernetes-e2e-kubeadm-gce-canary

dims · 2018-01-24T11:21:46Z

Please ignore "pull-kubernetes-e2e-kubeadm-gce-canary" failure. It's not a required job and there's a race between the build artifacts and when the job starts actually. So totally unrelated.

/test all

k8s-ci-robot · 2018-01-24T11:24:23Z

@dims: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-kubeadm-gce-canary	`3df1ce5`	link	`/test pull-kubernetes-e2e-kubeadm-gce-canary`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

dims · 2018-01-25T01:40:31Z

@brendandburns @dchen1107 @lavalamp @smarterclayton @thockin - can one of you please approve for "pkg/" thanks!

Please ignore the gce-canary failure, it's not a required test and failing for an unrelated known reason (job depends on output of bazel-build and there's a problem with the stored artifact url)

smarterclayton · 2018-01-26T00:39:10Z

/approve

Given other feedback and sig consensus, no voiced concerns, benefit for real users with the possibility of a more comprehensive user facing solution later on.

k8s-ci-robot · 2018-01-26T00:41:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, dims, sjenning, smarterclayton

Associated issue: #43783

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cmd/kubelet/OWNERS~~ [derekwaynecarr,smarterclayton]
~~pkg/OWNERS~~ [smarterclayton]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

dims · 2018-01-26T01:12:15Z

thanks @smarterclayton

k8s-github-robot · 2018-01-26T02:29:30Z

Automatic merge from submit-queue (batch tested with PRs 57973, 57990). If you want to cherry-pick this change to another branch, please follow the instructions here.

kubernetes/kubernetes#57973

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 8, 2018

k8s-github-robot assigned tmrts and mtaufen Jan 8, 2018

dims mentioned this pull request Jan 8, 2018

Set pids limit for docker containers #57716

Closed

k8s-ci-robot assigned dchen1107, tallclair and thockin Jan 8, 2018

tallclair reviewed Jan 8, 2018

View reviewed changes

dims force-pushed the set-pids-limit-at-pod-level branch 4 times, most recently from 7655504 to b266fdc Compare January 8, 2018 21:40

k8s-ci-robot requested a review from sjenning January 9, 2018 13:55

yujuhong self-assigned this Jan 9, 2018

k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Jan 11, 2018

sjenning reviewed Jan 11, 2018

View reviewed changes

k8s-ci-robot assigned sjenning Jan 11, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 22, 2018

dims mentioned this pull request Jan 23, 2018

Allow end users to tweak maximum number of pids kubernetes/enhancements#546

Closed

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 26, 2018

k8s-github-robot merged commit bf11116 into kubernetes:master Jan 26, 2018

BenTheElder mentioned this pull request Jan 26, 2018

Use PID limits kubernetes/test-infra#6462

Closed

tallclair mentioned this pull request Feb 13, 2018

Graduate kubeletconfig API group to beta #53833

Merged

33 tasks

tengqm mentioned this pull request Feb 13, 2018

Document the SupportPodPidsLimit feature kubernetes/website#7386

Closed

2 tasks

youkoulayley pushed a commit to youkoulayley/acs-engine that referenced this pull request Apr 20, 2018

add pid pod limit

6324a28

kubernetes/kubernetes#57973

This was referenced May 24, 2018

Set pids limit at pod level, cherry pick https://github.com/kubernetes/kubernetes/pull/57973 kpaas-io/kubernetes#40

Closed

Set pids limit at pod level kpaas-io/kubernetes#42

Merged

jackfrancis mentioned this pull request May 30, 2018

Add pod-pid-limit options to kubelet in kubernetes 1.10 Azure/acs-engine#2736

Merged

2 tasks

This was referenced Jun 19, 2018

Set pids limit at pod level (1.7) kpaas-io/kubernetes#44

Merged

Set pids limit at pod level (1.8) kpaas-io/kubernetes#45

Merged

stevekuznetsov mentioned this pull request Jul 23, 2018

figure out podutils and init / zombies kubernetes/test-infra#8779

Closed

dims mentioned this pull request Sep 21, 2018

Kubelet pod max PIDs not being inherited by Pod containers #68835

Closed

yujuhong mentioned this pull request Dec 17, 2018

SupportPodPidsLimit feature beta with tests #72076

Merged

manics mentioned this pull request Mar 4, 2019

Access to docker/runtime options (PID limits) jupyterhub/kubespawner#303

Closed

kshcherban mentioned this pull request Jun 11, 2019

Allow setting default docker run options aws/containers-roadmap#334

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set pids limit at pod level #57973

Set pids limit at pod level #57973

dims commented Jan 8, 2018

dims commented Jan 8, 2018

dims commented Jan 8, 2018

dims commented Jan 8, 2018

tallclair left a comment

tallclair Jan 8, 2018

dims Jan 8, 2018

tallclair Jan 8, 2018

dims Jan 8, 2018

tallclair Jan 8, 2018

dims Jan 8, 2018

tallclair Jan 8, 2018

dims Jan 8, 2018

dims commented Jan 9, 2018 •

edited

Loading

k8s-ci-robot commented Jan 9, 2018

derekwaynecarr commented Jan 9, 2018

dchen1107 commented Jan 9, 2018

yujuhong commented Jan 9, 2018

dims commented Jan 11, 2018

sjenning Jan 11, 2018

dims Jan 11, 2018

sjenning commented Jan 11, 2018

dims commented Jan 22, 2018 •

edited

Loading

dims commented Jan 23, 2018

dims commented Jan 23, 2018

BenTheElder commented Jan 23, 2018 via email

dims commented Jan 24, 2018

dims commented Jan 24, 2018

k8s-ci-robot commented Jan 24, 2018 •

edited

Loading

dims commented Jan 25, 2018

smarterclayton commented Jan 26, 2018

k8s-ci-robot commented Jan 26, 2018

dims commented Jan 26, 2018

k8s-github-robot commented Jan 26, 2018

Set pids limit at pod level #57973

Set pids limit at pod level #57973

Conversation

dims commented Jan 8, 2018

dims commented Jan 8, 2018

dims commented Jan 8, 2018

dims commented Jan 8, 2018

tallclair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims commented Jan 9, 2018 • edited Loading

k8s-ci-robot commented Jan 9, 2018

derekwaynecarr commented Jan 9, 2018

dchen1107 commented Jan 9, 2018

yujuhong commented Jan 9, 2018

dims commented Jan 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjenning commented Jan 11, 2018

dims commented Jan 22, 2018 • edited Loading

dims commented Jan 23, 2018

dims commented Jan 23, 2018

BenTheElder commented Jan 23, 2018 via email

dims commented Jan 24, 2018

dims commented Jan 24, 2018

k8s-ci-robot commented Jan 24, 2018 • edited Loading

dims commented Jan 25, 2018

smarterclayton commented Jan 26, 2018

k8s-ci-robot commented Jan 26, 2018

dims commented Jan 26, 2018

k8s-github-robot commented Jan 26, 2018

dims commented Jan 9, 2018 •

edited

Loading

dims commented Jan 22, 2018 •

edited

Loading

k8s-ci-robot commented Jan 24, 2018 •

edited

Loading