Okay the kernel option CONFIG_CFS_BANDWIDTH=y is not enabled on the AWS jessie OS.

Could be classified as a Kubernetes deploy bug, maybe. I think OpenShift should handle the situation that this cgroup feature is not enabled because end-users might deploy to bare metal with an unknown OS installed.

This particular problem doesn't apply to "wily" on AWS, and I think most newer distributions, so I'd say it's lower priority

@derekwaynecarr @mrunalp @smarterclayton what would we want to do in the case that the cgroup filesystem is not available? I really don't want to fallback to "no limit" in that scenario, but i'm not sure what else would be appropriate.

Is it even valid to run k8s/openshift w/o the CONFIG_CFS_BANDWIDTH=y kernel option, given that it makes our quotas impossible to enforce(I assume)?

In general, I would expect us to gracefully degrade if it's a configuration thing (warn). What I'm concerned with is flakes - cgroup being notoriously racy at the best of times, what happens if we tolerate missing cgroups, but instead of it failing, we continue (thus the admin has only a small chance of catching it)? That seems risky.

What might make sense is to have a Kube/Origin flag for "allow-missing-cgroups" that a deployer can opt in to that makes provision for these things not existing. For Origin, it would have to be in the controller config (so that it can be passed to the build pod). But for Kube, seems much more likely to be a node level thing (so maybe a separate flag).

If it's not blocking you now, we could potentially say "barring a new cgroup being introduced that Kube uses but no one supports, we will continue to error out".

Unfortunately this fails under the default AWS deploy for Kubernetes. @justinsb

Maybe we can escalate this to a "let's fix this in Kubernetes AWS deploy" then, if you want to enforce the option being enabled.

I think that either our installer or the openshift binary could check for the cgroups subsystems on startup and fail if the expected cgroups subsystems aren't found. This can be checked by either parsing /proc/self/cgroup or /proc/cgroups files.

We can fix this in our debian image, I think - or at least look into doing so. Can you open an issue please?

In the meantime, you might want to use wily (export KUBE_OS_DISTRIBUTION=wily)

On wily I now almost instantly get this build error:

I0317 19:29:24.730862       1 source.go:197] Downloading "https://github.com/enokd/docker-node-hello.git" ...
W0317 19:29:26.454135       1 common.go:89] An error occurred saving build revision: the server could not find the requested resource (put builds docker-node-hello-4)
Step 1 : FROM centos@sha256:ec1bf627545d77d05270b3bbd32a9acca713189c58bc118f21abd17ff2629e3f
 ---> ed452988fb6e
Step 2 : RUN rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
 ---> Using cache
 ---> c8993aa3f727
Step 3 : RUN yum install -y -q npm
 ---> [Warning] Your kernel does not support swap limit capabilities, memory limited without swap.
 ---> Running in c9ea818b4eb8
F0317 19:29:26.805547       1 builder.go:204] Error: build error: [8] System error: The minimum allowed cpu-shares is 1024

Cpu-shares.

@paralin can you provide the rest of the build log? I believe we dump out the cgroup settings we are using (at least if you enable loglevel 5 in your build)

@paralin What version of docker do are you running?

@mrunalp 1.9.1

@bparees how do I enable loglevel 5 in a build?

@paralin discussed here: https://docs.openshift.org/latest/dev_guide/builds.html#accessing-build-logs

Big props to your work on the web UI. Much easier to make quick tweaks to things like the trigger list and such.

Log level 5:

I0317 23:24:04.327588       1 builder.go:57] Master version "v1.1.4", Builder version "v1.1.4"
I0317 23:24:04.328387       1 builder.go:145] Running build with cgroup limits: api.CGroupLimits{MemoryLimitBytes:9223372036854771712, CPUShares:2, CPUPeriod:100000, CPUQuota:-1, MemorySwap:9223372036854771712}
I0317 23:24:04.328857       1 source.go:197] Downloading "https://github.com/enokd/docker-node-hello.git" ...
I0317 23:24:04.917862       1 source.go:208] Cloning source from https://github.com/enokd/docker-node-hello.git
W0317 23:24:06.298439       1 common.go:89] An error occurred saving build revision: the server could not find the requested resource (put builds docker-node-hello-6)
Step 1 : FROM centos@sha256:ec1bf627545d77d05270b3bbd32a9acca713189c58bc118f21abd17ff2629e3f
 ---> ed452988fb6e
Step 2 : ENV "BUILD_LOGLEVEL" "2"
 ---> Using cache
 ---> ad6a59f257b6
Step 3 : RUN rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
 ---> Using cache
 ---> 86f9329cc8f9
Step 4 : RUN yum install -y -q npm
 ---> Using cache
 ---> bbcea1291029
Step 5 : ADD . /src
 ---> dc3df86a396f
Removing intermediate container b8fa11a59617
Step 6 : RUN cd /src; npm install
 ---> [Warning] Your kernel does not support swap limit capabilities, memory limited without swap.
 ---> Running in 8acf707954ad
F0317 23:24:06.869910       1 builder.go:204] Error: build error: [8] System error: The minimum allowed cpu-shares is 1024

Odd also that it says it can't "put builds". This is probably related to another issue I made regarding the internal Kube stuff.

Now it's happening in the pod events too:

Events:
  FirstSeen     LastSeen        Count   From                                                    SubobjectPath      Type             Reason          Message
  ---------     --------        -----   ----                                                    -------------      -------- ------          -------
  1m            1m              1       {default-scheduler }                                                       Normal           Scheduled       Successfully assigned docker-registry-1-deploy to ip-172-20-0-77.us-west-2.compute.i
nternal
  1m            1m              1       {kubelet ip-172-20-0-77.us-west-2.compute.internal}     spec.containers{deployment} Normal          Pulling         pulling image "openshift/origin-deployer:v1.1.4"
  1m            1m              1       {kubelet ip-172-20-0-77.us-west-2.compute.internal}     spec.containers{deployment} Normal          Pulled          Successfully pulled image "openshift/origin-deployer:v1.1.4"
  1m            1m              1       {kubelet ip-172-20-0-77.us-west-2.compute.internal}     spec.containers{deployment} Normal          Created         Created container with docker id a03de03833a2
  1m            1m              1       {kubelet ip-172-20-0-77.us-west-2.compute.internal}     spec.containers{deployment} Warning         Failed          Failed to start container with docker id a03de03833a2 with error: API error 
(500): Cannot start container a03de03833a2917d9939904d06f8d834b89a6b0801ea3d2a9e7aeef7f9a470f8: [8] System error: The minimum allowed cpu-shares is 1024

  1m    1m      1       {kubelet ip-172-20-0-77.us-west-2.compute.internal}             Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "deployment" with RunContainerError: "runContainer: API erro
r (500): Cannot start container a03de03833a2917d9939904d06f8d834b89a6b0801ea3d2a9e7aeef7f9a470f8: [8] System error: The minimum allowed cpu-shares is 1024\n"

What a mess. I think the cgroup not existing happens in Jessie and this weird cpu-shares error happens under wily.

@mrunalp @derekwaynecarr any thoughts on
System error: The minimum allowed cpu-shares is 1024\n"

?

kubernetes/kubernetes#23143, I think

I think the Debian version of docker needs the patch that I added to our version for joining all cgroups.

Sent from my iPhone

On Mar 18, 2016, at 7:26 AM, Ben Parees notifications@github.com wrote:

@mrunalp @derekwaynecarr any thoughts on
System error: The minimum allowed cpu-shares is 1024\n"

?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

@mrunalp

the kubelet checks for the required stuff here on start-up.

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/container_manager_linux.go#L99

@bparees
Its a valid configuration of OpenShift to disable enforcement of cpu limits by passing the kubelet flag in the node-config.yaml for --cpu-cfs-quota=false

@paralin - are you enabling the --cpu-cfs-quota flag on the kubernetes installs you are testing against?

Its a valid configuration of OpenShift to disable enforcement of cpu limits by passing the kubelet flag in the node-config.yaml for --cpu-cfs-quota=false

@derekwaynecarr what are the implications of that for me? Presumably the build pod would not have a cgroup limit value set, so we'd just read the quota as "-1" (the file would exist, but with a value of -1) and we'd set the limit on the builder container to -1 as well. Do we need to actually understand that enforcement is disabled and not try to set a cgroup limit at all?

@bparees - I do not think it should impact you.

@bparees - I take that back slightly. I guess its possible that when not running on RHEL-based distros that we would not find the cpu quota file when validating. I am not sure that is an environment we really support. We support disabling the kubelet from enforcing cpu limits, but I think we still require that the underlying OS could enforce quota.

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/container_manager_linux.go#L137

@derekwaynecarr if you want to support running under kubernetes then you can't assume anything about the host operating system. People could be running on bare metal with specific limitations due to specifications/restrictions etc. So I would suggest gracefully falling back with warnings, much like Redis outputs kernel option suggestions on startup when certain performance impacting issues are found.

@bparees @derekwaynecarr - with the CRI-O related changes via 1c1f653#diff-bb3dfd2a9b83662fd66fdf176d55aa53 from last fall, we seem to tolerate the lack of a cgroup files.

See https://github.com/openshift/origin/blob/master/pkg/build/builder/util_linux.go#L59-L62 in particular

Seems like we can close this out as fixed .... agreed, or is there a nuance I'm overlooking in reviewing the history here?

thanks

@gabemontero huh. good for us :) Yeah I guess it can be closed out.