Cgroup 'cpu.cfs_quota_us' error on container creation when CPU limit is low #23113

antoineco · 2016-03-17T11:52:18Z

After upgrading my environment from Kubernetes 1.1.8 to Kubernetes 1.2.0, I see pods creation failing with the following resource limits set:

        resources:
          limits:
            cpu: 5m
            memory: 25Mi

Translated to Docker container info:

1.1.8	1.2.0
"Memory": 26214400,	"Memory": 26214400,
"MemoryReservation": 0,	"MemoryReservation": 0,
"CpuShares": 5,	"CpuShares": 5,
"CpuPeriod": 0,	"CpuPeriod": 100000,
"CpuQuota": 0,	"CpuQuota": 500,

The error message is:

Cannot start container <sha>: [9] System error: write /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-<sha>.scope/cpu.cfs_quota_us: invalid argument

I can reproduce this by creating containers manually:

$ docker run --rm --cpu-shares=5 --cpu-quota=0 --cpu-period=0 tianon/true
[exit code 0]
$ docker run --rm --cpu-shares=5 --cpu-quota=500 --cpu-period=100000 tianon/true
docker: Error response from daemon: Cannot start container <sha>: [9] System error: write /sys/fs/cgroup/cpu,cpuacct/docker/<sha>/cpu.cfs_quota_us: invalid argument.
[exit code 125]

Kubernetes should check that the defined CPU restrictions are valid before attempting to create containers.

The text was updated successfully, but these errors were encountered:

antoineco · 2016-03-17T12:24:06Z

It works with limits.cpu = 10m, translated to:

            "CpuShares": 10,
            "CpuPeriod": 100000,
            "CpuQuota": 1000,

So now that I understand better, it will always fail with limits.cpu < 10m

bgrant0607 · 2016-03-17T13:53:09Z

cc @vishh

adohe-zz · 2016-03-17T15:23:44Z

I tried to reproduce this on my local with HEAD:
the pod yaml file:

[tony@192 user-guide]$ cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        cpu: 5m 
        memory: 25Mi 
    ports:
    - containerPort: 80

create pod with yaml:

[tony@192 kubernetes]$ cluster/kubectl.sh create -f docs/user-guide/pod.yaml 
pod "nginx" created

and pod status:

[tony@192 kubernetes]$ cluster/kubectl.sh get pods nginx
NAME      READY     STATUS    RESTARTS   AGE
nginx     1/1       Running   0          12m
[tony@192 kubernetes]$ cluster/kubectl.sh describe pods nginx
Name:       nginx
Namespace:  default
Node:       127.0.0.1/127.0.0.1
Start Time: Thu, 17 Mar 2016 10:40:46 -0400
Labels:     app=nginx
Status:     Running
IP:     172.17.0.3
Controllers:    <none>
Containers:
  nginx:
    Container ID:   docker://c2cb6c436155206b896ebe04e3125221e3e56a7cd84b5daad1a52283aa11e409
    Image:      nginx
    Image ID:       docker://36b0adefb2f371685aa764c3d47eaaf12583814c7bccb13282a0fe9423526fb5
    Port:       80/TCP
    QoS Tier:
      cpu:  Guaranteed
      memory:   Guaranteed
    Limits:
      memory:   25Mi
      cpu:  5m
    Requests:
      cpu:      5m
      memory:       25Mi
    State:      Running
      Started:      Thu, 17 Mar 2016 10:43:34 -0400
    Ready:      True
    Restart Count:  0
    Environment Variables:
Conditions:
  Type      Status
  Ready     True 
Volumes:
  default-token-kbojt:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-kbojt
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath       Type        Reason      Message
  --------- --------    -----   ----            -------------       --------    ------      -------
  21m       21m     1   {default-scheduler }                Normal      Scheduled   Successfully assigned nginx to 127.0.0.1
  20m       20m     1   {kubelet 127.0.0.1} spec.containers{nginx}  Normal      Pulling     pulling image "nginx"
  18m       18m     1   {kubelet 127.0.0.1} spec.containers{nginx}  Normal      Pulled      Successfully pulled image "nginx"
  18m       18m     1   {kubelet 127.0.0.1} spec.containers{nginx}  Normal      Created     Created container with docker id c2cb6c436155
  18m       18m     1   {kubelet 127.0.0.1} spec.containers{nginx}  Normal      Started     Started container with docker id c2cb6c436155

and then I tried to reproduce directly with docker:

[tony@192 kubernetes]$ docker run --rm --cpu-shares=5 --cpu-quota=0 --cpu-period=0 tianon/true
[tony@192 kubernetes]$ docker run --rm --cpu-shares=5 --cpu-quota=500 --cpu-period=100000 tianon/true
Error response from daemon: Cannot start container 466d6b8b10faa1e020e1e5ef1072a12b3bf84de0a49613b0ac1ceaab5b1ef578: [8] System error: write /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-466d6b8b10faa1e020e1e5ef1072a12b3bf84de0a49613b0ac1ceaab5b1ef578.scope/cpu.cfs_quota_us: invalid argument

looks like a little bit tricky.

antoineco · 2016-03-17T15:54:29Z

@adohe did you docker inspect the container c2cb6c436155? It could be that the values of the Cpu* settings have already been adjusted but not cherry picked in the release 1.2 branch (?)

derekwaynecarr · 2016-03-17T16:56:09Z

A few details would help here:

operating system
docker version
docker cgroup driver
where/how you install docker

There have been some fixes in RHEL based distros to solve this problem, so it may be related.

derekwaynecarr · 2016-03-17T16:57:30Z

openshift/origin#6822 discusses similar problems.

projectatomic/docker#76 (comment) is fix we are carrying/evaluating.

mrunalp · 2016-03-17T17:18:09Z

This issue may need more investigation. Even with the Delegate=yes patch, I am seeing this. However, this isn't isolated to the systemd cgroups driver. I see the same issue with the cgroupfs driver as well. I will check if there is some kernel limitation that is preventing setting these values.

antoineco · 2016-03-17T17:18:15Z

@derekwaynecarr the OS is CoreOS 983.0.0 with Docker 1.10.2 (preinstalled). It uses the systemd cgroup driver, but cgroupfs shows the same results. From what I've seen this particular issue is purely related to the value of CpuQuota.

mrunalp · 2016-03-17T17:19:51Z

@antoineco Could you try the cgroupfs driver as well?
I see the same issue with the cgroupfs driver

[root@dhcp-16-129 ~]# /root/gosrc/src/github.com/docker/docker/bundles/latest/dynbinary/docker run --rm -it --cpu-shares=5 --cpu-quota=500 --cpu-period=100000 busybox
/root/gosrc/src/github.com/docker/docker/bundles/latest/dynbinary/docker: Error response from daemon: Cannot start container 4d45ab918f1e931bd9b7e08f54da7642135f8db8955471712b7fa44c81ef9c52: [10] System error: write /sys/fs/cgroup/cpu,cpuacct/docker/4d45ab918f1e931bd9b7e08f54da7642135f8db8955471712b7fa44c81ef9c52/cpu.cfs_quota_us: invalid argument.

mrunalp · 2016-03-17T17:24:11Z

Here is the reproduction, isolated from docker.

[root@dhcp-16-129 system.slice]# mkdir container-abcd.scope
[root@dhcp-16-129 system.slice]# cd container-abcd.scope/
[root@dhcp-16-129 container-abcd.scope]# ls
cgroup.clone_children  cgroup.procs  cpuacct.stat  cpuacct.usage  cpuacct.usage_percpu  cpu.cfs_period_us  cpu.cfs_quota_us  cpu.shares  cpu.stat  notify_on_release  tasks

[root@dhcp-16-129 container-abcd.scope]# echo -n 5 > cpu.shares 
[root@dhcp-16-129 container-abcd.scope]# echo -n 100000 > cpu.cfs_period_us 
[root@dhcp-16-129 container-abcd.scope]# cat cpu.shares 
5
[root@dhcp-16-129 container-abcd.scope]# cat cpu.cfs_period_us 
100000
[root@dhcp-16-129 container-abcd.scope]# echo -n 500 > cpu.cfs_quota_us 
-bash: echo: write error: Invalid argument
[root@dhcp-16-129 container-abcd.scope]# uname -a
Linux dhcp-16-129.sjc.redhat.com 4.5.0-0.rc7.git2.1.fc25.x86_64 #1 SMP Wed Mar 9 17:07:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@dhcp-16-129 container-abcd.scope]# pwd
/sys/fs/cgroup/cpu/system.slice/container-abcd.scope

vishh · 2016-03-17T17:28:42Z

1 millisecond is the minimum value for CFS quota. We need to fix it in code.

mrunalp · 2016-03-17T17:29:33Z

@vishh Thanks! That explains it.

vishh · 2016-03-17T17:30:16Z

I can post a patch for that.

mrunalp · 2016-03-17T17:31:14Z

@vishh yeah, makes sense to catch this in runc/libcontainer.

mrunalp · 2016-03-17T17:31:47Z

For posterity, it is documented here https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt

mrunalp · 2016-03-17T17:34:15Z

This could also be caught here in kubernetes to prevent even trying to spawn a container.

vishh · 2016-03-17T17:35:50Z

@mrunalp: Thats what I intend on doing. It might be worth validating in libcontainer though :)

mrunalp · 2016-03-17T17:39:50Z

@vishh I'll add that :)

antoineco · 2016-03-17T18:11:52Z

But @adohe mentioned he can not reproduce it on HEAD with a CPU limit of 5m. It means on head to CpuQuota is set properly (or '0')

mwielgus added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Mar 17, 2016

vishh added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Mar 17, 2016

vishh self-assigned this Mar 17, 2016

vishh mentioned this issue Mar 17, 2016

Validate minimum CPU limits to be >= 10m #23143

Merged

k8s-github-robot closed this as completed in #23143 Mar 18, 2016

xordspar0 mentioned this issue Oct 6, 2020

Add support for resource limits to play kube containers/podman#7853

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cgroup 'cpu.cfs_quota_us' error on container creation when CPU limit is low #23113

Cgroup 'cpu.cfs_quota_us' error on container creation when CPU limit is low #23113

antoineco commented Mar 17, 2016

antoineco commented Mar 17, 2016

bgrant0607 commented Mar 17, 2016

adohe-zz commented Mar 17, 2016

antoineco commented Mar 17, 2016

derekwaynecarr commented Mar 17, 2016

derekwaynecarr commented Mar 17, 2016

mrunalp commented Mar 17, 2016

antoineco commented Mar 17, 2016

mrunalp commented Mar 17, 2016

mrunalp commented Mar 17, 2016

vishh commented Mar 17, 2016

mrunalp commented Mar 17, 2016

vishh commented Mar 17, 2016

mrunalp commented Mar 17, 2016

mrunalp commented Mar 17, 2016

mrunalp commented Mar 17, 2016

vishh commented Mar 17, 2016

mrunalp commented Mar 17, 2016

antoineco commented Mar 17, 2016

Cgroup 'cpu.cfs_quota_us' error on container creation when CPU limit is low #23113

Cgroup 'cpu.cfs_quota_us' error on container creation when CPU limit is low #23113

Comments

antoineco commented Mar 17, 2016

antoineco commented Mar 17, 2016

bgrant0607 commented Mar 17, 2016

adohe-zz commented Mar 17, 2016

antoineco commented Mar 17, 2016

derekwaynecarr commented Mar 17, 2016

derekwaynecarr commented Mar 17, 2016

mrunalp commented Mar 17, 2016

antoineco commented Mar 17, 2016

mrunalp commented Mar 17, 2016

mrunalp commented Mar 17, 2016

vishh commented Mar 17, 2016

mrunalp commented Mar 17, 2016

vishh commented Mar 17, 2016

mrunalp commented Mar 17, 2016

mrunalp commented Mar 17, 2016

mrunalp commented Mar 17, 2016

vishh commented Mar 17, 2016

mrunalp commented Mar 17, 2016

antoineco commented Mar 17, 2016