System error: The minimum allowed cpu-shares is 1024 #22818
Description
I have a 10-node kubernetes cluster running on AWS m3.medium instances with CoreOS. When I run the kubernetes density e2e tests, containers fail to start with the error "System error: The minimum allowed cpu-shares is 1024". The requested CPU shares is 10 for each container. I have not seen this error when using AWS m3.large instances.
Output of docker version:
core@autoscaled ~ $ docker version
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 8acee1b
Built:
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 8acee1b
Built:
OS/Arch: linux/amd64
core@autoscaled ~ $
Output of docker info:
core@autoscaled ~ $ docker info
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 2
Server Version: 1.10.3
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.5.0-coreos-r1
Operating System: CoreOS 1010.1.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.679 GiB
Name: autoscaled
ID: R6H5:DVUQ:XSAD:K6RP:LQA2:MWXO:NUD2:UEP4:TNN6:Q3MN:3P66:BTSL
core@autoscaled ~ $
Additional environment details (AWS, VirtualBox, physical, etc.):
The non control plane nodes are m3.mediums (1 vCPU and 3.75 GiB RAM) running CoreOS version 1010.1.0 with docker 1.10, however I have also seen the error at least once using CoreOS 991.2.0 with docker 1.09.
Steps to reproduce the issue:
- Create a kubernetes cluster with m3.medium nodes. My cluster was created using kraken with the following terraform configuration:
aws_user_prefix = "dew"
node_count = "10"
aws_node_type = "m3.medium"
aws_etcd_type = "m4.2xlarge"
aws_master_type = "m4.4xlarge"
apiserver_count = "1"
aws_apiserver_type = "m4.2xlarge"
special_node_type = "m3.xlarge"
coreos_update_channel = "alpha"
coreos_version = "1010.1.0"
- Run the e2e density test.
Describe the results you received:
core@autoscaled ~ $ journalctl | grep cpu-shares
May 16 19:38:16 autoscaled dockerd[1083]: time="2016-05-16T19:38:16.992403355Z" level=error msg="Handler for POST /containers/f7cbef37f15fb830c5b71934ff3e410f81605bbd4a85124cf6d0aaaa95214008/start returned error: Cannot start container f7cbef37f15fb830c5b71934ff3e410f81605bbd4a85124cf6d0aaaa95214008: [9] System error: The minimum allowed cpu-shares is 1024"
May 16 19:38:16 autoscaled hyperkube[3827]: E0516 19:38:16.993365 3827 manager.go:1953] Error running pod "density30-bbaecf52-1b9d-11e6-978f-f45c898cf22b-es0fh_e2e-tests-density-mmkq0(bba4d015-1b9d-11e6-86f1-02580e2da74b)" container "density30-bbaecf52-1b9d-11e6-978f-f45c898cf22b": runContainer: API error (500): Cannot start container f7cbef37f15fb830c5b71934ff3e410f81605bbd4a85124cf6d0aaaa95214008: [9] System error: The minimum allowed cpu-shares is 1024
May 16 19:38:16 autoscaled hyperkube[3827]: E0516 19:38:16.993739 3827 pod_workers.go:138] Error syncing pod bba4d015-1b9d-11e6-86f1-02580e2da74b, skipping: failed to "StartContainer" for "density30-bbaecf52-1b9d-11e6-978f-f45c898cf22b" with RunContainerError: "runContainer: API error (500): Cannot start container f7cbef37f15fb830c5b71934ff3e410f81605bbd4a85124cf6d0aaaa95214008: [9] System error: The minimum allowed cpu-shares is 1024\n"
core@autoscaled ~ $
Describe the results you expected:
I expect the container to start.
Additional information you deem important (e.g. issue happens only occasionally):
I can reliably reproduce this on a 10 node cluster.