From d57263b94be30a9133005383e7900b2efbb085f4 Mon Sep 17 00:00:00 2001 From: dinghaiyang Date: Fri, 4 Sep 2015 18:44:56 +0800 Subject: [PATCH] Replace limits with request where appropriate --- docs/design/resources.md | 5 +- docs/proposals/resource-qos.md | 3 +- docs/user-guide/compute-resources.md | 77 ++++++++++++++++------------ docs/user-guide/production-pods.md | 7 ++- 4 files changed, 54 insertions(+), 38 deletions(-) diff --git a/docs/design/resources.md b/docs/design/resources.md index fe6f0ec76ab49..f9bbc8db98bb8 100644 --- a/docs/design/resources.md +++ b/docs/design/resources.md @@ -33,8 +33,8 @@ Documentation for other releases can be found at **Note: this is a design doc, which describes features that have not been completely implemented. User documentation of the current state is [here](../user-guide/compute-resources.md). The tracking issue for implementation of this model is -[#168](http://issue.k8s.io/168). Currently, only memory and -cpu limits on containers (not pods) are supported. "memory" is in bytes and "cpu" is in +[#168](http://issue.k8s.io/168). Currently, both limits and requests of memory and +cpu on containers (not pods) are supported. "memory" is in bytes and "cpu" is in milli-cores.** # The Kubernetes resource model @@ -123,7 +123,6 @@ Where: * Internally, the Kubernetes master can decide the defaulting behavior and the kubelet implementation may expected an absolute specification. For example, if the master decided that "the default is unbounded" it would pass 2^64 to the kubelet. - ## Kubernetes-defined resource types The following resource types are predefined ("reserved") by Kubernetes in the `kubernetes.io` namespace, and so cannot be used for user-defined resources. Note that the syntax of all resource types in the resource spec is deliberately similar, but some resource types (e.g., CPU) may receive significantly more support than simply tracking quantities in the schedulers and/or the Kubelet. diff --git a/docs/proposals/resource-qos.md b/docs/proposals/resource-qos.md index 6d7ddcced2ccf..f76a0306ebfe8 100644 --- a/docs/proposals/resource-qos.md +++ b/docs/proposals/resource-qos.md @@ -101,7 +101,8 @@ API changes for request - Add validation code that checks request <= limit, and validation test cases (api/validation/validation.go) Scheduler Changes -- Use requests instead of limits in CheckPodsExceedingCapacity and PodFitsResources (scheduler/algorithm/predicates.go) +- Predicates: Use requests instead of limits in CheckPodsExceedingCapacity and PodFitsResources (scheduler/algorithm/predicates/predicates.go) +- Priorities: Use requests instead of limits in LeastRequestedPriority and BalancedResourceAllocation(scheduler/algorithm/priorities/priorities.go)(PR #12718) Container Manager Changes - Use requests to assign CPU shares for Docker (kubelet/dockertools/container_manager.go) diff --git a/docs/user-guide/compute-resources.md b/docs/user-guide/compute-resources.md index 7ee8a73073010..c7c2a9e2dac9d 100644 --- a/docs/user-guide/compute-resources.md +++ b/docs/user-guide/compute-resources.md @@ -37,8 +37,8 @@ Documentation for other releases can be found at - [Compute Resources](#compute-resources) - - [Container and Pod Resource Limits](#container-and-pod-resource-limits) - - [How Pods with Resource Limits are Scheduled](#how-pods-with-resource-limits-are-scheduled) + - [Resource Requests and Limits of Pod and Container](#resource-requests-and-limits-of-pod-and-container) + - [How Pods with Resource Requests are Scheduled](#how-pods-with-resource-requests-are-scheduled) - [How Pods with Resource Limits are Run](#how-pods-with-resource-limits-are-run) - [Monitoring Compute Resource Usage](#monitoring-compute-resource-usage) - [Troubleshooting](#troubleshooting) @@ -49,9 +49,11 @@ Documentation for other releases can be found at When specifying a [pod](pods.md), you can optionally specify how much CPU and memory (RAM) each -container needs. When containers have resource limits, the scheduler is able to make better -decisions about which nodes to place pods on, and contention for resources can be handled in a -consistent manner. +container needs. When containers have their resource requests specified, the scheduler is +able to make better decisions about which nodes to place pods on; and when containers have their +limits specified, contention for resources on a node can be handled in a specified manner. For +more details about the difference between requests and limits, please refer to +[Resource QoS](../proposals/resource-qos.md). *CPU* and *memory* are each a *resource type*. A resource type has a base unit. CPU is specified in units of cores. Memory is specified in units of bytes. @@ -62,22 +64,26 @@ distinct from [API resources](working-with-resources.md). API resources, such a [services](services.md) are objects that can be written to and retrieved from the Kubernetes API server. -## Container and Pod Resource Limits +## Resource Requests and Limits of Pod and Container Each container of a Pod can optionally specify `spec.container[].resources.limits.cpu` and/or -`spec.container[].resources.limits.memory`. The `spec.container[].resources.requests` field is not -currently used and need not be set. - -Specifying resource limits is optional. In some clusters, an unset value may be replaced with a -default value when a pod is created or updated. The default value depends on how the cluster is -configured. - -Although limits can only be specified on individual containers, it is convenient to talk about pod -resource limits. A *pod resource limit* for a particular resource type is the sum of the resource -limits of that type for each container in the pod, with unset values treated as zero. - -The following pod has two containers. Each has a limit of 0.5 core of cpu and 128MiB -(220 bytes) of memory. The pod can be said to have a limit of 1 core and 256MiB of +`spec.container[].resources.limits.memory` and/or `spec.container[].resources.requests.cpu` +and/or `spec.container[].resources.requests.memory`. + +Specifying resource requests and/or limits is optional. In some clusters, unset limits or requests +may be replaced with default values when a pod is created or updated. The default value depends on +how the cluster is configured. If value of requests is not specified, they are set to be equal +to limits by default. Please note that resource limits must be greater than or equal to resource +requests. + +Although requests/limits can only be specified on individual containers, it is convenient to talk +about pod resource requests/limits. A *pod resource request/limit* for a particular resource +type is the sum of the resource requests/limits of that type for each container in the pod, with +unset values treated as zero (or equal to default values in some cluster configurations). + +The following pod has two containers. Each has a request of 0.25 core of cpu and 64MiB +(220 bytes) of memory and a limit of 0.5 core of cpu and 128MiB of memory. The pod can +be said to have a request of 0.5 core and 128 MiB of memory and a limit of 1 core and 256MiB of memory. ```yaml @@ -90,32 +96,35 @@ spec: - name: db image: mysql resources: + requests: + memory: "64Mi" + cpu: "250m" limits: memory: "128Mi" cpu: "500m" - name: wp image: wordpress resources: - limits: + requests: + memory: "64Mi" + cpu: "250m" + limits: memory: "128Mi" cpu: "500m" ``` -## How Pods with Resource Limits are Scheduled +## How Pods with Resource Requests are Scheduled When a pod is created, the Kubernetes scheduler selects a node for the pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for pods. The scheduler ensures that, -for each resource type (CPU and memory), the sum of the resource limits of the +for each resource type (CPU and memory), the sum of the resource requests of the containers scheduled to the node is less than the capacity of the node. Note that although actual memory or CPU resource usage on nodes is very low, the scheduler will still refuse to place pods onto nodes if the capacity check fails. This protects against a resource shortage on a node when resource usage later increases, such as due to a daily peak in request rate. -Note: Although the scheduler normally spreads pods out across nodes, there are currently some cases -where pods with no limits (unset values) might all land on the same node. - ## How Pods with Resource Limits are Run When kubelet starts a container of a pod, it passes the CPU and memory limits to the container @@ -157,13 +166,17 @@ until a place can be found. An event will be produced each time the scheduler place for the pod, like this: ```console -$ kubectl describe pods/frontend | grep -A 3 Events +$ kubectl describe pod frontend | grep -A 3 Events Events: - FirstSeen LastSeen Count From SubobjectPath Reason Message - Tue, 30 Jun 2015 09:01:41 -0700 Tue, 30 Jun 2015 09:39:27 -0700 128 {scheduler } failedScheduling Error scheduling: For each of these fitness predicates, pod frontend failed on at least one node: PodFitsResources. + FirstSeen LastSeen Count From Subobject PathReason Message + 36s 5s 6 {scheduler } FailedScheduling Failed for reason PodExceedsFreeCPU and possibly others + ``` -If a pod or pods are pending with this message, then there are several things to try: +In the case shown above, the pod "frontend" fails to be scheduled due to insufficient +CPU resource on the node. Similar error messages can also suggest failure due to insufficient +memory (PodExceedsFreeMemory). In general, if a pod or pods are pending with this message and +alike, then there are several things to try: - Add more nodes to the cluster. - Terminate unneeded pods to make room for pending pods. - Check that the pod is not larger than all the nodes. For example, if all the nodes @@ -266,13 +279,11 @@ The current system only allows resource quantities to be specified on a containe It is planned to improve accounting for resources which are shared by all containers in a pod, such as [EmptyDir volumes](volumes.md#emptydir). -The current system only supports container limits for CPU and Memory. +The current system only supports container requests and limits for CPU and Memory. It is planned to add new resource types, including a node disk space resource, and a framework for adding custom [resource types](../design/resources.md#resource-types). -The current system does not facilitate overcommitment of resources because resources reserved -with container limits are assured. It is planned to support multiple levels of [Quality of -Service](http://issue.k8s.io/168). +Kubernetes supports overcommitment of resources by supporting multiple levels of [Quality of Service](http://issue.k8s.io/168). Currently, one unit of CPU means different things on different cloud providers, and on different machine types within the same cloud providers. For example, on AWS, the capacity of a node diff --git a/docs/user-guide/production-pods.md b/docs/user-guide/production-pods.md index 75bf31edd356f..2d16dfc974d55 100644 --- a/docs/user-guide/production-pods.md +++ b/docs/user-guide/production-pods.md @@ -272,9 +272,14 @@ spec: cpu: 500m # memory units are bytes memory: 64Mi + requests: + # cpu units are cores + cpu: 500m + # memory units are bytes + memory: 64Mi ``` -The container will die due to OOM (out of memory) if it exceeds its specified limit, so specifying a value a little higher than expected generally improves reliability. +The container will die due to OOM (out of memory) if it exceeds its specified limit, so specifying a value a little higher than expected generally improves reliability. By specifying request, pod is guaranteed to be able to use that much of resource when needed. See [Resource QoS](../proposals/resource-qos.md) for the difference between resource limits and requests. If you’re not sure how much resources to request, you can first launch the application without specifying resources, and use [resource usage monitoring](monitoring.md) to determine appropriate values.