• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • Risks and Mitigations
• Design Details
  • Proposed Change
  • Test Plan
    • Prerequisite testing updates
    • Unit tests
    • Integration tests
    • e2e tests
  • Graduation Criteria
    • Alpha
    • Beta
    • GA
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
• Production Readiness Review Questionnaire
  • Feature Enablement and Rollback
  • Rollout, Upgrade and Rollback Planning
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Alternatives

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  • e2e Tests for all Beta API Operations (endpoints)
  • (R) Ensure GA e2e tests for meet requirements for Conformance Tests
  • (R) Minimum Two Week Window for GA e2e tests to prove flake free
• (R) Graduation criteria is in place
  • (R) all GA Endpoints must be hit by Conformance Tests
• (R) Production readiness review completed
• (R) Production readiness review approved
• "Implementation History" section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Starting with Kubernetes 1.22, a new CPUManager flag has facilitated the use of CPUManager Policy options(#2625) which enable users to customize their behavior based on workload requirements without having to introduce an entirely new policy. These policy options work together to ensure an optimized cpu set is allocated for workloads running on a cluster. The two policy options that already exist are full-pcpus-only(#2625) and distribute-cpus-across-numa (#2902). With this KEP, a new CPUManager policy option is introduced which ensures that all CPUs on a socket are considered to be aligned. Thus, the CPUManager will send a broader set of preferred hints to TopologyManager, enabling the increased likelihood of the best hint to be socket aligned with respect to CPU and other devices managed by DeviceManager.

With the evolution of CPU architectures, the number of NUMA nodes per socket has increased. The devices managed by DeviceManager may not be uniformly distributed across all NUMA nodes. Thus there can be scenarios where perfect alignment between devices and CPU may not be possible. Latency sensitive applications desire resources to be aligned at least within the same socket if NUMA alignment is not possible for optimal performance. By default, the CPUManager prefers CPU allocations which require a minimum number of NUMA nodes. However, if the NUMA nodes selected for allocation are spread across sockets, it results in degraded performance. By ensuring the selected NUMA nodes are socket aligned, predictable performance can be achieved. The best possible alignment of CPUs with other resources(viz. Which are managed by DeviceManager) is crucial to guarantee predictable performance for latency sensitive applications.

• Ensure CPUs are aligned at socket boundary rather than NUMA node boundary.

• Guarantee optimal NUMA allocation for cpu distribution.

We propose to add a new CPUManager policy option called align-by-socket to the static policy of CPUManager. With this policy option, the CPUManager will prefer those hints in which all CPUs are within same socket in addition to exisiting hints which require minimum NUMA nodes. With this policy option CPUs will be considered aligned at socket boundary instead of NUMA boundary during allocation. Thus if best hint consist of NUMA nodes within one socket, CPUManager may try to assign available CPUs from all NUMA nodes of socket.

The risks of adding this new feature are quite low. It is isolated to a specific policy option within the CPUManager, and is protected both by the option itself, as well as the CPUManagerPolicyAlphaOptions feature gate (which is disabled by default).

When align-by-socket is enabled as a policy option, the CPUManager’s function GetTopologyHints will prefer hints which are socket aligned in addition to hints which require minimum number of NUMA nodes.

To achieve this, the following updates are needed to the generateCPUTopologyHints function of static policy of CPUManager:

func (p *staticPolicy) generateCPUTopologyHints(availableCPUs cpuset.CPUSet, reusableCPUs cpuset.CPUSet, request int) []topologymanager.TopologyHint {
	...

    // Loop back through all hints and update the 'Preferred' field based on
    // counting the number of bits sets in the affinity mask and comparing it
    // to the minAffinitySize. Those with an equal number of bits set (and
    // with a minimal set of numa nodes) will be considered preferred.
    // If align-by-socket policy option is enabled, socket aligned hints are
    // also considered preferred.
    for i := range hints {
      if p.options.AlignBySocket && isSocketAligned(hints[i].NUMANodeAffinity) {
        hints[i].Preferred = true
        continue
      }
      if hints[i].NUMANodeAffinity.Count() == minAffinitySize {
        hints[i].Preferred = true
      }
    }

	return hints
}

At the end, we will have a list of desired hints. These hints will then be passed to the topology manager whose job it is to select the best hint (with an increased likelihood of selecting a hint that has CPUs which are aligned by socket now).

During CPU allocation, in function allocatedCPUs(), alignedCPUs will consist of CPUs which are socket aligned instead of CPUs from NUMA nodes in numaAffinity hint when align-by-socket policy option is enabled.

 func (p *staticPolicy) allocateCPUs(s state.State, numCPUs int, numaAffinity bitmask.BitMask, reusableCPUs cpuset.CPUSet) (cpuset.CPUSet, error) {
	...
     if numaAffinity != nil {
         alignedCPUs := cpuset.NewCPUSet()

         bits := numaAffinity.GetBits()
         // If align-by-socket policy option is enabled, NUMA based hint is expanded to
         // socket aligned hint. It will ensure that first socket aligned available CPUs are
         // allocated before we try to find CPUs across socket to satify allocation request.
         if p.PolicyOptions.AlignBySocket {
           bits = p.topology.ExpandToFullSocketBits(bits)
         }
         for _, numaNodeID := range bits {
           alignedCPUs = alignedCPUs.Union(assignableCPUs.Intersection(p.topology.CPUDetails.CPUsInNUMANodes(numaNodeID)))
         }

	 ...
}

This will ensure that for purpose of allocation, CPUs are considered aligned at socket boundary rather than NUMA boundary

align-by-socket policy options will work well for general case where number of NUMA nodes per socket are one or more. In rare cases like DualNumaMultiSocketPerNumaHT where one NUMA can span multiple socket, above option is not applicable. We will error out in cases when align-by-socket is enabled and underlying topology consist of multiple socket per NUMA. We may address such scenarios in future if there is a usecase for it in real world.

In cases where number of NUMA nodes per socket is one or more as well as TopologyManager single-numa-node policy is enabled, the policy option of align-by-socket is redundant since allocation guarantees within the same NUMA are by definition socket aligned. Hence, we will error out in case the policy option of align-by-socket is enabled along with TopologyManager single-numa-node policy.

The policyOption align-by-socket can work in conjunction with TopologyManager best-effort and restricted policy without any conflict.

[X] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

• k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/policy_static.go: 06-13-2022 - 91.1

• These cases will be added in the existing integration tests:
  • Feature gate enable/disable tests
  • align-by-socket policy option works as expected. When policy option is enabled
    • generateCPUTopologyHints prefers socket aligned hints in conjunction with hints with minimum NUMA nodes.
    • allocateCPUs allocated CPU at socket boundary.
  • Verify no significant performance degradation

• These cases will be added in the existing e2e tests:
  • Feature gate enable/disable tests
  • align-by-socket policy option works as expected.

• Implement the new policy option.
• Ensure proper unit tests are in place.
• Ensure proper e2e node tests are in place.

• Gather feedback from consumers of the new policy option.
• Verify no major bugs reported in the previous cycle.

• Allow time for feedback (1 year).
• Make sure all risks have been addressed.

We expect no impact. The new policy option is opt-in and orthogonal to the existing ones.

No changes needed

• Feature gate (also fill in values in kep.yaml)
  • Feature gate name: CPUManagerPolicyAlphaOptions
  • Components depending on the feature gate: kubelet
• Change the kubelet configuration to set a CPUManager policy of static and a CPUManager policy option of align-by-socket
  • Will enabling / disabling the feature require downtime of the control plane? No
  • Will enabling / disabling the feature require downtime or reprovisioning of a node? Yes -- a kubelet restart is required.

No. In order to trigger any of the new logic, three things have to be true:

1. The CPUManagerPolicyAlphaOptions feature gate must be enabled
2. The static CPUManager policy must be selected
3. The new align-by-socket policy option must be selected

Yes, the feature can be disabled by either:

1. Disabling the CPUManagerPolicyAlphaOptions feature gate
2. Switching the CPUManager policy to none
3. Removing align-by-socket from the list of CPUManager policy options

Existing workloads will continue to run uninterrupted, with any future workloads having their CPUs allocated according to the policy in place after the rollback.

No changes. Existing container will not see their allocation changed. New containers will.

• A specific e2e test will demonstrate that the default behaviour is preserved when the feature gate is disabled, or when the feature is not used (2 separate tests)

Inspect the kubelet configuration of a node -- check for the presence of the feature gate and usage of the new policy option.

In order to verify this feature is working, one should: Pick a node with at least 2 Sockets and multiple NUMA nodes per socket Ensure no other pods with exclusive CPUs are running on that node Launch a 2 pods with a nodeSelector to that node that has a single container in it Run a sleep infinity command and request exclusive CPUs for the container in the amount of (4*NUM_CPUS_PER_NUMA_NODE - 8) Verify that for both pods, all CPU’s are within same socket instead of cpu’s distributed across sockets

To verify the list of CPUs allocated to the container, one can either:

• exec into uthe container and run taskset -cp 1 (assuming this command is available in the container).
• Call the GetCPUS() method of the CPUProvider interface in the kubelet's podresources API.

There are no specific SLOs for this feature. Parallel workloads will benefit from this feature in application specific ways.

None

None

This feature is linux specific, and requires a version of CRI that includes the LinuxContainerResources.CpusetCpus field. This has been available since v1alpha2.

No

No

No

No

The algorithm required to implement this feature could delay:

1. Pod admission time
2. The time it takes to launch each container after pod admission

This delay should be minimal.

No, the algorithm will run on a single goroutine with minimal memory requirements.

• 2022-06-02: Initial KEP created
• 2022-06-08: Addressed review comments on KEP

align-by-socket policy option when enabled, it might result CPU allocation which may not be perfectly aligned by NUMA with other resources managed by DeviceManager.

align-by-socket can alternatively be introduced as policy of TopologyManager which can choose hints from hint provider which are socket aligned. However, it is concluded that fuction of TopologyManager is to act as arbitrator for hints provider and should make decisions based on hints provided rather than influencing the BestHint based on its own policy.