Would it make sense to add a reference to the PR that added swap support in the stats summary? kubernetes/kubernetes#118865

Although, I see that section only has links to issues, not PRs.

This section does need some love. I don't really know the dates but I guess I can find PRs some I can fill this in.

I updated the section with what I could find. There are alot of PRs and I decided to focus on promotion to beta1. PTAL.

+1000

Anything for the debuggability enhancement since enabling the feature might affect the performance of the node. I know we introduced swap metrics, but what about kubectl describe node?

FWIW we are still exploring what the best way of monitoring swap is. If it's direct metrics - swap traffic, or indirect, such as *wait times.

I'll put this as something to look into @dchen1107! The issue with describe node is that I think it gives capacity for swap (ie how much swap the node actually has) but a user may get confused because they are only allowed to use up to computed limit on swap for Burstable QoS. So its a bit misleading to add a capacity when you can't really use it.

I think we should add some way to detect if swap is enabled on a node. I talked with nfd team about potentially adding a way to detect if swap is on a node so we can add useful labels for nodes for swap.

I think the node-level swap capacity would still be a useful information even if the pods may not be able to consume it. That's the same for other resources on the node too.

Q: Is there a good way to expose per-pod swap usage?

A quick way to examine whether system-critical daemons are using swap would also be helpful

A quick way to examine whether system-critical daemons are using swap would also be helpful

I don't have a great way to do this. @fabiand any ideas here?

I would look at free and then see what is using swap.

I think the node-level swap capacity would still be a useful information even if the pods may not be able to consume it. That's the same for other resources on the node too.

I've been asked that a few times so I think we can add this to the node information when doing kubectl describe node.

Q: Is there a good way to expose per-pod swap usage?
I was actually thinking that a crictl stats or something like that may be useful here.

I think one could view the summary api using kubectl --raw and view the swap stats.

It could maybe be useful to expose information at a lower level tool if that is of interest.

We would look into the container and view memory.swap.current to see if swap was being used. The scoping was a bit confusing to me but exec into the pod and viewing the memory.swap.current seems to be the best way to get swap information.

@iholder101 any other ways you found useful to look at per-pod swap usage?

I've been asked that a few times so I think we can add this to the node information when doing kubectl describe node

I think it makes sense. I'll get to it.

@iholder101 any other ways you found useful to look at per-pod swap usage?

What about using kubectl top to get this information? I can dive deeper into it.

A quick way to examine whether system-critical daemons are using swap would also be helpful

Critical daemons can be provided in two areas

1. node level (part of os)
2. as containers

(1) is easy to monitor: system.slice contains all these services. system.slice should also be configured to not use swap (part of the kep)
(2) is more difficult i.e. rook or other CNI/CSI projects will have critical pods.

Monitoring and configuration of those is more difficult.

If we assume that pods are correctly assigned to PriorityClasses, then monitoring can be improve by displaying swap consumption by priority class.
Further - unsure about it - we could consider to implictily control swap for workloads based on priority class. I.e. disable for node- and cluster-critical workloads.

Addition thoughts:
I've often seen that pods from (2) do run as burstable, making them eligible to swap.
We either need to find a way to (a) let pods opt-out of swap independent of the qos class or (b) make pods to opt-in to swap - regardless of the qos class (burstable pods can also benefit from swap as a safety mechanism).

The broad decision we should think about is: Do we want opt-in or opt-out for workloads.

I've been asked that a few times so I think we can add this to the node information when doing kubectl describe node

@iholder101 +1 in adding to kubelet and then kubectl. Let's include two metrics: One around amount of swap, and the other on amount of swap traffic. The latter is worse than the first.