-
Notifications
You must be signed in to change notification settings - Fork 40k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Limit the potentially unbounded list of images in the node status #23355
Comments
If the reason we put the images in the node status was just for the scheduler, we probably don't need it at all. Scheduler can just watch which pods get bound to which nodes, and keep the mapping from nodes to images in its memory. Kubelet would just need to somehow report when it GCs images so scheduler can update its information. When scheduler restarts it would lose all the information, but putting pods on nodes that already have the image installed is a best-effort feature anyway. If there was some additional reason we were reporting images that I have forgotten, then never mind. |
I believe it was added for image-affinity scheduling, and only that.
This'd make things more complicated than necessary. If we're aiming for best-effort only, scheduler can construct the list of images by looking at only running pods (which may be too limited?). |
@davidopp putting pods on nodes with images installed is the only use case so far. But some users want that feature support is not best-effort since they pre-cached the images to the node for speed up the container startup. |
The suggestion in the first entry in this issue (sort and drop the smaller packages from the report) also makes the feature best-effort. |
Yes, I guess one argument will be that users usually pre-pull if image size is significant, and scheduler will disregard this "optimization* by looking at pod mappings only. BTW, it was just a random idea i threw out... |
Fixed by #25328 |
Forked from #23323 (comment)
In short, kubelet reports the list of images on the node as part of the node status. The kubelet image GC kicks in only when the disk space hits a threshold, so the list can grow very long with many small images.
@lavalamp mentioned that this could be a problem for the apiserver, and we should handle it somehow.
A naive solution (off the top of my head): kubelet sorts the images by size and report only the top N images.
/cc @davidopp
The text was updated successfully, but these errors were encountered: