cc @kubernetes/rh-cluster-infra

The issue is filed based on several offline design discussions including several attempts of improving Kubelet Metrics API made by @timstclair (#15691, #15862, #16621, #17708, #18544, ...).

cc/ @erictune

if we're going forward with collecting the non-core metrics using a separate cadvisor pod, how will the metrics information be exposed to collectors such as heapster? Will kubelet provide a directory service that helps heapster find where cadvisor pod can be accessed or will kubelet act as a middle-man (similar to the way the code works today) that gets metrics from cadvisor and servicing them out to heapster?

Heapster will have to get metrics from cAdvisor directly. Heapster can
query the IP:Port information for all the cAdvisor's in the
cluster(essentially one per node), and can collect metrics using cAdvisor's
IP directly.

On Fri, Dec 18, 2015 at 2:58 PM, Hai Huang notifications@github.com wrote:

if we're going forward with collecting the non-core metrics using a
separate cadvisor pod, how will the metrics information be exposed to
collectors such as heapster? Will kubelet provide a directory service that
helps heapster find where cadvisor pod can be accessed or will kubelet act
as a middle-man (similar to the way the code works today) that gets metrics
from cadvisor and servicing them out to heapster?

—
Reply to this email directly or view it on GitHub
#18770 (comment)
.

So are you expecting cadvisor pod to be a daemon set?

Yes

On Fri, Dec 18, 2015 at 3:07 PM, Jimmi Dyson notifications@github.com
wrote:

So are you expecting cadvisor pod to be a daemon set?

—
Reply to this email directly or view it on GitHub
#18770 (comment)
.

How will heapster find the port that cadvisor listens to? If each cadvisor is running in a separate pod, it could be mapped to different host port on different host.

Host port is not required. Remember Pods can talk to each other directly ;)

On Fri, Dec 18, 2015 at 4:45 PM, Hai Huang notifications@github.com wrote:

How will heapster find the port that cadvisor listens to? If each cadvisor
is running in a separate pod, it could be mapped to different host port on
different host.

—
Reply to this email directly or view it on GitHub
#18770 (comment)
.

OK, I just saw that you're expecting cadvisor pods to be running as a daemon set, so the host ips are known to the heapster pod collecting the metrics information. Reading how to communicate with a daemon set, I found some information here: https://github.com/kubernetes/kubernetes/blob/master/docs/admin/daemons.md#communicating-with-daemonset-pods, and it says

Clients knows the the list of nodes ips somehow, and know the port by convention.

I take it this means cadvisor will need to listen to the same port on every host, e.g., 4194, so whoever is communicating with it knows how to find it. Or it has to register with DNS.

I was not suggesting using services. I was instead suggesting using the
Pod IPs directly.

How will the consumer of such metrics get the list of Pod IPs of cadvisor Pods? Sorry if I missed this being already described somewhere.

I like the idea of making cAdvisor standalone, but I have a few questions:

1. Could cAdvisor be running not as a pod, e.g., standalone of kubelet?
2. If above question is true, how could heapster find out the list of those cAdvisor apis?

Hi, @feiskyer , Some of my opinions

Could cAdvisor be running not as a pod, e.g., standalone of kubelet?

In my opinion, cAdvisor running as a pod(maybe with daemon set) is more better than running standalone. If a daemon pod is killed or stopped, the DaemonSet will create a new replica of the daemon pod on the node. This pod can be managed by kubernetes.
In addition, When a new node is added to the cluster, the DaemonSet controller will start daemon pods on the node. So, cAdvisor will be add on the node automatically.

If above question is true, how could heapster find out the list of those cAdvisor apis?

Maybe, we can get the cAdvisor list by "get" and "describe" commands of "DaemonSet "

@huangyuqi For some container runtimes, e.g. Hyper, cAdvisor must be running outside of a pod.

/cc @kubernetes/sig-scalability @mattf @abhgupta

*this is exactly what I've observed in #19341

Is this deployment change expected for 1.2?

IMHO I would like to do this, if only to cap the CPU usage of the pod for maximum density.

In #18769 I'm trying to make the metrics/health collector (currently using a built-in cadvisor) a pluggable interface, which allows users to choose if k8s should use cadvisor running in build-in mode, pod mode, or standalone mode. Furthermore, one could also plug in one's favorite monitoring tool, other than cadvisor. That should complement this really well.

@feiskyer:

Could cAdvisor be running not as a pod, e.g., standalone of kubelet?

Yeah. The idea is to run it as an optional DaemonSet.

If above question is true, how could heapster find out the list of those cAdvisor apis?

cAdvisor's APIs are not discoverable like that of Kubernetes APIs. We intend to fix that. Until then, heapster will be hardcoded to query specific cAdvisor APIs.

@timothysc: I assume that cAdvisor will be run by default on all OpenShift clusters. If thats true, it is important to optimize not only the kubelet, but cAdvisor as well. Separating them will help us with performance measurements, but otherwise the work that needs to go in for optimizing resource consumption has to happen anyways.

On hyper's use case, we have to provide a way to run cAdvisor as a daemon on the host node, similar to Kubelet.

@dchen1107 to be sure I understand, you're saying that we should make it possible to run directly on the host node, but it's still okay if the default way to run it is as a pod, as discussed above in this issue?

As an aside, I suspect that node-problem-detector and kube-proxy, which are both currently run as privleged pods, also will not work that way with Hyper.

We could avoid making Heapster handle two sources if we say that the non-core monitoring agent must supply a super-set of the metrics provided by core cAdvisor (either by fetching from core cAdvisor and merging with its own metrics, or something else).

xref: #27097 (comment)

re @davidopp's comment: Yes, non-core monitoring agent should supply the super-set of the metrics provided by core cAdvisor which is defined at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go

re @euank's comment: #18770 (comment)

What I mentioned above of running cAdvisor directly on the host node is an intermediate workaround to support hyperContainer before we prioritized today's container runtime effort.

But now the story is changed, especially at #27097 (comment), I tried to re-define what the core metrics is: node-level and pod-level. If we agreed upon that (I think we are largely convergent already), I think running containers within a pod in a hyper or as native pod on the host node is punted to hyper runtime implementation, neither part of kubelet core system, nor cAdvisor config.

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close