I'm not super familiar with kubeadm, but the change in 1.7.3 was unrelated to the cAdvisor metric endpoint exposed on port 4194, the metrics were only removed from the /metric endpoint of the kubelet and moved to /metric/cadvisor.

I think in terms of this issue this was actually a firewall issue, that your worker that runs the Prometheus pod doesn't have network access to the kubelet 4194 port.

@brancz I've dug into the actual kubelet configuration and you're right: the default kubeadm configuration disables the cAdvisor port (with the --cadvisor-port=0 flag).

Actually I've just found the original thread on the Prometheus repository, which already covered this explanation.

Personally, I'm not very willing to change the kubeadm settings and redeploy the cluster just to enable that port, while there is already another endpoint working (and changing a manifest is enough to exploit it). But that's just me 😄
Feel free to decide whether the alternative is worth being included or not; I can use the workaround for now (and perhaps other people might find this useful too).

metoo, with kube-aws. seems like eventually scraping the /metrics/cadvisor endpoint will be preferred? We'll maintain a fork till this is resolved (see commit pingback)

We'll probably move to the /metrics/cadvisor endpoint when we upgrade everything to be target 1.8.0+.

Is it meant to be adding in a kubelet/4194 target by default?

We have: k8s 1.7.7 built with bootkube
prometheus-operator 0.14.0 (not meddled with the versions, to prometheus 2.0.0-rc1)

We, as the original poster described, get kubelet metrics but no cadvisor metrics. Checking prometheus' targets, its scraping :10250 as per normal.

I don't really mind whether it would get cadvisor metrics from 4194/metrics or 10250/metrics/cadvisor - but how do we get it to do... either?

It's clear from the kubernetes issues that they don't really consider breaking them out and making the separate a big thing - so they're unlikely to put them back into 10250/metrics. For people hand-rolling their prometheus config it's not too difficult, but when operator is doing it, we lack the control to add things in (one day, prometheus will get the hang of multiple config sources).

So, what have people who use prometheus-operator been doing to get those metrics back?

Can you explain more what you think is not working today? I’m saying this because I think all combinations are possible today, but possibly not well enough documented.

If one wants to use the 4194/metrics endpoint then that’s already reflected in the kube-prometheus manifests.

If one wants to use the 10250/metrics/cadvisor metrics then one has to modify the servicemonitor endpoint provided in kube-prometheus and set the explicit metrics endpoint.

Does that clear things up?

@ghostflame To use the 10250/metrics/cadvisor endpoint, it should be enough to edit the kube-prometheus/manifests/prometheus/prometheus-k8s-service-monitor-kubelet.yaml manifest the way I reported it in the issue description:

change

- port: cadvisor
    interval: 30s
    honorLabels: true

to

- path: /metrics/cadvisor
    port: http-metrics
    interval: 30s
    honorLabels: true

and redeploy.

Hope this helps...

In my case, I just let 4194 been listened, and they worked.

--cadvisor-port=0 disables cAdvisor from listening to 0.0.0.0:4194 by default. cAdvisor will still be run inside of the kubelet and its API can be accessed at https://{node-ip}:10250/stats/. If you want to enable cAdvisor to listen on a wide-open port, run:

sed -e "/cadvisor-port=0/d" -i /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl restart kubelet

Actually CIS requires cadvisor-port=0, so I recommend everyone to use the /metrics/cadvisor path. I'll leave this open as we need to fix it for kube-prometheus.

@lorenzo-biava
as you have mentioned the correct servicemonitor configs

- path: /metrics/cadvisor
    port: http-metrics
    interval: 30s
    honorLabels: true

or

spec:
  jobLabel: k8s-app
  endpoints:
  - port: http-metrics
    interval: 30s
  # This is for cAdvisor in K8s 1.7.3+
  - path: /metrics/cadvisor
    port: http-metrics
    interval: 30s
    honorLabels: true

I did not get it work, the alarm manager still reports k8skubeletdown.
Could you give some hints?

My setup is k8s v1.9.2 and master branch of prometheus-operator as of today.

[root@master1 kube-prometheus]# k describe svc kubelet
Name:              kubelet
Namespace:         kube-system
Labels:            k8s-app=kubelet
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                None
Port:              https-metrics  10250/TCP
TargetPort:        10250/TCP
Endpoints:         192.168.50.57:10250,192.168.50.58:10250,192.168.50.59:10250 + 8 more...
Session Affinity:  None
Events:            <none>

[root@master1 kube-prometheus]# k get svc kubelet -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-02-09T10:09:39Z
  labels:
    k8s-app: kubelet
  name: kubelet
  namespace: kube-system
  resourceVersion: "446305"
  selfLink: /api/v1/namespaces/kube-system/services/kubelet
  uid: 57005c64-0d81-11e8-91f9-005056a3367f
spec:
  clusterIP: None
  ports:
  - name: https-metrics
    port: 10250
    protocol: TCP
    targetPort: 10250
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

[root@master1 kube-prometheus]# cat ./manifests/prometheus/prometheus-k8s-service-monitor-kubelet.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubelet
  labels:
    k8s-app: kubelet
spec:
  jobLabel: k8s-app
  endpoints:
  #- port: https-metrics
  #  scheme: https
  #  interval: 30s
  #  tlsConfig:
  #    insecureSkipVerify: true
  #  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  - port: https-metrics
    scheme: https
    path: /metrics/cadvisor
    interval: 30s
    honorLabels: true
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  selector:
    matchLabels:
      k8s-app: kubelet
  namespaceSelector:
    matchNames:
    - kube-system

to resolve the issue in my case above, just do below on all nodes, according to https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/docs/kube-prometheus-on-kubeadm.md

sed -e "s/--authorization-mode=Webhook/--authentication-token-webhook=true --authorization-mode=Webhook/" -i /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload; systemctl restart kubelet

@lzbgt thks

@lzbgt Sorry I've missed this; I've not been working with Kubernetes and Prometheus Operator lately, but it's definitely good to know that there's a dedicated doc for kubeadm configurations now. Thanks for pointing that out 😉

Closing as sample configurations are available and specific documentation for certain platforms as well.

Thanks @lorenzo-biava This I work in RKE cluster