Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

how to configure multi alertmanagers? #2891

Closed
happywzy opened this issue Nov 29, 2019 · 20 comments
Closed

how to configure multi alertmanagers? #2891

happywzy opened this issue Nov 29, 2019 · 20 comments
Labels
kind/support

Comments

@happywzy
Copy link

happywzy commented Nov 29, 2019 •
edited
Loading

Prometheus Operator version: v0.34.0

Kubernetes cluster kind: kubeadm

hello, l have deploy a prometheus-operator in k8s with "helm install stable/prometheus-operator", now i have some question about prometheus config.

in default, install prometheus-operator with helm will start a prometheus named prometheus-operator-prometheus, a alertmanager named prometheus-operator-alertmanager , and so on..(I have configuration the ingress to access alert manager UI)
image

when the prometheus has a new alert , it will send to the prometheus-operator-alertmanager and i can see it in the alertmanager dashboard.
image
and i have cat the config of prometheus,
image
now I want test create a new alertmanager named prometheus-operator-alertmanager-test with the CRD Alertmanager , and the new alertmanager configuration is the same as prometheus-operator-alertmanager , I have just change the name.
image

and i have create a service type of NodePort, to prove the new Alertmanager is ok!

At last , I have modify the configuration of prometheus-operator-prometheus ,as following:
image
and it worked, I have checked in the dashboard of prometheus-Configuration.

now, Something strange happened.
when the prometheus has produce new alert, prometheus-operator-prometheus can't receive,only prometheus-operator-alertmanager-test can receive! why ??please help me.Looking forward to your reply. Thank you!
@happywzy
Copy link
Author

image

@happywzy happywzy changed the title how to config multi alertmanagers? how to configure multi alertmanagers? Dec 2, 2019
@brancz
Copy link
Contributor

brancz commented Dec 4, 2019

Can you check under the /status endpoint on your Prometheus which Alertmanagers were actually discovered? Generally speaking Prometheus should be sending the alerts to all discovered Alertmanagers unconditionally.

@happywzy
Copy link
Author

happywzy commented Dec 4, 2019

Can you check under the /status endpoint on your Prometheus which Alertmanagers were actually discovered? Generally speaking Prometheus should be sending the alerts to all discovered Alertmanagers unconditionally.

Only have one endpoints.
image
but I have configure two alertmanagers in the Prometheus.
why doesn't it worked?

@brancz
Copy link
Contributor

brancz commented Dec 4, 2019 •
edited
Loading

Can you share the output of kubectl -n monitoring get endpoints? That's where Prometheus ultimately discovers the Alertmanager instances from.

@happywzy
Copy link
Author

happywzy commented Dec 4, 2019

image

@brancz
Copy link
Contributor

brancz commented Dec 4, 2019

hmm ... can you share the yaml representation? kubectl -n monitoring get prometheus-operator-alertmanager-test -oyaml and kubectl -n monitoring get prometheus-operator-alertmanager -oyaml.

So far it looks like you're doing everything right. Which version of Prometheus are you running?

@happywzy
Copy link
Author

happywzy commented Dec 4, 2019

hmm ... can you share the yaml representation? kubectl -n monitoring get prometheus-operator-alertmanager-test -oyaml and kubectl -n monitoring get prometheus-operator-alertmanager -oyaml.

So far it looks like you're doing everything right. Which version of Prometheus are you running?

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  creationTimestamp: "2019-11-29T02:04:58Z"
  generation: 1
  labels:
    app: prometheus-operator-alertmanager
    chart: prometheus-operator-8.2.4
    heritage: Tiller
    release: prometheus-operator
  name: prometheus-operator-alertmanager-test
  namespace: monitoring
  resourceVersion: "6496244"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/alertmanagers/prometheus-operator-alertmanager-test
  uid: 2de06863-462d-4980-9e7f-063ed80c403a
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              alertmanager: prometheus-operator-alertmanager-test
              app: alertmanager
          topologyKey: kubernetes.io/hostname
        weight: 100
  baseImage: quay.io/prometheus/alertmanager
  portName: web
  replicas: 1
  retention: 120h
  routePrefix: /
  serviceAccountName: prometheus-operator-alertmanager
  storage:
    volumeClaimTemplate:
      selector: {}
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: nfs-client
  version: v0.19.0
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  creationTimestamp: "2019-11-28T02:42:48Z"
  generation: 1
  labels:
    app: prometheus-operator-alertmanager
    chart: prometheus-operator-8.2.4
    heritage: Tiller
    release: prometheus-operator
  name: prometheus-operator-alertmanager
  namespace: monitoring
  resourceVersion: "6316431"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/alertmanagers/prometheus-operator-alertmanager
  uid: 65b64268-259e-4f33-8d32-26f2ea78ac21
spec:
  baseImage: quay.io/prometheus/alertmanager
  externalUrl: http://alert.deri.com/
  listenLocal: false
  logFormat: logfmt
  logLevel: info
  paused: false
  portName: web
  replicas: 1
  retention: 120h
  routePrefix: /
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-operator-alertmanager
  storage:
    volumeClaimTemplate:
      selector: {}
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: nfs-client
  version: v0.19.0

Prometheus Version :v2.13.1

@brancz
Copy link
Contributor

brancz commented Dec 4, 2019

Sorry I meant the endpoints, not alertmanagers.

@happywzy
Copy link
Author

happywzy commented Dec 4, 2019

Sorry I meant the endpoints, not alertmanagers.

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2019-11-28T02:43:02Z"
  creationTimestamp: "2019-11-28T02:42:48Z"
  labels:
    app: prometheus-operator-alertmanager
    chart: prometheus-operator-8.2.4
    heritage: Tiller
    release: prometheus-operator
  name: prometheus-operator-alertmanager
  namespace: monitoring
  resourceVersion: "6316596"
  selfLink: /api/v1/namespaces/monitoring/endpoints/prometheus-operator-alertmanager
  uid: 07b76b88-9d2e-4763-991c-b3b25fa8eb4e
subsets:
- addresses:
  - ip: 10.244.3.227
    nodeName: k8s-02
    targetRef:
      kind: Pod
      name: alertmanager-prometheus-operator-alertmanager-0
      namespace: monitoring
      resourceVersion: "6316593"
      uid: 9de9bdb9-095c-4d10-aa91-267314c812f5
  ports:
  - name: web
    port: 9093
    protocol: TCP
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2019-11-29T02:05:11Z"
  creationTimestamp: "2019-11-29T02:05:11Z"
  labels:
    app: prometheus-operator-alertmanager
    chart: prometheus-operator-8.2.4
    heritage: Tiller
    release: prometheus-operator
  name: prometheus-operator-alertmanager-test
  namespace: monitoring
  resourceVersion: "6496300"
  selfLink: /api/v1/namespaces/monitoring/endpoints/prometheus-operator-alertmanager-test
  uid: ae0edab1-0f84-45cc-bec3-93924691f169
subsets:
- addresses:
  - ip: 10.244.3.232
    nodeName: k8s-02
    targetRef:
      kind: Pod
      name: alertmanager-prometheus-operator-alertmanager-test-0
      namespace: monitoring
      resourceVersion: "6496271"
      uid: 3ae292be-24be-4146-8596-cca226223ed1
  ports:
  - name: web
    port: 9093
    protocol: TCP

@brancz
Copy link
Contributor

brancz commented Dec 4, 2019

Still all looks correct. Could you share the whole config in the prometheus-k8s secret?

@happywzy
Copy link
Author

happywzy commented Dec 4, 2019

config

[root@k8s-master ~]# kubectl get secret -n monitoring
NAME                                                       TYPE                                  DATA   AGE
alertmanager-prometheus-operator-alertmanager              Opaque                                2      30h
alertmanager-prometheus-operator-alertmanager-test         Opaque                                2      30h
default-token-bqmbl                                        kubernetes.io/service-account-token   3      6d9h
mysql-exporter-prometheus-mysql-exporter                   Opaque                                1      6d
prometheus-operator-admission                              Opaque                                3      6d9h
prometheus-operator-alertmanager-token-k26qk               kubernetes.io/service-account-token   3      6d9h
prometheus-operator-grafana                                Opaque                                3      6d9h
prometheus-operator-grafana-test-token-bn9fn               kubernetes.io/service-account-token   3      6d9h
prometheus-operator-grafana-token-qrb9h                    kubernetes.io/service-account-token   3      6d9h
prometheus-operator-kube-state-metrics-token-nl72j         kubernetes.io/service-account-token   3      6d9h
prometheus-operator-operator-token-prz7w                   kubernetes.io/service-account-token   3      6d9h
prometheus-operator-prometheus-node-exporter-token-ks5cp   kubernetes.io/service-account-token   3      6d9h
prometheus-operator-prometheus-token-cnjsl                 kubernetes.io/service-account-token   3      6d9h
prometheus-prometheus-operator-prometheus                  Opaque                                1      6d9h
prometheus-prometheus-operator-prometheus-tls-assets       Opaque                                0      6d9h

which you need?
Secret:prometheus-prometheus-operator-prometheus,it looks like this in dashboard,
image
I cat it in the contanier prometheus-prometheus-operator-prometheus-0, path /etc/prometheus/config_out

global:
  evaluation_interval: 30s
  scrape_interval: 30s
  external_labels:
    prometheus: monitoring/prometheus-operator-prometheus
    prometheus_replica: prometheus-prometheus-operator-prometheus-0
rule_files:
- /etc/prometheus/rules/prometheus-prometheus-operator-prometheus-rulefiles-0/*.yaml
scrape_configs:
- job_name: monitoring/prometheus-operator-alertmanager/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - monitoring
  metrics_path: /metrics
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-operator-alertmanager
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: web
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: web
- job_name: monitoring/prometheus-operator-apiserver/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - default
  scheme: https
  tls_config:
    insecure_skip_verify: false
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server_name: kubernetes
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_component
    regex: apiserver
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_provider
    regex: kubernetes
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: https
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_component
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: https
- job_name: monitoring/prometheus-operator-coredns/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-operator-coredns
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobLabel
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http-metrics
- job_name: monitoring/prometheus-operator-grafana/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - monitoring
  metrics_path: /metrics
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: grafana
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: service
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: service
- job_name: monitoring/prometheus-operator-kube-controller-manager/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-operator-kube-controller-manager
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobLabel
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http-metrics
- job_name: monitoring/prometheus-operator-kube-etcd/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-operator-kube-etcd
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobLabel
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http-metrics
- job_name: monitoring/prometheus-operator-kube-proxy/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-operator-kube-proxy
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobLabel
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http-metrics
- job_name: monitoring/prometheus-operator-kube-scheduler/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-operator-kube-scheduler
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobLabel
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http-metrics
- job_name: monitoring/prometheus-operator-kube-state-metrics/0
  honor_labels: true
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - monitoring
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app_kubernetes_io_instance
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app_kubernetes_io_name
    regex: kube-state-metrics
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_app_kubernetes_io_name
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http
- job_name: monitoring/prometheus-operator-kubelet/0
  honor_labels: true
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  scheme: https
  tls_config:
    insecure_skip_verify: true
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_k8s_app
    regex: kubelet
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: https-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_k8s_app
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: https-metrics
- job_name: monitoring/prometheus-operator-kubelet/1
  honor_labels: true
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  metrics_path: /metrics/cadvisor
  scheme: https
  tls_config:
    insecure_skip_verify: true
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_k8s_app
    regex: kubelet
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: https-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_k8s_app
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: https-metrics
- job_name: monitoring/prometheus-operator-node-exporter/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - monitoring
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-node-exporter
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobLabel
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: metrics
- job_name: monitoring/prometheus-operator-operator/0
  honor_labels: true
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - monitoring
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-operator-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: http
- job_name: monitoring/prometheus-operator-prometheus/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - monitoring
  metrics_path: /metrics
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: prometheus-operator-prometheus
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: prometheus-operator
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_self_monitor
    regex: "true"
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: web
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: web
alerting:
  alert_relabel_configs:
  - action: labeldrop
    regex: prometheus_replica
  alertmanagers:
  - path_prefix: /
    scheme: http
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - monitoring
    relabel_configs:
    - action: keep
      source_labels:
      - __meta_kubernetes_service_name
      regex: prometheus-operator-alertmanager
    - action: keep
      source_labels:
      - __meta_kubernetes_endpoint_port_name
      regex: web
  - path_prefix: /
    scheme: http
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - monitoring
    relabel_configs:
    - action: keep
      source_labels:
      - __meta_kubernetes_service_name
      regex: prometheus-operator-alertmanager-test
    - action: keep
      source_labels:
      - __meta_kubernetes_endpoint_port_name
      regex: web

@simonpasquier
Copy link
Contributor

I'm looking into this. There might be an issue indeed.

@simonpasquier simonpasquier mentioned this issue Dec 12, 2019
Merged
@simonpasquier
Copy link
Contributor

It's a problem with Prometheus upstream. I've submitted prometheus/prometheus#6455 to fix it.

@happywzy
Copy link
Author

It's a problem with Prometheus upstream. I've submitted prometheus/prometheus#6455 to fix it.

ok,thank you.

@raqqun
Copy link

raqqun commented Jan 3, 2020

is there any temporary solution for this problem ? Is the fix prometheus/prometheus#6455 in any production version so that we update our images ?

@brancz
Copy link
Contributor

brancz commented Jan 6, 2020

If I am not mistaken the fix should be available in Prometheus 2.15.

@raqqun
Copy link

raqqun commented Jan 6, 2020

@brancz I figure that out. Thanks for replying.

@mattharr-is
Copy link

Sorry to revive this closed ticket, but i was wondering if you guys have any documentation on using multiple alertmanagers with the prometheus operator. do all alertmanager instances need to exist in the monitoring namespace? Are they automatically discovered by prometheus, regardless of where they run? Are there other steps to wire up discovery of additional alertmanagers besides just applying an Alertmanager resource to the cluster? Currently, i'm not able to discover any alertmanager instances outside of the one installed by p8s itself. I am using prometheus version 2.15

@shabbir-lb
Copy link

shabbir-lb commented Jun 30, 2023 •
edited
Loading

Sorry to revive this closed ticket, but i was wondering if you guys have any documentation on using multiple alertmanagers with the prometheus operator. do all alertmanager instances need to exist in the monitoring namespace? Are they automatically discovered by prometheus, regardless of where they run? Are there other steps to wire up discovery of additional alertmanagers besides just applying an Alertmanager resource to the cluster? Currently, i'm not able to discover any alertmanager instances outside of the one installed by p8s itself. I am using prometheus version 2.15

Hey. I just wanted to ask if you did afterall find any documentation or resources for configuring alertmanager using name and namespaces. Because i was trying to edit prometheus config using static_configs which does not work for prometheus operator helm chart.

So now I am trying to specify the name and namespace of the alertmanagers in the prometheus configuration but prometheus does not recognize them and does not find endpoints.

@shabbir-lb
Copy link

shabbir-lb commented Jun 30, 2023 •
edited
Loading

Can you please tell me what steps you followed to create the 2nd alertmanager ??
The one named prometheus-operator-alertmanager-test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@mattharr-is @simonpasquier @raqqun @brancz @happywzy @shabbir-lb