inconsistent label cardinality when trying to report metrics #1634
Closed
Description
Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT
What happened:
When enabling metrics config on the workflow controller, the controller crashes trying to create metrics:
enabled: true
path: /metrics
port: 8080time="2019-10-02T15:32:00Z" level=info msg="Processing workflow" namespace=ddseapy workflow=hello-world-hflln
panic: inconsistent label cardinality
goroutine 303 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(0xc0001f97a0, 0x2, 0x0, 0xc0008ead40, 0x3, 0x4, 0x1, 0x1)
/go/src/github.com/prometheus/client_golang/prometheus/value.go:172 +0xb0
github.com/argoproj/argo/workflow/metrics.(*workflowCollector).collectWorkflow.func1(0xc0001f97a0, 0x2, 0x0, 0xc000842760, 0x1, 0x1)
/go/src/github.com/argoproj/argo/workflow/metrics/collector.go:100 +0x16d
github.com/argoproj/argo/workflow/metrics.(*workflowCollector).collectWorkflow.func2(0xc0001f97a0, 0x0, 0xc000842760, 0x1, 0x1)
/go/src/github.com/argoproj/argo/workflow/metrics/collector.go:103 +0x62
github.com/argoproj/argo/workflow/metrics.(*workflowCollector).collectWorkflow(0xc0007f6ff0, 0xc00094c540, 0xc000792638, 0x8, 0xc000176440, 0x14, 0xc000176560, 0x11, 0xc000792670, 0xc, ...)
/go/src/github.com/argoproj/argo/workflow/metrics/collector.go:115 +0x248
github.com/argoproj/argo/workflow/metrics.(*workflowCollector).Collect(0xc0007f6ff0, 0xc00094c540)
/go/src/github.com/argoproj/argo/workflow/metrics/collector.go:93 +0xc3
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc0006aa0b0, 0xc00094c540, 0x1891f40, 0xc0007f6ff0)
/go/src/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x61
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
/go/src/github.com/prometheus/client_golang/prometheus/registry.go:431 +0x302
What you expected to happen:
No error should occur when metrics are enabled.
How to reproduce it (as minimally and precisely as possible):
argo submit the helloworld example:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: hello-world-
spec:
entrypoint: whalesay
templates:
- name: whalesay
container:
image: docker/whalesay:latest
command: [cowsay]
args: ["hello world"]Environment:
- Argo version:
latest - Kubernetes version :
v1.13.1
Metadata
Assignees
Labels
No labels