Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a globing operator in top-line metrics dashboard for the "all" value #4057

Merged
merged 2 commits into from
Mar 10, 2020

Conversation

mmiller1
Copy link
Contributor

Use a globing operator in top-line metrics dashboard for the "all" value

In clusters with many namespaces and deployments, the grafana query to prometheus for "all" may be too large for prometheus to return results.

Instead of having grafana query prometheus for every namespace/deployment literally, we can use a globing operator to shorten the request length so results can be returned in clusters where there are large numbers of deployments and namespaces.

Signed-off-by: Matt Miller <mamiller@rosettastone.com>
Copy link
Member

@alpeb alpeb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mmiller1, this works great! 👍

Before:

sum(irate(response_total{classification="success", deployment=~"emoji|linkerd-controller|linkerd-destination|linkerd-grafana|linkerd-identity|linkerd-prometheus|linkerd-proxy-injector|linkerd-sp-validator|linkerd-tap|linkerd-web|vote-bot|voting|web"}[30s])) / sum(irate(response_total{deployment=~"emoji|linkerd-controller|linkerd-destination|linkerd-grafana|linkerd-identity|linkerd-prometheus|linkerd-proxy-injector|linkerd-sp-validator|linkerd-tap|linkerd-web|vote-bot|voting|web"}[30s]))

After:

sum(irate(response_total{classification="success", deployment=~".*"}[30s])) / sum(irate(response_total{deployment=~".*"}[30s]))

@ihcsim
Copy link
Contributor

ihcsim commented Feb 28, 2020

Cool! Does this change correspond to what is documented here under Custom all value? Just want to better understand the context around this change.

There are a number of other places that still use "allValue": null. How do you feel about updating them too?

grep -rniI allValue grafana                                                                    
grafana/dashboards/replicaset.json:2211:          "allValue": null,
grafana/dashboards/replicaset.json:2236:          "allValue": null,
grafana/dashboards/replicaset.json:2261:          "allValue": null,
grafana/dashboards/replicaset.json:2286:          "allValue": null,
grafana/dashboards/namespace.json:896:        "allValue": null,
grafana/dashboards/namespace.json:918:        "allValue": null,
grafana/dashboards/service.json:1183:        "allValue": null,
grafana/dashboards/service.json:1205:        "allValue": null,
grafana/dashboards/service.json:1227:        "allValue": null,
grafana/dashboards/service.json:1252:        "allValue": null,
grafana/dashboards/authority.json:1183:        "allValue": null,
grafana/dashboards/authority.json:1205:        "allValue": null,
grafana/dashboards/daemonset.json:2177:        "allValue": null,
grafana/dashboards/daemonset.json:2199:        "allValue": null,
grafana/dashboards/daemonset.json:2221:        "allValue": null,
grafana/dashboards/daemonset.json:2243:        "allValue": null,
grafana/dashboards/cronjob.json:2177:          "allValue": null,
grafana/dashboards/cronjob.json:2199:          "allValue": null,
grafana/dashboards/cronjob.json:2221:          "allValue": null,
grafana/dashboards/cronjob.json:2243:          "allValue": null,
grafana/dashboards/pod.json:2124:        "allValue": null,
grafana/dashboards/pod.json:2146:        "allValue": null,
grafana/dashboards/pod.json:2168:        "allValue": null,
grafana/dashboards/pod.json:2190:        "allValue": null,
grafana/dashboards/pod.json:2215:        "allValue": null,
grafana/dashboards/health.json:2236:        "allValue": null,
grafana/dashboards/health.json:2261:        "allValue": null,
grafana/dashboards/health.json:2286:        "allValue": null,
grafana/dashboards/kubernetes.json:2188:        "allValue": ".*",
grafana/dashboards/statefulset.json:2177:        "allValue": null,
grafana/dashboards/statefulset.json:2199:        "allValue": null,
grafana/dashboards/statefulset.json:2221:        "allValue": null,
grafana/dashboards/statefulset.json:2243:        "allValue": null,
grafana/dashboards/deployment.json:2177:        "allValue": null,
grafana/dashboards/deployment.json:2199:        "allValue": null,
grafana/dashboards/deployment.json:2221:        "allValue": null,
grafana/dashboards/deployment.json:2243:        "allValue": null,
grafana/dashboards/replicationcontroller.json:2177:        "allValue": null,
grafana/dashboards/replicationcontroller.json:2199:        "allValue": null,
grafana/dashboards/replicationcontroller.json:2221:        "allValue": null,
grafana/dashboards/replicationcontroller.json:2243:        "allValue": null,
grafana/dashboards/prometheus-benchmark.json:3644:        "allValue": null,
grafana/dashboards/prometheus-benchmark.json:3666:        "allValue": null,
grafana/dashboards/job.json:2177:        "allValue": null,
grafana/dashboards/job.json:2199:        "allValue": null,
grafana/dashboards/job.json:2221:        "allValue": null,
grafana/dashboards/job.json:2243:        "allValue": null,

Signed-off-by: Matt Miller <mamiller@rosettastone.com>
@mmiller1
Copy link
Contributor Author

mmiller1 commented Mar 4, 2020

Yes, @ihcsim that is the documentation section that discusses what this changes, I don't see any reason why no to make the change to the other variables, so I added a commit to do so.

Copy link
Contributor

@ihcsim ihcsim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mmiller1 Thanks for the changes! I did some comparison with the 2.7 Grafana dashboards. LGTM 👍!

If you haven't already, please fill up this form so that we can send you some awesome Linkerd swags.

@ihcsim ihcsim merged commit c39b698 into linkerd:master Mar 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants