Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add plugin input filter output metrics #101

Merged
merged 11 commits into from
Apr 26, 2023
Merged

Add plugin input filter output metrics #101

merged 11 commits into from
Apr 26, 2023

Conversation

excalq
Copy link
Contributor

@excalq excalq commented Apr 21, 2023 •
edited
Loading

Resolves #87. Adds metrics for pipeline plugins (inputs, outputs, filters, codecs)

These metrics have four labels:

  • pipeline
  • plugin (e.g. mutate)
  • plugin_type (e.g. input)
  • plugin_id (e.g. drop_spam_80pct or last 8-chars of autogenerated UUID)

To accommodate this, I changed NewDescWithHelpAndLabel()'s label parameter to be variadic, and pluralized both the param and the function name, hence: NewDescWithHelpAndLabels(name string, help string, labels ...string).

New metrics:

logstash_stats_pipeline_plugin_events_duration
logstash_stats_pipeline_plugin_events_in
logstash_stats_pipeline_plugin_events_out
logstash_stats_pipeline_plugin_events_queue_push_duration

What these look like:

image

@excalq excalq marked this pull request as draft April 21, 2023 04:17
@codecov
Copy link

codecov bot commented Apr 21, 2023 •
edited
Loading

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (5683d2a) 100.00% compared to head (6b3e913) 100.00%.

Additional details and impacted files 
@@            Coverage Diff            @@
##            master      #101   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           12        12           
  Lines          434       480   +46     
=========================================
+ Hits           434       480   +46
Impacted Files Coverage Δ
collectors/nodestats/nodestats_collector.go 100.00% <100.00%> (ø)
collectors/nodestats/pipeline_subcollector.go 100.00% <100.00%> (ø)
prometheus_helper/prometheus_helper.go 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@excalq
Copy link
Contributor Author

excalq commented Apr 21, 2023 •
edited
Loading

Yeah, don't merge this yet, it breaks when pipeline actions (e.g. drop{}) exist more than once in a pipeline. This will need some more work.

An error has occurred while serving metrics:

3 error(s) occurred:
* collected metric "logstash_stats_pipeline_filters_events_in" { label:<name:"input" value:"drop" > label:<name:"pipeline_id" value:"algod-json" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "logstash_stats_pipeline_filters_events_out" { label:<name:"input" value:"drop" > label:<name:"pipeline_id" value:"algod-json" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "logstash_stats_pipeline_filters_events_duration" { label:<name:"input" value:"drop" > label:<name:"pipeline_id" value:"algod-json" > gauge:<value:0 > } was collected before with the same name and label values

Update: Resolved by adding plugin-id label

@excalq excalq force-pushed the excalq/add-plugin-input-filter-output-metrics branch from e89f0c6 to bfd1f39 Compare April 21, 2023 22:05
@excalq
Copy link
Contributor Author

excalq commented Apr 24, 2023

@kuskoman I'm pretty much ready to publish this PR, once I'm done testing its outputs in Grafana. If you'd like to give it a review, you're welcome to.

@kuskoman
Copy link
Owner

@excalq

At first glance it looks really nice, but you need to give me some time to checkout the code locally and play with it.
Thank you for the effort!

@excalq
Copy link
Contributor Author

excalq commented Apr 24, 2023

Of course, no rush hence this WIP PR being still a draft. I have a comprehensive Grafana dashboard now using these metrics that's in the "cleanup" stages.

@excalq
Copy link
Contributor Author

excalq commented Apr 24, 2023

I'll need to rebase this as well.

@excalq excalq force-pushed the excalq/add-plugin-input-filter-output-metrics branch from fb81b51 to eb1ea4e Compare April 25, 2023 22:47
@excalq
Copy link
Contributor Author

excalq commented Apr 25, 2023

Ok, this looks good on my end, so I'll publish the PR! Btw, I reduced the number of metrics quite a bit, as the plugin type (input, output, filter, codec) should really just be a label.

@excalq excalq marked this pull request as ready for review April 25, 2023 23:27
@kuskoman kuskoman merged commit f94dbb0 into kuskoman:master Apr 26, 2023
@excalq excalq mentioned this pull request Apr 26, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Additional Logstash Metrics
2 participants
@excalq @kuskoman