Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New metric since logstash 8.13: worker_utilization #365

Closed
VCCPlindsten opened this issue Oct 9, 2024 · 6 comments
Closed

New metric since logstash 8.13: worker_utilization #365

VCCPlindsten opened this issue Oct 9, 2024 · 6 comments
Labels
enhancement New feature or request

Comments

@VCCPlindsten
Copy link
Contributor

Feature Description

There is a new metric on pipeline level since 8.13: https://www.elastic.co/guide/en/logstash/current/logstash-8-13-0.html#notable-8.13.0

Please collect and include it.

_node/stats sample (logstash 8.15.0) 
"main": {
      "events": {
        "duration_in_millis": 1544425,
        "filtered": 4510626,
        "out": 4510626,
        "in": 4510877,
        "queue_push_duration_in_millis": 1269044
      },
      "flow": {
        "input_throughput": {
          "current": 1765,
          "last_1_minute": 2083,
          "last_5_minutes": 2111,
          "last_15_minutes": 1754,
          "lifetime": 1487
        },
        "worker_concurrency": {
          "current": 0.5364,
          "last_1_minute": 0.6848,
          "last_5_minutes": 0.6716,
          "last_15_minutes": 0.5716,
          "lifetime": 0.5093
        },
        "filter_throughput": {
          "current": 1765,
          "last_1_minute": 2079,
          "last_5_minutes": 2111,
          "last_15_minutes": 1754,
          "lifetime": 1487
        },
        "output_throughput": {
          "current": 1765,
          "last_1_minute": 2079,
          "last_5_minutes": 2111,
          "last_15_minutes": 1754,
          "lifetime": 1487
        },
        "worker_utilization": {
          "current": 53.64,
          "last_1_minute": 68.48,
          "last_5_minutes": 67.16,
          "last_15_minutes": 57.16,
          "lifetime": 50.93
        },
        "queue_backpressure": {
          "current": 0.4336,
          "last_1_minute": 0.5846,
          "last_5_minutes": 0.5729,
          "last_15_minutes": 0.4807,
          "lifetime": 0.4185
        }
      },

Version of logstash-exporter this feature request applies to

None

Motivation and Goals

This is a useful metric to determine how saturated logstash is since it uses the pipeline-level configured worker count, 100% means its saturated, no matter how many workers are currently configured.

Alternatives Considered

Using a different way to collect logstash metrics - not a good option.

Additional Context

There doesn't seem to be another way to get at this number - because the amount of workers is a per-pipeline attribute, which doesn't seem to be exposed by logstash.
logstash_info_pipeline_workers only collects the default number.
@VCCPlindsten VCCPlindsten added the enhancement New feature or request label Oct 9, 2024
This was referenced Oct 9, 2024
Merged
Merged
Merged
@kuskoman
Copy link
Owner

kuskoman commented Oct 9, 2024

@VCCPlindsten are you using v1, or v2?
I create a PR for v1: #369
but v2 is in rather unstable state on the branches that i am working on RN and i wonder whether i have to rebase them, or just fix that later on

@kuskoman
Copy link
Owner

kuskoman commented Oct 9, 2024

also, i forgot to ask. could you do the review on the PR i just posted?if it is allright i can merge it soon and create a PR

@VCCPlindsten
Copy link
Contributor Author

Looking a little bit closer at the _node api, it seems per-pipeline workers is exposed, under the "pipelines" key. Not sure if that is a new thing as well - I've not noticed it before.

Reviewing shortly.

@kuskoman
Copy link
Owner

kuskoman commented Oct 9, 2024

i am currently working and should not be doing this task, but it seems like a small one :D
the PR was created really quick, so there may be these mistakes in it

Are you using v1 version of the exporter? I will replicate the PR to v2 later on

@VCCPlindsten
Copy link
Contributor Author

I am using v1, yes - since v2 is not stable yet.

@VCCPlindsten
Copy link
Contributor Author

Working in release 1.8.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kuskoman @VCCPlindsten