Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/kubelet/logs: Add container log size metric #70749

Merged
merged 1 commit into from
Nov 13, 2018
Merged

pkg/kubelet/logs: Add container log size metric #70749

merged 1 commit into from
Nov 13, 2018

Conversation

brancz
Copy link
Member

@brancz brancz commented Nov 7, 2018

What type of PR is this?

/kind feature

What this PR does / why we need it:

Unless an additional DaemonSet that has the same permissions as the kubelet is deployed there is currently no way to monitor the size and thus the amount of logs written by a Kubernetes cluster's pods/containers.

Special notes for your reviewer:

This is not technically a feature, but more of a small enhancement for monitoring logs, in my opinion it should be fine to merge for 1.13 as long as it merges until code-freeze.

Does this PR introduce a user-facing change?:

Not an end-user facing change, but an additional metrics to be used for monitoring by cluster admins.

Add `kubelet_container_log_size_bytes` metric representing the log file size of a container.

/sig instrumentation

@kubernetes/sig-instrumentation-pr-reviews
@kubernetes/sig-node-pr-reviews

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/node Categorizes an issue or PR as relevant to SIG Node. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/kubelet labels Nov 7, 2018
@derekwaynecarr
Copy link
Member

/assign @derekwaynecarr

i am wondering why we would not put some of this logic in cri stats provider so we can avoid listing containers all over code base, i need to take a deeper look to know pros/cons.

metalmatze
metalmatze reviewed Nov 7, 2018
View reviewed changes
pkg/kubelet/logs/metrics.go Outdated
}
podMetadata := pod.GetMetadata()

ch <- prometheus.MustNewConstMetric(descLogSize, prometheus.GaugeValue, float64(info.Size()), podMetadata.GetNamespace(), podMetadata.GetName(), container.GetMetadata().GetName(), id)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit: would you mind putting this across multiple lines? Thanks.

@brancz
Copy link
Member Author

brancz commented Nov 7, 2018

I did indeed find something in the cri stats provider so I will rework this PR to reuse that logic.

@brancz brancz force-pushed the log-metrics branch 3 times, most recently from d798814 to 553a574 Compare November 7, 2018 20:37
derekwaynecarr
derekwaynecarr suggested changes Nov 9, 2018
View reviewed changes
Copy link
Member

@derekwaynecarr derekwaynecarr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a structural change, otherwise this is good for me.

pkg/kubelet/kubelet.go Outdated
@@ -711,6 +712,8 @@ func NewMainKubelet(kubeCfg *kubeletconfiginternal.KubeletConfiguration,
runtimeService,
imageService,
stats.NewLogMetricsService())

prometheus.MustRegister(klet.StatsProvider.Collector())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typically, we have put registration of collectors under kubelet/metrics/metrics.go, see the Register function there.

Can you move this under initializeModules where it calls metrics.Register and pass in the additional collector?

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 9, 2018
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 9, 2018
@brancz
Copy link
Member Author

brancz commented Nov 12, 2018

/retest

@brancz
Copy link
Member Author

brancz commented Nov 13, 2018

/test pull-kubernetes-local-e2e-containerized

@nikopen
Copy link
Contributor

nikopen commented Nov 13, 2018

/milestone v1.13

/priority important-soon

@k8s-ci-robot k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Nov 13, 2018
@k8s-ci-robot k8s-ci-robot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Nov 13, 2018
@k8s-ci-robot k8s-ci-robot added this to the v1.13 milestone Nov 13, 2018
@derekwaynecarr
Copy link
Member

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 13, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brancz, derekwaynecarr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 13, 2018
@k8s-ci-robot k8s-ci-robot merged commit 67b7d97 into kubernetes:master Nov 13, 2018
@brancz brancz deleted the log-metrics branch November 13, 2018 15:21
@Random-Liu Random-Liu mentioned this pull request Nov 17, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@metalmatze metalmatze metalmatze left review comments

@derekwaynecarr derekwaynecarr derekwaynecarr requested changes

@krmayankk krmayankk Awaiting requested review from krmayankk

@tmrts tmrts Awaiting requested review from tmrts

Assignees

@derekwaynecarr derekwaynecarr

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/node Categorizes an issue or PR as relevant to SIG Node. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
v1.13
Development

Successfully merging this pull request may close these issues.
5 participants
@brancz @derekwaynecarr @nikopen @k8s-ci-robot @metalmatze