Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

make get pod log with follow option as CONNECT verb #50123

Merged
merged 1 commit into from
Sep 1, 2017
Merged

make get pod log with follow option as CONNECT verb #50123

merged 1 commit into from
Sep 1, 2017

Conversation

WIZARD-CXY
Copy link
Contributor

@WIZARD-CXY WIZARD-CXY commented Aug 4, 2017 •
edited by smarterclayton
Loading

What this PR does / why we need it:
Don't make the get log with follow option request mix with GET pods request. Make it reported as a WATCH pod log request.
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Fixes #49998

Pod log attempts are now reported in apiserver prometheus metrics with verb `CONNECT` since they can run for very long periods of time.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 4, 2017
@k8s-ci-robot
Copy link
Contributor

Hi @WIZARD-CXY. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 4, 2017
@k8s-github-robot k8s-github-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. release-note-label-needed labels Aug 4, 2017
@WIZARD-CXY
Copy link
Contributor Author

WIZARD-CXY commented Aug 4, 2017 •
edited
Loading

Before the fix, some get pod log with follow request will make apiserver_request_latency very high and give my cluster a api latency high alert, see this pic. Just one getting pod log with follow request makes 0.99 quantile of api lantency up to 41s...
2017-08-04 10 56 45
After the fix, I make the get pod log with follow request as a watch pod log request.
2017-08-04 11 35 33
It makes the api latency monitoring more fine-grained IMHO

@WIZARD-CXY WIZARD-CXY mentioned this pull request Aug 4, 2017
Closed
@wojtek-t wojtek-t assigned mml and jimmidyson and unassigned jimmidyson and wojtek-t Aug 4, 2017
jimmidyson
jimmidyson suggested changes Aug 4, 2017
View reviewed changes
Copy link
Member

@jimmidyson jimmidyson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor nits, but otherwise LGTM.

staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go Outdated
@@ -106,6 +106,15 @@ func MonitorRequest(request *http.Request, verb, resource, subresource, scope, c
}
}
}

// make get pod log with follow option api request reported as watch
if verb == "GET" && resource == "pods" && subresource == "log"{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs gofmt'ing please.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do sir

staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go Outdated
// make get pod log with follow option api request reported as watch
if verb == "GET" && resource == "pods" && subresource == "log"{
if values := request.URL.Query()["follow"]; len(values) > 0 {
if values[0] == "true" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use strings.ToLower(values[0]) to ignore case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do sir

@WIZARD-CXY
Copy link
Contributor Author

@jimmidyson fixed, PTAL

@jimmidyson
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 4, 2017
@WIZARD-CXY
Copy link
Contributor Author

/retest

@WIZARD-CXY
Copy link
Contributor Author

/test pull-kubernetes-unit

@WIZARD-CXY
Copy link
Contributor Author

encountered some flaky test, ok now

@WIZARD-CXY
Copy link
Contributor Author

@mml @jimmidyson PTAL

@jimmidyson
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 7, 2017
@k8s-github-robot
Copy link

Adding do-not-merge because the release note process has not been followed.
One of the following labels is required "release-note", "release-note-action-required", "release-note-experimental" or "release-note-none".
Please see: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes.

@k8s-github-robot k8s-github-robot added the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Aug 7, 2017
@jimmidyson
Copy link
Member

/assign @deads2k

@sttts
Copy link
Contributor

sttts commented Aug 31, 2017

@WIZARD-CXY it's the other way around. The apiserver repo is published from the staging repo. All changes must be done in staging.

@WIZARD-CXY
Copy link
Contributor Author

@sttts OIC, thanks for pointing it out.

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 31, 2017
@k8s-github-robot k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 31, 2017
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 31, 2017
@WIZARD-CXY
Copy link
Contributor Author

@smarterclayton changed my pr according to your suggestion. PTAL. Tested it ok as you can see below the get pod log request is reported "CONNECT" as verb
2017-08-31 6 27 56

@WIZARD-CXY
Copy link
Contributor Author

For now, it only converts GET to CONNECT for pod log resource. Other storage may add its own StorageMetricsOverride implementation in the future if needed.

make api request verb can be overrided and make "GET" pod log request…
e49315f 
… reported as "CONNECT" pod log request for metrics
@WIZARD-CXY
Copy link
Contributor Author

/retest

@mml
Copy link
Contributor

mml commented Aug 31, 2017

/unnasign

@mml
Copy link
Contributor

mml commented Aug 31, 2017

/unassign

@smarterclayton
Copy link
Contributor

/retest
/lgtm

Thanks!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 31, 2017
@smarterclayton smarterclayton added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. release-note-label-needed labels Aug 31, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: WIZARD-CXY, jimmidyson, smarterclayton

Associated issue: 49998

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 31, 2017
@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 51480, 49616, 50123, 50846, 50404)

@k8s-github-robot k8s-github-robot merged commit 00846fc into kubernetes:master Sep 1, 2017
@WIZARD-CXY WIZARD-CXY deleted the fixlog branch September 1, 2017 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deads2k deads2k deads2k left review comments

@sttts sttts sttts left review comments

@jimmidyson jimmidyson jimmidyson approved these changes

Assignees

@jimmidyson jimmidyson

@smarterclayton smarterclayton

@deads2k deads2k

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

get pod log request maybe account in the wrong api latency metric report
9 participants
@WIZARD-CXY @k8s-ci-robot @jimmidyson @k8s-github-robot @deads2k @sttts @smarterclayton @mml @wojtek-t