Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds metrics to PortForward Websockets #126187

Merged
merged 1 commit into from
Jul 22, 2024
Merged

Adds metrics to PortForward Websockets #126187

merged 1 commit into from
Jul 22, 2024

Conversation

seans3
Copy link
Contributor

@seans3 seans3 commented Jul 18, 2024 •
edited
Loading

  • Adds metric to PortForward tunneling through WebSockets: stream_tunnel_requests_total.
  • Includes unit tests for the new metric.
    • Current unit test coverage for this proxy package: 84.0%
    • ok k8s.io/apiserver/pkg/util/proxy 2.921s coverage: 84.0% of statements

/kind cleanup

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 18, 2024
@seans3
Copy link
Contributor Author

seans3 commented Jul 18, 2024

/sig api-machinery
/priority important-soon

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. area/apiserver and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 18, 2024
@seans3
Copy link
Contributor Author

seans3 commented Jul 18, 2024

/assign @liggitt
/assign @aojea

aojea
aojea reviewed Jul 18, 2024
View reviewed changes
staging/src/k8s.io/apiserver/pkg/util/proxy/metrics/metrics.go
Help: "Total number of requests that were handled by the StreamTunnelProxy, which processes streaming PortForward/V2",
StabilityLevel: metrics.ALPHA,
},
[]string{statuscode},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this only considers the backend status code, right?

Copy link
Contributor Author

@seans3 seans3 Jul 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessarily.

  1. Within the TunnelingHandler.ServeHTTP there is one BadRequest response if the request does not contain subprotocols. In this case we return the 400-level response (and increment the metric) but do not consider either the backend/upstream SPDY upgrade status code or the WebSocket upgrade.

  2. Within the InitializeWrite, we consider the backend/upstream SPDY upgrade response. Only after we determine the SPDY upgrade succeeds, do we attempt the WebSocket upgrade. So the successful status code represents both the successful SPDY upgrade and the successful WebSocket upgrade. It is also possible in the situation where the SPDY upgrade was successful, but the WebSocket upgrade was not (see your next comment), where the metric is incremented with a 500-level error response.

Please let me know what you think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking more about an end user or admin or SRE and how it will interpret this metrics, in case this metrics goes up ... where should they start looking , apiserver, kubelet, containerd?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that if the non-successful metrics rise, investigators would initially look at the error statements in the apiserver logs. I think they would have to look at these logs no matter what. Please let me know if we should separate the SPDY from WebSocket status codes. The one place where the SPDY vs. WebSocket status code might be indeterminate is the TunnelingHandler.ServeHTTP returning a BadRequest status code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nah, too complex to be able to express with metrics

/Lgtm

Thanks for the explanation

aojea
aojea reviewed Jul 18, 2024
View reviewed changes
staging/src/k8s.io/apiserver/pkg/util/proxy/streamtunnel.go Outdated
@@ -352,11 +357,13 @@ func (u *tunnelingWebsocketUpgraderConn) InitializeWrite(backendResponse *http.R
}

klog.V(4).Infof("websocket connection created: %s", conn.Subprotocol())
metrics.IncStreamTunnelRequest(context.Background(), strconv.Itoa(http.StatusOK))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is an exit on line 356, should this metric incremented there too?

Copy link
Contributor Author

@seans3 seans3 Jul 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes--you are correct. There should be a 500 internal server error metric incremented in this case. I have added it.

This case represents the situation where the SPDY upgrade was successful, but the subsequent WebSocket upgrade failed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be recording StatusOK or StatusSwitchingProtocols?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed from StatusOK (200) to StatusSwitchingProtocols (101).

@aojea
Copy link
Member

aojea commented Jul 18, 2024

overall lgtm, a question if we need to get metrics on all. possible return status and a nit on te tests

@seans3 seans3 force-pushed the portforward-websockets-metrics branch from 763ad26 to 98ea294 Compare July 18, 2024 19:43
@cici37
Copy link
Contributor

cici37 commented Jul 18, 2024

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 18, 2024
@seans3 seans3 force-pushed the portforward-websockets-metrics branch from 98ea294 to 293a8db Compare July 18, 2024 21:04
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 19, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: f66a66695b6a6a78e7739a09adf1e67ff8c8f533

@liggitt liggitt added this to the v1.31 milestone Jul 20, 2024
@seans3 seans3 force-pushed the portforward-websockets-metrics branch from 293a8db to 90d70ed Compare July 22, 2024 21:08
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 22, 2024
@k8s-ci-robot k8s-ci-robot requested review from aojea and liggitt July 22, 2024 21:08
@liggitt
Copy link
Member

liggitt commented Jul 22, 2024

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 22, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 20aba58079378cc7664283b61d747e0bdb3b5fe6

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, seans3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 22, 2024
@k8s-ci-robot k8s-ci-robot merged commit 04cc0a1 into kubernetes:master Jul 22, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@caesarxuchao caesarxuchao Awaiting requested review from caesarxuchao

@sttts sttts Awaiting requested review from sttts

@aojea aojea Awaiting requested review from aojea

@liggitt liggitt Awaiting requested review from liggitt

Assignees

@liggitt liggitt

@aojea aojea

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
[sig-release] Bug Triage
Archived in project
Milestone
v1.31
Development

Successfully merging this pull request may close these issues.
5 participants
@seans3 @aojea @cici37 @k8s-ci-robot @liggitt