What would you like to be added?

Following up from this discussion - #116420 (comment)

There are different components that fall in the critical path for the k8s API today (apiserver (core), authn/authz webhooks, mutating/validating/conversion webhooks, etcd, extension apiservers - maybe more I'm missing). While some of those do, bunch of them don't seem to have metrics tracking request/error counts and latency metrics. Here's what I found so far (will update as we learn more):

• Authentication webhook
  • request/error counts (xref)

    request latencies (xref)
    To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.
• Authorization webhook
  • request/error counts (#117211)

    request latencies (#117211)
    To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.
• Admission webhooks (mutating/validating)
  • request/error counts (xref)

    request latencies (xref)
    To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.
• CRD Conversion webhook
  • request/error counts (#118292)

    request latencies (#118292)
    To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.
• Etcd
  • request/error counts (#117222)

    request latencies (xref)
    To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.
• Extension apiserver
  • request/error counts

    request latencies
    To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.

Finally, wrt the apiserver itself, we measure request/error counts and these flavors of latency metrics today:

• e2e latency (capturing customer experience)

  SLI-based latency (measuring cloud-provider QoS)

  We discussed adding a third flavor of latency metrics that measures only the apiserver "core" latency (not including any external callback/webhook mechanisms) here.
  We don't have consensus that benefits of doing so outweigh the additional metric churn. We can revisit later as TBD
  To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.

Why is this needed?

Metrics at component/dependency level allow us to:

• track each component's performance in isolation
• set internal (non-customer-facing) SLOs for teams owning those components
• narrow down the root-cause for API errors/latencies easily

/sig api-machinery
/sig auth
/sig scalability
/kind feature
/help