Skip to content
This repository has been archived by the owner on Jul 7, 2023. It is now read-only.

Metrics scraper using recycled tokens in kubernetes 1.21 #42

Closed
shalver opened this issue Apr 13, 2021 · 9 comments · Fixed by #47 or #49
Closed

Metrics scraper using recycled tokens in kubernetes 1.21 #42

shalver opened this issue Apr 13, 2021 · 9 comments · Fixed by #47 or #49
Assignees
@maciaszczykm
Labels
lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness.

Comments

@shalver
Copy link

shalver commented Apr 13, 2021

Metric scraper version: v1.0.6

The Bound Service Account Tokens feature is now enabled by default in kubernetes 1.21

Service account tokens are now regularly regenerated and replaced on pods. Kubernetes currently extends the life of the token after they are swapped, but also increments the serviceaccount_stale_tokens_total of the kube-apiserver and also audits the information about the offender. The dashboard-metrics-scraper pod is using these tokens past their life and is being flagged in audit log (note the annotation: authentication.k8s.io/stale-token). Moving up to go 1.15 or greater may possible correct this problem.

{
  "level": "Metadata",
  "auditID": "41b24987-fc6f-468c-ac17-0b990d96d214",
  "stage": "RequestReceived",
  "requestURI": "/apis/metrics.k8s.io/v1beta1/nodes",
  "verb": "list",
  "user": {
    "username": "system:serviceaccount:kube-system:kubernetes-dashboard",
    "uid": "90021a00-b991-497b-9b70-93c657e6c569",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ],
    "extra": {
      "authentication.kubernetes.io/pod-name": [
        "dashboard-metrics-scraper-79f744b7dd-jpld7"
      ],
      "authentication.kubernetes.io/pod-uid": [
        "d3e55f47-b195-41f2-b52e-a14d567d4782"
      ]
    }
  },
  "sourceIPs": [
    "172.18.137.0"
  ],
  "userAgent": "metrics-sidecar/v0.0.0 (linux/amd64) kubernetes/$Format",
  "objectRef": {
    "resource": "nodes",
    "apiGroup": "metrics.k8s.io",
    "apiVersion": "v1beta1"
  },
  "requestReceivedTimestamp": "2021-04-13T18:58:14.483224Z",
  "stageTimestamp": "2021-04-13T18:58:14.483224Z",
  "annotations": {
    "authentication.k8s.io/stale-token": "subject: system:serviceaccount:kube-system:kubernetes-dashboard, seconds after warning threshold: 82321"
  }
}
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 12, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 11, 2021
@shalver
Copy link
Author

shalver commented Aug 14, 2021

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 14, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 12, 2021
@shalver
Copy link
Author

shalver commented Nov 12, 2021

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 12, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 10, 2022
@shalver
Copy link
Author

shalver commented Feb 18, 2022

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 18, 2022
@maciaszczykm
Copy link
Member

/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label May 10, 2022
@maciaszczykm maciaszczykm mentioned this issue May 10, 2022
Merged
@maciaszczykm maciaszczykm self-assigned this May 10, 2022
@vgaddavcg
Copy link

vgaddavcg commented May 24, 2022 •
edited
Loading

@maciaszczykm we are using EKS 1.21 and see metrics scraper is using stale token as described by @shalver .
We need to update the kubernetes client-go version to 0.15.7 or higher.

current version: k8s.io/client-go v0.0.0-20190222093734-6e378217e628

BoundServiceAccountTokenVolume graduated to beta and is enabled by default in Kubernetes version 1.21. This feature improves security of service account tokens by allowing workloads running on Kubernetes to request JSON web tokens that are audience, time, and key bound. Service account tokens now have an expiration of one hour. In previous Kubernetes versions, they didn't have an expiration. This means that clients that rely on these tokens must refresh the tokens within an hour. The following Kubernetes client SDKs refresh tokens automatically within the required time frame:

Go v0.15.7 and later

Python v12.0.0 and later

Java v9.0.0 and later

JavaScript v0.10.3 and later

Ruby master branch

Haskell v0.3.0.0

C# v7.0.5 and later

@vgaddavcg vgaddavcg mentioned this issue May 24, 2022
Closed
@maciaszczykm maciaszczykm reopened this May 25, 2022
@maciaszczykm maciaszczykm mentioned this issue May 25, 2022
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@maciaszczykm maciaszczykm

Labels
lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness.
Projects
None yet
Milestone
No milestone
6 participants
@maciaszczykm @shalver @k8s-ci-robot @vgaddavcg @fejta-bot @k8s-triage-robot