Note: this issue is only meant to document the current situation, provide historical context and elucidate any operational aid that exists for this particular situation.

What's Happening?

From the go1.23rc1 release notes (https://tip.golang.org/doc/go1.23):

The x509sha1 GODEBUG setting will be removed in the next Go major release (Go 1.24). This will mean that crypto/x509 will no longer support verifying signatures on certificates that use SHA-1 based signature algorithms.

The decision to stop support for SHA-1 by default was made in go1.18: https://golang.org/doc/go1.18#sha1

The decision to fully remove support for this in go1.24 is because of the extended Go compatibility work to keep GODEBUGs around for a min. of 2 years: https://go.dev/blog/compat

Issue In The Context of Kubernetes

While it is established that SHA-1 isn't safe anymore, there might still be cases where user-provided certificates are signed using SHA-1 based algorithms through private authorities. This can happen in aggregated API Servers or while using webhooks.

If you have relied on SHA-1 based certs in the past, and assuming you are on a version of Kubernetes that uses a version of Go greater than 1.18, the way you would have continued support for this would be to set the env variable GODEBUG=x509sha1=1 at kube-apiserver startup time.

Once the x509sha1 GODEBUG goes away in go1.24, your cluster will not continue to work as expected.

Past tracking issues:

• go1.18: [Tracker] go1.18 breaks CSR validation (fix pending in go1.18.1) #108910
• go1.18: Add metrics for SHA-1 certificates that will be rejected by go1.18 #108940

Observability Around SHA-1 Certs

As part of absorbing the go1.18 release, we've added in some observability to help detect the use of these certs and also where they are being used.

Detecting If SHA-1 Certs Are Being Used

The following metrics are exposed by the kube-apiserver that can be monitored (added in #109024):

apiserver_kube_aggregator_x509_insecure_sha1_total
apiserver_webhooks_x509_insecure_sha1_total

You should also be able to query this using a client like kubectl:

❯ kubectl get --raw '/metrics' \
 | prom2json \
 | jq '.[] | select(.name=="apiserver_kube_aggregator_x509_insecure_sha1_total")'

Furthermore, since go1.21, the Go runtime also exports metrics around usage of individual GODEBUGs and Kubernetes by default exposes all of these metrics, if you'd like to verify the number of times the crypto/x509 package processed a cert with SHA-1 signatures because GODEBUG=x509sha1=1 was enabled, you can do the following:

❯ kubectl get --raw '/metrics' \
 | prom2json \
 | jq '.[] | select(.name=="go_godebug_non_default_behavior_x509sha1_events_total")'

Detecting Where SHA-1 Certs Are Being Used

If you see non-zero usage of SHA-1 certs, you can also get information about the specific hosts that are serving these certificates via (added in #109107):

• kube-apiserver log (log lines containing invalid-cert.kubernetes.io)
• kube-apiserver audit log: https://kubernetes.io/docs/reference/labels-annotations-taints/audit-annotations/#insecure-sha1-invalid-cert-kubernetes-io-hostname

Approximate and Highly Tentative Timelines

Go major versions are typically released every August and February. go1.23 is slated for August of 2024 and by extrapolation, go1.24 should come out in Feb of 2025.

Typically, HEAD will start using go1.24 ~immediately once released, and ~3 months post the release of go1.24, all active release branches will also move to go1.24 (please see KEP 3744 for more information).

This means that by May/June of 2025, all supported Kubernetes versions will fully start rejecting SHA-1 based certs, with no possibility of override. Given that timeline, please start rotating those certs in case you rely on them!

/sig architecture auth
/triage accpeted
/priority important-longterm
/cc @dims @liggitt