/sig arch

@kubernetes/sig-arch

/assign @timothysc

/cc @kubernetes/sig-architecture-feature-requests @kubernetes/cncf-conformance-wg
/sig testing
/sig architecture
/kind feature

/assign @spiffxp

/area conformance

@klizhentas please start a discussion on the kubernetes-sig-architecture mailing list.

We discussed this in sig-auth today and recounted some of the issues with the certificates API that we'd previously discussed.

• API shape/issues
  • Requires requesters to know all the info about the end certificate.
  • Use for higher-level requests (“give me a node client cert”, “give me a serving cert for a pod backing a service”).
  • Requested certificate attributes are split unthoughfully between encoded CSR and fields in request spec which create difference in semantics.
• Approval flow/issues
  • Cannot limit or add components to request (limit or add SANs, usages, etc)
• Signing flow/issues
  • Method for multiple signers to interact (or approver to indicate what signer should be used)
• Guarantees on issued certificates
  • No (current) guarantee all requested extensions/SANs are issued
  • No (current) guarantee issued client certificates will be accepted as API client certs

We now need to discuss whether we should graduate the current API to v1 and solve these in a v2alpha or whether we need to solve these before v1 of this API. We are also unsure whether we can even include this API and specific guarantees on its behavior (e.g. able to issue a client certificate against the Kubernetes API) in conformance.

@kubernetes/sig-auth-api-reviews

what are the work items remaining on this item ? I would like to help .

The first step is to start describing the use cases for the signing profiles and parameters, to figure out which actors would be providing/overriding/consuming that info.

That information would help inform where in the API the data would make sense (in spec, in status, able to be influenced during approval, etc)

Once we have several good use cases, look at API required to support it.

Then we'd figure out if there's a straightforward first step that addresses enough use cases to add (maybe in 1.15?), and look to promote to v1 in 1.16 or beyond.

@liggitt @mikedanese Is this issue still exist, If here is anything where I can Contribute?

/priority important-longterm

Since #82952 is closed/duped - adding the comment:
Until there is a solution for trust distribution - can the doc be updated to make a recommendation to use the public key of apiserver ( which is already distributed ), and to not support using a separate root for cert signing ?

Otherwise the feature is not usable, and we can't depend on it since on some clusters we might
get certs without knowing the public key. When trust distribution is solved you can add back the comment and indicate how other root CAs are integrated.

I would also add that prioritizing #80057 would help a lot any app using Spiffee.

Do you have any info on the schedule and plans for implementing various features ? Should we
send patches to make it happen sooner ?

can the doc be updated to make a recommendation to use the public key of apiserver ( which is already distributed ), and to not support using a separate root for cert signing

I disagree that we should not support using a separate root. We should update the documentation to make it clearer that multiple signers, or signers that are not part of the API server serving cert CA bundle are a possibility.

Do you have any info on the schedule and plans for implementing various features

The KEP describing the current state of the CSR API just merged (https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/20190607-certificates-api.md). I think @mikedanese is planning to start design discussions for the GA issues this release.

I didn't say 'we should not support a separate root' - quite the opposite, we should, when a mechanism to distribute those separate roots is defined. Right now there is no way for the user of the API to know which public key was used to sign his cert, or to have that public key available in other namespaces or workloads. So the docs should reflect the current status - only apiserver's CA ( or the root CA that signed the apiserver ) in current version, with plans for future version to distribute the other supported CAs.

Regarding the KEP - not sure what's the process to comment on it, but it doesn't seem to address the issue of how to distribute the signer's public key. One option is to save a full chain in the status (including the public key of the signer) - but that is only 1/2 of the problem. Having other workloads in the cluster discover the root CAs that are authorize to sign keys in the cluster is the other half.

Right now there is one mechanism - kubelet distributing the 'primary' CA in the cluster. It can distribute the other roots, or some other mechanism can be provided to allow the list to be visible from regular workloads ( including apps that don't have permissions or don't talk with the apiserver )

The KEP only reflects the current Beta status.
We should add all the new requirements (including signer info distribution) when updating it for GA.

For distribution, we discussed using ConfigMaps in the past.
Since signer certs are generally not sensitive, those ConfigMaps could be made readable to anyone in the cluster.

So the docs should reflect the current status - only apiserver's CA ( or the root CA that signed the apiserver ) in current version

The current status is that consumers of the CSR API must know/obtain trust roots out of band. I agree the docs should be clearer about this.

not sure what's the process to comment on it, but it doesn't seem to address the issue of how to distribute the signer's public key

@mikedanese will be opening a PR to fill in the "Beta to GA" requirements section

Right now there is one mechanism - kubelet distributing the 'primary' CA in the cluster

It is important to note that this is not the "primary" CA, but is just the CA used to verify the API server's serving certificate. There is no requirement for that to overlap with the CA used to verify client cert authentication, kubelet serving certificates, etc.

I think that it's important to support multiple signers before we more to GA so that we can be confident of the shape of the API. Distribution of trust is not the same problem and could reasonably be handled separately.

@munnerz do you have a concrete design for multiple signers via a CSR endpoint? As I recall cert-manager either has that power or considered that power at some point.

I don't think ConfigMap is viable - unless it is distributed to all namespaces. I think an important use case for the API is to use the CSR API to generate certificates for services running in a namespace, and use TLS to connect from a different namespace. That requires both client and server to know the root CAs that are trusted.

Again: I am extremely happy to have multiple CAs, it fits perfectly for multi-cluster use cases where certs can be issued in different clusters and the clusters trust each other.

But until this is fully implemented - my concern is to make sure in the current version of K8S and with current CSR API, which are quite valuable and can be used - if only the doc is clarified that 'multiple signers' is not yet fully supported but just planned.

It should be possible to allow CA bundle ConfigMap access across namespaces, right?

It should be possible to allow CA bundle ConfigMap access across namespaces, right?

Simplest solution is an injector. Jetstack and openshift both have mechanisms for doing that.

AFAIK injector can't inject a configmap - just modify the pod specification to add a mount of an existing configmap. It is possible to have a controller that creates configmaps in each namespace, and it is possible to have the injector add a base64-encoded env variable or similar. Unless there is some other mechanism I don't know.

Do you have any links to the mechanisms used by Jetstack or openshift ? For Istio we auto-create a secret in each namespace, and autoinject a mount - but that's what we want to replace with CSR API and k8s-managed certs.

Root cert distribution is just 1/2 of the gap - the other problem is that the generated cert doesn't include the signing cert (is not a chain). I think the sha of the signer is available and could be used to guess, but not very clear in the docs.

Root cert distribution is just 1/2 of the gap - the other problem is that the generated cert doesn't include the signing cert (is not a chain). I think the sha of the signer is available and could be used to guess, but not very clear in the docs.

Good point, this is also a part of GA improvements.
We should make the CSR->Signer mapping more transparent and deterministic.

One option is to have some signer ID in the CSR, and then all other signers ignore it.
Another option is to have the CSR controller attach that ID based on which signer was used.

I personally like the former, but we can hash it out in the KEP PR once @mikedanese or someone else gets to writing one.

I personally like returning a full chain - including the any intermediary certs up to the root. I assume intermediary certs will also be needed at some point. It's also simpler than singer ID or key ID, and safer for root key rotations.

@munnerz do you have a concrete design for multiple signers via a CSR endpoint? As I recall cert-manager either has that power or considered that power at some point.

We have a design document that describes our own 'CertificateRequest' resource type, and how it can be used to build "out-of-tree" issuers: https://github.com/jetstack/cert-manager/blob/master/design/20190708.certificate-request-crd.md

You can see the actual API type definition for our resource here: https://github.com/jetstack/cert-manager/blob/d474a5f0b3d5832695571eaa49e6587e1b8e9f19/pkg/apis/certmanager/v1alpha2/types_certificaterequest.go#L60-L88

We implement an almost identical API to the built-in type, so it seems logical to explore ways we can move to adopt the in-built CertificateSigningRequest resource (especially as we'd like to implement an 'approval' workflow for cert-manager too, and this would do a lot of that leg-work for us).

The key part here that makes it possible is the issuerRef field, which allows a user to specify a resource name, group and kind that is the 'issuing source'. The signed certificate (and optionally the CA certificate) is then stored on the CertificateRequest after the appropriate certificate request controller has fetched the certificate.

Something like this would be ideal, and (barring a few additional fields such as duration and isCA), would allow us to adopt the in-built type.

@deads2k mentioned that 'certificate class' might be a more appropriate way to model this, which would be a fairly significant change for cert-manager users that up until now have always specified an issuerRef. We have a preference for issuerRef because it allows multiple instances of the same type of issuer (i.e. letsencrypt-staging & letsencrypt-prod), without having to invent new 'magic strings' for the 'certificate class'. That said, the cert-manager project has not been through a formal API review by sig-api-machinery, so I'm keen to hear thoughts 😄

xref kubernetes/enhancements#1332

For the problem that pods in a namespace don't know the root CA, they only have access to the public key of Apiserver.

One option is to expose the signing key on a 'well known' location, served by apiserver. Clients can connect to apiserver using a https:// URL checking against the mounted apiserver root. For example something like: https://kubernetes/api/v1/namespaces/kubesystem/services/rootca/proxy/.well_known/root.pem

It will require deployment of a light component that just serves the root.pem file - but no changes needed to kubelet or other components to automount the root into pods, or configmaps and yaml changes to distribute the root.

Ideally the JWT tokens issued by K8S would be signed by same root as the certificates - and support OIDC which provides a clear way to discovery the info about the signer. I don't know what's the current requirement or options for K8S JWT and if they share the key.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

design moved to https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/20190607-certificates-api.md

history at https://github.com/kubernetes/enhancements/commits/master/keps/sig-auth/20190607-certificates-api.md

v1 API adjustments described in kubernetes/enhancements#1691