kubeadm: enhanced TLS validation for token-based discovery in `kubeadm join` #49520

mattmoyer · 2017-07-24T22:35:28Z

What this PR does / why we need it:
This PR implements enhanced TLS validation for kubeadm join when using token-based TLS discovery. Without this enhancement, kubeadm join has some less-than-ideal security properties. Specifically, in the case where a bootstrap token is compromised, the attacker can impersonate the API server to newly bootstrapping clients (more discussion in the design proposal).

The gist of this enhancement is to support public key pinning in the style of RFC7469. When bootstrapping, kubeadm can now be configured with a whitelist of root CA public keys. It can then validate that the cluster it connects to is operated by the owner of one of those public keys.

These public key hashes are short enough that the entire kubeadm join command can still be copy-pasted relatively easily (not as easily as before, but ~160 characters). Using a public key hash rather than a hash over the entire certificate allows certificates to be reissued with updated expirations without invalidating existing key pins.

This change adds two new command line flags (and associated config parameters):

--discovery-token-ca-cert-hash sha256:<hash>:
Validates that the cluster root CA has a public key fingerprint that matches one of the specified values. If this flag is not passed when token-based discovery is being used, a warning is printed. This warning will become an error in 1.9.
--discovery-token-unsafe-skip-ca-verification:
Disables the warning message when no keys are pinned. In 1.9, this flag will be required unless --discovery-token-unsafe-skip-ca-verification is used.

This is fully backwards compatible and client side (kubeadm) only. It will be a breaking change when the flag becomes required in v1.9.

This validation is done after and in addition to the existing bootstrap token signing/MAC mechanism.

Example from `kubeadm init`:

$ kubeadm init
[...]

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token a66ae0.1f8a5ed9a210e187 192.168.42.10:6443 --discovery-token-ca-cert-hash sha256:547c102383c0f26387b961b4e9b8f842dc07c074c8316f238dbcf5563fc3ac35

Example from `kubeadm join`:

$ kubeadm join --token a66ae0.1f8a5ed9a210e187 192.168.42.10:6443 --discovery-token-ca-cert-hash sha256:547c102383c0f26387b961b4e9b8f842dc07c074c8316f238dbcf5563fc3ac35
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "192.168.42.10:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.42.10:6443"
[discovery] Requesting info from "https://192.168.42.10:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.42.10:6443"
[discovery] Successfully established connection with API Server "192.168.42.10:6443"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

Which issue this PR fixes:
ref kubernetes/enhancements#130
fixes: kubernetes/kubeadm#365

Special notes for your reviewer:
This was proposed and discussed briefly by SIG-cluster-lifecycle and SIG-auth. The design proposal is in Google Docs.

There is a documentation change needed to explain the security properties of kubeadm join with and without --discovery-token-ca-cert-hash. This page should be linked by to by the warning message when you don't pass either of the new flags (I have it pointing here for now, which I think will be the right place). I will follow up with this documentation shortly.

Release note:

kubeadm: added enhanced TLS validation for token-based discovery in `kubeadm join` using a new `--discovery-token-ca-cert-hash` flag.

/cc @luxas @jbeda @ericchiang

k8s-ci-robot · 2017-07-24T22:35:36Z

Hi @mattmoyer. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ericchiang · 2017-07-24T22:47:28Z

/ok-to-test

High level opinion that I brought up in the design doc, but wanted to reiterate here.

The value of --tls-discovery-root-ca for kubeadm join can't be known until kubeadm init runs on the master to generate the CA. This means I can't provision my workers until my masters have been provisioned and kubeadm init has run on the master

This won't work for anyone who provisions the masters and the workers at the same time. For example, Kubernetes Anywhere would have to use the --tls-discovery-allow-unsafe:

https://github.com/kubernetes/kubernetes-anywhere/blob/8d2c7788c63693b0021278dedcf03fad935efef1/phase1/gce/configure-vm-kubeadm.sh#L40

I am for this fix, but I think a majority of auto-scaling setups will have to use the unsafe flag.

ericchiang · 2017-07-24T22:53:54Z

cmd/kubeadm/app/util/pubkeypin/pubkeypin.go

+
+// Set is a set of pinned x509 public keys.
+type Set struct {
+	sha256Hashes [][]byte


This can probably just be map[string]bool. You're always passing around the string value anyway.

+1, will fix. I originally had this code supporting other hash types/lengths (truncated SHA-256/128 and SHA-256 encoded as base58). In the end I just left plain hex-encoded SHA-256 so this doesn't make a lot of sense.

ericchiang · 2017-07-24T22:57:46Z

cmd/kubeadm/app/discovery/token/token.go

+type certChain []*x509.Certificate
+
+// saveCertificateChain parses and saves the array of DER-encoded certificates into a certChain
+func (c *certChain) saveCertificateChain(rawCerts [][]byte, _ [][]*x509.Certificate) error {


Why do you need to save the certificate chain? Don't you just need the CA that's part of the cluster-info?

I think Matt's intention here is to make sure the server presents the same CA key the first time we're contacting it insecurely as after, when we have validated the authenticity of the cluster-info ConfigMap and actually have a CA cert to trust.

Yeah, validating the entire initial connection gives better trust in the other parts of the cluster-info ConfigMap since we use it for more than just the cluster CA. I think we could also accomplish this by making two requests if we want to go that route.

We need the full chain since the server might have some intermediate CAs included that we need to chain through to get from the root to the leaf cert.

ericchiang · 2017-07-24T23:04:20Z

cmd/kubeadm/app/discovery/token/token.go

+	var certificates certChain
+
+	// Create an HTTP client with a custom TLS transport that saves off the certificate chain for later validation
+	insecureClient := http.Client{


We had issues constantly re-initializing transports like this since they have a tendency to keep TCP connections alive.

That makes sense. I think it shouldn't be a huge issue for kubeadm since it's relatively short-lived. That's still pretty non-ideal, though.

I'll see if I can find a way to rework this so the http.Client is long lived.

ericchiang · 2017-07-24T23:05:04Z

cmd/kubeadm/app/discovery/token/token.go

+
+	// Make sure the wrapper is being used correctly
+	if len(certificates) == 0 {
+		return nil, nil, errInsecureClientNoRequest


If you error here you won't close the response body.

Ah, thanks. Will fix.

ericchiang · 2017-07-24T23:06:46Z

cmd/kubeadm/app/discovery/token/token.go

 	tokenutil "k8s.io/kubernetes/cmd/kubeadm/app/util/token"
 	bootstrapapi "k8s.io/kubernetes/pkg/bootstrap/api"
 	"k8s.io/kubernetes/pkg/controller/bootstrap"
 )

 const BootstrapUser = "token-bootstrap-client"

+var (


Does kubeadm normally declare errors like this? I'd expect the fmt.Errorf call to just be inlined. e.g.

kubernetes/cmd/kubeadm/app/util/config/masterconfig.go

Line 58 in 93b144c

return fmt.Errorf("this version of kubeadm only supports deploying clusters with the control plane version >= %s. Current version: %s", kubeadmconstants.MinimumControlPlaneVersion.String(), cfg.KubernetesVersion)

No we haven't done this anywhere else.
I see it's used for testing.
I don't have a strong opinion here, though.

Yeah, this was mainly so I can test that particular error cases return the correct error.

I'm ok with it, but we aren't doing it in other places. This seems to be a good place to use this method though...

ericchiang · 2017-07-24T23:07:59Z

cmd/kubeadm/app/discovery/token/token.go

+			defer response.Body.Close()
+
+			// Read the entire response body
+			responseBytes, err := ioutil.ReadAll(response.Body)


You don't trust the server at this point. Probably good practice to limit the number of bytes read to something reasonable.

+1 will fix.

ericchiang · 2017-07-24T23:10:54Z

cmd/kubeadm/app/discovery/token/token.go

+			chainCerts.AddCert(chainCert)
+		}
+
+		// Now that we know the root CA pool, validate the original certificate chain


It seems much simpler and less prone to error to just make a second request to get the cluster-info once you have the CA and let the Go TLS library automatically do the validation.

Making double requests seemed worse to me, but I think it's sound from a security perspective.

luxas · 2017-07-25T07:47:09Z

cmd/kubeadm/app/apis/kubeadm/validation/validation.go

+	}
+
+	// TODO: convert this warning to an error after v1.8
+	if len(cfg.DiscoveryFile) == 0 && len(cfg.TLSDiscoveryRootCAPubKeys) == 0 && !cfg.TLSDiscoveryAllowUnsafe {


Do we care about if DiscoveryFile is unset and/or DiscoveryToken is set here or can this be simplified to just check PubKeys and AllowUnsafe?

The way it's written now, --tls-discovery-root-ca only applies to the plain token-based case. If you're using DiscoveryFile with local files or a remote HTTPS endpoint, it doesn't do any validation.

I think that's reasonable because DiscoveryFile should be transmitted through some secure means that protects its integrity. Either the local file will have been copied there by some secure orchestrator or the remote file will be authenticated by the web PKI and the system roots. I don't think we need to consider these cases unsafe or require the user to pass --tls-discovery-allow-unsafe.

I have the validation error TLSDiscoveryRootCAPubKeys cannot be used with DiscoveryFile to make sure you don't get a false sense of security from thinking you've pinned the CA when you're actually just relying on the security of the discovery file.

luxas · 2017-07-25T07:50:25Z

cmd/kubeadm/app/discovery/token/token.go

 	tokenutil "k8s.io/kubernetes/cmd/kubeadm/app/util/token"
 	bootstrapapi "k8s.io/kubernetes/pkg/bootstrap/api"
 	"k8s.io/kubernetes/pkg/controller/bootstrap"
 )

 const BootstrapUser = "token-bootstrap-client"

+var (


No we haven't done this anywhere else.
I see it's used for testing.
I don't have a strong opinion here, though.

luxas · 2017-07-25T07:54:19Z

cmd/kubeadm/app/discovery/token/token.go

+
+		// For each cluster, validate that its CA matches a pinned key and save that root into an x509.CertPool
+		pinnedRoots := x509.NewCertPool()
+		for _, cluster := range finalConfig.Clusters {


The spec states that there must only be one cluster in the cluster-info kubeconfig file.
Basically that single Cluster's CA cert and server endpoint is everything we're using

Thanks. Instead of looping I added an assertion that there is only a single cluster.

luxas · 2017-07-25T07:55:55Z

cmd/kubeadm/app/discovery/token/token.go

 		if err != nil {
-			return nil, err
+			return nil, fmt.Errorf("invalid URL, can't connect: %v", err)


suggest: invalid URL %q, ...

Thanks, will fix.

luxas · 2017-07-25T07:57:25Z

cmd/kubeadm/app/discovery/token/token.go

+			chainCerts.AddCert(chainCert)
+		}
+
+		// Now that we know the root CA pool, validate the original certificate chain


luxas · 2017-07-25T07:59:12Z

cmd/kubeadm/app/discovery/token/token.go

+type certChain []*x509.Certificate
+
+// saveCertificateChain parses and saves the array of DER-encoded certificates into a certChain
+func (c *certChain) saveCertificateChain(rawCerts [][]byte, _ [][]*x509.Certificate) error {


I think Matt's intention here is to make sure the server presents the same CA key the first time we're contacting it insecurely as after, when we have validated the authenticity of the cluster-info ConfigMap and actually have a CA cert to trust.

luxas · 2017-07-25T08:01:05Z

cmd/kubeadm/app/discovery/token/token.go

+// which should perform a single HTTP request and return the response. The temporary
+// client will not validate TLS certificates in real time, but instead will capture
+// and return the certificates for post-hoc validation.
+func withInsecureHTTPClient(


Is there no way to save the cert chain with the normal kube client?
So far we've been able to just skip TLS verification with the normal kube client.

We could extend the normal API client to give more control over the underlying http.Client. I think extending it to support this particular case wouldn't be worth it, but a more general change that allowed passing in a custom http.Client could be more generally useful.

The thinking with doing this as a "raw" http.Client instead of using the normal API client was that this is fairly simple from an HTTP standpoint (single GET request) and that the TLS bits were such an edge case that we probably don't want that complexity in the main API client code.

luxas · 2017-07-25T08:02:28Z

cmd/kubeadm/app/util/pubkeypin/pubkeypin.go

+
+// Package pubkeypin provides primitives for x509 public key pinning in the
+// style of RFC7469.
+package pubkeypin


Do we consider this kubeadm-specific or can we move this to pkg/util or something like k8s.io/apimachinery/pkg/util/sets?

Right now kubeadm is the only thing that knows or cares about these public key pins. It might be useful somewhere else (maybe in kubectl also?).

I don't feel strongly about this either way.

Please keep it in kubeadm. The more dependencies we create, the harder it's going to be to split the repos in the future.

luxas · 2017-07-25T08:13:23Z

[discovery] Failed to connect to API Server "192.168.42.10:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "ec2651" is invalid for this cluster, can't connect

@mattmoyer Are you sure why this happened? Was the token actually not there or is it an error in this code?

The value of --tls-discovery-root-ca for kubeadm join can't be known until kubeadm init runs on the master to generate the CA. This means I can't provision my workers until my masters have been provisioned and kubeadm init has run on the master

@ericchiang The user still has the power to pre-provision the CA, so they can know what the CA hash will be in beforehand
Basically doing this:

// generate CA private key
openssl --foo -out /etc/kubernetes/pki/ca.key
// generate CA certificate
openssl --bar -out /etc/kubernetes/pki/ca.crt
// get CA hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>&/dev/null | openssl dgst -sha256 -hex > my-CA-hash.txt

kubeadm init --token <id>.<secret>

ericchiang · 2017-07-25T15:26:21Z

The user still has the power to pre-provision the CA, so they can know what the CA hash will be in beforehand

@luxas ah, great! Is this actual endorsed behavior or a nice consequence of the code? Might be worth updating the docs as part of this PR.

ericchiang · 2017-07-25T15:48:49Z

cc @kubernetes/sig-auth-pr-reviews

mattmoyer · 2017-07-25T15:56:01Z

@luxas

[discovery] Failed to connect to API Server "192.168.42.10:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "ec2651" is invalid for this cluster, can't connect

@mattmoyer Are you sure why this happened? Was the token actually not there or is it an error in this code?

I think this was just an artifact of my local vagrant environment. It's not reproducible locally, so I think the server just wasn't up quite yet in this case.

jbeda

LGTM.

I'm torn on the "validate current conn" vs. "make a new conn". I could go either way. So I won't express an opinion.

jbeda · 2017-07-25T18:13:11Z

cmd/kubeadm/app/apis/kubeadm/types.go

@@ -101,4 +101,16 @@ type NodeConfiguration struct {
 	NodeName                 string
 	TLSBootstrapToken        string
 	Token                    string
+
+	// TLSDiscoveryRootCAPubKeys specifies a set of public key pins. The root CA
+	// discovered during TLS bootstrapping must match one of these values.


Technically, this is during the bootstrap discovery, not TLS. We use this so that we can establish the kubelet trusting the master.

Ah, yeah I'll fix that comment and see if I made the same mistake elsewhere. I originally approached this code without a clear understanding of the "discovery" vs. "TLS bootstrapping" terminology.

jbeda · 2017-07-25T18:22:14Z

cmd/kubeadm/app/apis/kubeadm/types.go

+
+	// TLSDiscoveryRootCAPubKeys specifies a set of public key pins. The root CA
+	// discovered during TLS bootstrapping must match one of these values.
+	// Specifying an empty set disables root CA pinning. Each hash is a hex encoded


I'd still love to see us do something like base 36 or base 58 here to shorten this. But that can come in a later PR.

I tried to implement the pubkeypin so it's easy to swap in support for multiple encodings and lengths.

I think if we do that we should also switch the default token encodings as well.

mattmoyer · 2017-07-25T21:01:05Z

pull-kubernetes-kubemark-e2e-gce failure is #49320

timothysc

going to recuse myself from this one other then to say, please squash commits.

mattmoyer · 2017-07-26T20:45:58Z

I think the latest change addresses all the reviews so far, including ripping out the post-hoc validation in favor of just making two requests.

I'm leaving the original commits in https://github.com/mattmoyer/kubernetes/commits/8d9feb79c1619d8a80350f8ce1aa6dbd47350e89 and will squash/rebase this PR against current master for a final review.

Thanks for all the feedback.

mattmoyer · 2017-08-06T17:53:02Z

@mikedanese : Thanks for the review. I agree that the names weren't perfect. I just updated with the names you suggested, added an explicit type on the hash value (so it's sha256:<value>), and expanded the usage/docs a little.

I'll update the PR body with the new flags as well.

This change adds a `k8s.io/kubernetes/cmd/kubeadm/app/util/pubkeypin` package which implements x509 public key pinning in the style of RFC7469. This is the public key hash format used by the new `kubeadm join --discovery-token-ca-cert-hash` flag. Hashes are namespaced with a short type, with "sha256" being the only currently-supported format. Type "sha256" is a hex-encoded SHA-256 hash over the Subject Public Key Info (SPKI) object in DER-encoded ASN.1.

This change adds the `--discovery-token-ca-cert-hash` and `--discovery-token-unsafe-skip-ca-verification` flags for `kubeadm join` and corresponding fields on the kubeadm NodeConfiguration struct. These flags configure enhanced TLS validation for token-based discovery. The enhanced TLS validation works by pinning the public key hashes of the cluster CA. This is done by connecting to the `cluster-info` endpoint initially using an unvalidated/unsafe TLS connection. After the cluster info has been loaded, parsed, and validated with the existing symmetric signature/MAC scheme, the root CA is validated against the pinned public key set. A second request is made using validated/safe TLS using the newly-known CA and the result is validated to make sure the same `cluster-info` was returned from both requests. This validation prevents a class of attacks where a leaked bootstrap token (such as from a compromised worker node) allows an attacker to impersonate the API server. This change also update `kubeadm init` to print the correct `--discovery-token-ca-cert-hash` flag in the example `kubeadm join` command it prints at the end of initialization.

…is/kubeadm` and `k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha1`.

mikedanese · 2017-08-10T17:35:33Z

/lgtm

k8s-github-robot · 2017-08-10T17:35:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ericchiang, luxas, mattmoyer, mikedanese

Associated issue: 130

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cmd/kubeadm/OWNERS~~ [luxas,mikedanese]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

mikedanese · 2017-08-10T18:34:59Z

/retest

k8s-github-robot · 2017-08-10T22:17:43Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-08-10T23:09:53Z

Automatic merge from submit-queue

This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520.

This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps.

) This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps.

@chenopis

* GC now supports non-core resources * Add two examples about how to analysis audits of kube-apiserver (#4264) * Deprecate system:nodes binding * [1.8] StatefulSet `initialized` annotation is now ignored. * inits the kubeadm upgrade docs addresses /issues/4689 * adds kubeadm upgrade cmd to ToC addresses /issues/4689 * add workload placement docs * ScaleIO - document udpate for 1.8 * Add documentation on storageClass.mountOptions and PV.mountOptions (#5254) * Add documentation on storageClass.mountOptions and PV.mountOptions * convert notes into callouts * Add docs for CustomResource validation add info about supported fields * advanced audit beta features (#5300) * Update job workload doc with backoff failure policy (#5319) Add to the Jobs documentation how to use the new backoffLimit field that limit the number of Pod failure before considering the Job as failed. * Documented additional AWS Service annotations (#4864) * Add device plugin doc under concepts/cluster-administration. (#5261) * Add device plugin doc under concepts/cluster-administration. * Update device-plugins.md * Update device-plugins.md Add meta description. Fix typo. Change bare metal deployment to manual deployment. * Update device-plugins.md Fix typo again. * Update page.version. (#5341) * Add documentation on storageClass.reclaimPolicy (#5171) * [Advanced audit] use new herf for audit-api (#5349) This tag contains all the changes in v1beta1 version. Update it now. * Added documentation around creating the InitializerConfiguration for the persistent volume label controller in the cloud-controller-manager (#5255) * Documentation for kubectl plugins (#5294) * Documentation for kubectl plugins * Update kubectl-plugins.md * Update kubectl-plugins.md * Updated CPU manager docs to match implementation. (#5332) * Noted limitation of alpha static cpumanager. * Updated CPU manager docs to match implementation. - Removed references to CPU pressure node condition and evictions. - Added note about new --cpu-manager-reconcile-period flag. - Added note about node allocatable requirements for static policy. - Noted limitation of alpha static cpumanager. * Move cpu-manager task link to rsc mgmt section. * init containers annotation removed in 1.8 (#5390) * Add documentation for TaintNodesByCondition (#5352) * Add documentation for TaintNodesByCondition * Update nodes.md * Update taint-and-toleration.md * Update daemonset.md * Update nodes.md * Update taint-and-toleration.md * Update daemonset.md * Fix deployments (#5421) * Document extended resources and OIR deprecation. (#5399) * Document extended resources and OIR deprecation. * Updated extended resources doc per reviews. * reverts extra spacing in _data/tasks.yml * addresses `kubeadm upgrade` review comments Feedback from @chenopis, @luxas, and @steveperry-53 addressed with this commit * HugePages documentation (#5419) * Update cpu-management-policies.md (#5407) Fixed the bad link. Modified "cpu" to "CPU". Added more 'yaml' as supplement. * Update RBAC docs for v1 (#5445) * Add user docs for pod priority and preemption (#5328) * Add user docs for pod priority and preemption * Update pod-priority-preemption.md * More updates * Update docs/admin/kubeadm.md for 1.8 (#5440) - Made a couple of minor wording changes (not strictly 1.8 related). - Did some reformatting (not strictly 1.8 related). - Updated references to the default token TTL (was infinite, now 24 hours). - Documented the new `--discovery-token-ca-cert-hash` and `--discovery-token-unsafe-skip-ca-verification` flags for `kubeadm join`. - Added references to the new `--discovery-token-ca-cert-hash` flag in all the default examples. - Added a new _Security model_ section that describes the security tradeoffs of the various discovery modes. - Documented the new `--groups` flag for `kubeadm token create`. - Added a note of caution under _Automating kubeadm_ that references the _Security model_ section. - Updated the component version table to drop 1.6 and add 1.8. - Update `_data/reference.yml` to try to get the sidebar fixed up and more consistent with `kubefed`. * Update StatefulSet Basics for 1.8 release (#5398) * addresses `kubeadm upgrade` review comments 2nd iteration review comments by @luxas * adds kubelet upgrade section to kubeadm upgrade * Fix a bulleted list on docs/admin/kubeadm.md. (#5458) I updated this doc yesterday and I was absolutely sure I fixed this, but I just saw that this commit got lost somehow. This was introduced recently in #5440. * Clarify the API to check for device plugins * Moving Flexvolume to separate out-of-tree section * addresses `kubeadm upgrade` review comments CC: @luxas * fixes kubeadm upgrade index * Update Stackdriver Logging documentation (#5495) * Re-update WordPress and MySQL PV doc to use apps/v1beta2 APIs (#5526) * Update statefulset concepts doc to use apps/v1beta2 APIs (#5420) * add document on kubectl's behavior regarding initializers (#5505) * Update docs/admin/kubeadm.md to cover self-hosting in 1.8. (#5497) This is a new beta feature in 1.8. * Update kubectl patch doc to use apps/v1beta2 APIs (#5422) * [1.8] Update "Run Applications" tasks to apps/v1beta2. (#5525) * Update replicated stateful application task for 1.8. * Update single instance stateful app task for 1.8. * Update stateless app task for 1.8. * Update kubectl patch task for 1.8. * fix the link of persistent storage (#5515) * update the admission-controllers.md index.md what-is-kubernetes.md link * fix the link of persistent storage * Add quota support for local ephemeral storage (#5493) * Add quota support for local ephemeral storage update the doc to this alpha feature * Update resource-quotas.md * Updated Deployments concepts doc (#5491) * Updated Deployments concepts doc * Addressed comments * Addressed more comments * Modify allocatable storage to ephemeral-storage (#5490) Update the doc to use ephemeral-storage instead of storage * Revamped concepts doc for ReplicaSet (#5463) * Revamped concepts doc for ReplicaSet * Minor changes to call out specific versions for selector defaulting and immutability * Addressed doc review comments * Remove petset documentations (#5395) * Update docs to use batch/v1beta1 cronjobs (#5475) * add federation job doc (#5485) * add federation job doc * Update job.md Edits for clarity and consistency * Update job.md Fixed a typo * update DaemonSet concept for 1.8 release (#5397) * update DaemonSet concept for 1.8 release * Update daemonset.md Fix typo. than -> then * Update bootstrap tokens doc for 1.8. (#5479) * Update bootstrap tokens doc for 1.8. This has some changes I missed when I was updating the main kubeadm documention: - Bootstrap tokens are now beta, not alpha (kubernetes/enhancements#130) - The apiserver flag to enable the authenticator changedin 1.8 (kubernetes/kubernetes#51198) - Added `auth-extra-groups` documentaion (kubernetes/kubernetes#50933) - Updated the _Token Management with `kubeadm`_ section to link to the main kubeadm docs, since it was just duplicated information. * Update bootstrap-tokens.md * Updated the Cassandra tutorial to use apps/v1beta2 (#5548) * add docs for AllowPrivilegeEscalation (#5448) Signed-off-by: Jess Frazelle <acidburn@microsoft.com> * Add local ephemeral storage alpha feature in managing compute resource (#5522) * Add local ephemeral storage alpha feature in managing compute resource Since 1.8, we add the local ephemeral storage alpha feature as one resource type to manage. Add this feature into the doc. * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Added documentation for Metrics Server (#5560) * authorization: improve authorization debugging docs (#5549) * Document mount propagation (#5544) * Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8. (#5524) This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in kubernetes/kubernetes#49520 and some version bumps. * Add task doc for alpha dynamic kubelet configuration (#5523) * Fix input/output of selfsubjectaccess review (#5593) * Add docs for implementing resize (#5528) * Add docs for implementing resize * Update admission-controllers.md * Added link to PVC section * minor typo fixes * Update NetworkPolicy concept guide with egress and CIDR changes (#5529) * update zookeeper tutorial for 1.8 release * add doc for hostpath type (#5503) * Federated Hpa feature doc (#5487) * Federated Hpa feature doc * Federated Hpa feature doc review fixes * Update hpa.md * Update hpa.md * update cloud controller manager docs for v1.8 * Update cronjob with defaults information (#5556) * Kubernetes 1.8 reference docs (#5632) * Kubernetes 1.8 reference docs * Kubectl reference docs for 1.8 * Update side bar with 1.8 kubectl and api ref docs links * remove petset.md * update on state of HostAlias in 1.8 with hostNetwork Pod support (#5644) * Fix cron job deletion section (#5655) * update imported docs (#5656) * Add documentation for certificate rotation. (#5639) * Link to using kubeadm page * fix the command output fix the command output * fix typo in api/resources reference: "Worloads" * Add documentation for certificate rotation. * Create TOC entry for cloud controller manager. (#5662) * Updates for new versions of API types * Followup 5655: fix link to garbage collection (#5666) * Temporarily redirect resources-reference to api-reference. (#5668) * Update config for 1.8 release. (#5661) * Update config for 1.8 release. * Address reviewer comments. * Switch references in HPA docs from alpha to beta (#5671) The HPA docs still referenced the alpha version. This switches them to talk about v2beta1, which is the appropriate version for Kubernetes 1.8 * Deprecate openstack heat (#5670) * Fix typo in pod preset conflict example Move container port definition to the correct line. * Highlight openstack-heat provider deprecation The openstack-heat provider for kube-up is being deprecated and will be removed in a future release. * Temporarily fix broken links by redirecting. (#5672) * Fix broken links. (#5675) * Fix render of code block (#5674) * Fix broken links. (#5677) * Add a small note about auto-bootstrapped CSR ClusterRoles (#5660) * Update kubeadm install doc for v1.8 (#5676) * add draft workloads api content for 1.8 (#5650) * add draft workloads api content for 1.8 * edits per review, add tables, for 1.8 workloads api doc * fix typo * Minor fixes to kubeadm 1.8 upgrade guide. (#5678) - The kubelet upgrade instructions should be done on every host, not just worker nodes. - We should just upgrade all packages, instead of calling out kubelet specifically. This will also upgrade kubectl, kubeadm, and kubernetes-cni, if installed. - Draining nodes should also ignore daemonsets, and master errors can be ignored. - Make sure that the new kubeadm download is chmoded correctly. - Add a step to run `kubeadm version` to verify after downloading. - Manually approve new kubelet CSRs if rotation is enabled (known issue). * Release 1.8 (#5680) * Fix versions for 1.8 API ref docs * Updates for 1.8 kubectl reference docs * Kubeadm /docs/admin/kubeadm.md cleanup, editing. (#5681) * Update docs/admin/kubeadm.md (mostly 1.8 related). This is Fabrizio's work, which I'm committing along with my edits (in a commit on top of this). * A few of my own edits to clarify and clean up some Markdown.

kubeadm got a new option in version 1.8: --discovery-token-unsafe-skip-ca-verification See: kubernetes/kubernetes#49520 Since version 1.9, it became mandatory to either use it to skip verification, or to use --discovery-token-ca-cert-hash=... See: kubernetes/kubernetes#55468 Since kube-spawn is a developer tool used on one physical machine, use the former.

k8s-ci-robot requested review from ericchiang, jbeda and luxas July 24, 2017 22:35

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 24, 2017

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 24, 2017

k8s-github-robot assigned errordeveloper and justinsb Jul 24, 2017

k8s-github-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jul 24, 2017

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 24, 2017

ericchiang reviewed Jul 24, 2017

View reviewed changes

luxas reviewed Jul 25, 2017

View reviewed changes

k8s-ci-robot added the sig/auth Categorizes an issue or PR as relevant to SIG Auth. label Jul 25, 2017

jbeda approved these changes Jul 25, 2017

View reviewed changes

luxas assigned jbeda, timothysc, ericchiang and luxas and unassigned justinsb and errordeveloper Jul 26, 2017

luxas added this to the v1.8 milestone Jul 26, 2017

timothysc reviewed Jul 26, 2017

View reviewed changes

mattmoyer force-pushed the bootstrap-pinning branch from bf8b416 to b17e3f8 Compare August 6, 2017 17:49

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2017

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 10, 2017

mattmoyer added 3 commits August 10, 2017 11:37

kubeadm: generated deepcopy for `k8s.io/kubernetes/cmd/kubeadm/app/ap…

358806e

…is/kubeadm` and `k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha1`.

mattmoyer force-pushed the bootstrap-pinning branch from b17e3f8 to 358806e Compare August 10, 2017 16:43

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 10, 2017

mikedanese added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. labels Aug 10, 2017

k8s-ci-robot assigned mikedanese Aug 10, 2017

k8s-github-robot merged commit b42a7b3 into kubernetes:master Aug 10, 2017

mattmoyer deleted the bootstrap-pinning branch August 11, 2017 14:18

mattmoyer mentioned this pull request Sep 18, 2017

Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8. kubernetes/website#5524

Merged

alban mentioned this pull request Nov 26, 2017

kubeadm join: fix TLS on Kubernetes >=1.9 kinvolk/kube-spawn#222

Merged

kubeadm: enhanced TLS validation for token-based discovery in kubeadm join #49520

kubeadm: enhanced TLS validation for token-based discovery in kubeadm join #49520

Conversation

mattmoyer commented Jul 24, 2017 • edited Loading

Example from kubeadm init:

Example from kubeadm join:

k8s-ci-robot commented Jul 24, 2017

ericchiang commented Jul 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas commented Jul 25, 2017

ericchiang commented Jul 25, 2017

ericchiang commented Jul 25, 2017

mattmoyer commented Jul 25, 2017

jbeda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmoyer commented Jul 25, 2017

timothysc left a comment

Choose a reason for hiding this comment

mattmoyer commented Jul 26, 2017

mattmoyer commented Aug 6, 2017

mikedanese commented Aug 10, 2017

k8s-github-robot commented Aug 10, 2017

mikedanese commented Aug 10, 2017

k8s-github-robot commented Aug 10, 2017

k8s-github-robot commented Aug 10, 2017

kubeadm: enhanced TLS validation for token-based discovery in `kubeadm join` #49520

kubeadm: enhanced TLS validation for token-based discovery in `kubeadm join` #49520

mattmoyer commented Jul 24, 2017 •

edited

Loading

Example from `kubeadm init`:

Example from `kubeadm join`: