Closed
Description
What happened:
rm -rf '/tmp/kubernetes-v1.21.0'
git clone --recurse-submodules --branch=v1.21 --depth=1 'https://github.com/kubernetes/kubernetes' '/tmp/kubernetes-v1.21.0'
kind build node-image --image 'kindest/node:v1.21.0' --kube-root '/tmp/kubernetes-v1.21.0'The kind command fails with:
Click here
Starting to build Kubernetes
+++ [0408 12:06:17] Verifying Prerequisites....
+++ [0408 12:06:17] Building Docker image kube-build:build-8406c34423-5-v1.16.1-1
+++ [0408 12:11:29] Creating data container kube-build-data-8406c34423-5-v1.16.1-1
+++ [0408 12:11:30] Syncing sources to container
+++ [0408 12:11:34] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0408 12:11:34] Running build command...
+++ [0408 12:11:42] Building go targets for linux/amd64:
./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
Generating prerelease lifecycle code for 27 targets
+++ [0408 12:11:45] Building go targets for linux/amd64:
./vendor/k8s.io/code-generator/cmd/deepcopy-gen
Generating deepcopy code for 227 targets
+++ [0408 12:11:52] Building go targets for linux/amd64:
./vendor/k8s.io/code-generator/cmd/defaulter-gen
Generating defaulter code for 89 targets
+++ [0408 12:12:00] Building go targets for linux/amd64:
./vendor/k8s.io/code-generator/cmd/conversion-gen
Generating conversion code for 124 targets
+++ [0408 12:12:18] Building go targets for linux/amd64:
./vendor/k8s.io/kube-openapi/cmd/openapi-gen
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [0408 12:12:29] Building go targets for linux/amd64:
./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0408 12:12:30] Building go targets for linux/amd64:
cmd/kubeadm
cmd/kubectl
cmd/kubelet
+++ [0408 12:13:06] Syncing out of container
+++ [0408 12:13:08] Verifying Prerequisites....
+++ [0408 12:13:09] Building Docker image kube-build:build-8406c34423-5-v1.16.1-1
+++ [0408 12:13:18] Syncing sources to container
+++ [0408 12:13:21] Running build command...
Generating prerelease lifecycle code for 27 targets
Generating deepcopy code for 227 targets
Generating defaulter code for 89 targets
Generating conversion code for 124 targets
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [0408 12:14:06] Building go targets for linux/amd64:
cmd/kube-apiserver
cmd/kube-controller-manager
cmd/kube-scheduler
cmd/kube-proxy
+++ [0408 12:14:24] Syncing out of container
+++ [0408 12:14:27] Building images: linux-amd64
+++ [0408 12:14:27] Starting docker build for image: kube-apiserver-amd64
+++ [0408 12:14:27] Starting docker build for image: kube-controller-manager-amd64
+++ [0408 12:14:27] Starting docker build for image: kube-scheduler-amd64
+++ [0408 12:14:27] Starting docker build for image: kube-proxy-amd64
unknown shorthand flag: 'f' in -f
See 'docker --help'.
Usage: docker [OPTIONS] COMMAND
A self-sufficient runtime for containers
Options:
--config string Location of client config files (default
"/home/arnavion/.docker")
-c, --context string Name of the context to use to connect to the
daemon (overrides DOCKER_HOST env var and
default context set with "docker context use")
-D, --debug Enable debug mode
-H, --host list Daemon socket(s) to connect to
-l, --log-level string Set the logging level
("debug"|"info"|"warn"|"error"|"fatal")
(default "info")
--tls Use TLS; implied by --tlsverify
--tlscacert string Trust certs signed only by this CA (default
"/home/arnavion/.docker/ca.pem")
--tlscert string Path to TLS certificate file (default
"/home/arnavion/.docker/cert.pem")
--tlskey string Path to TLS key file (default
"/home/arnavion/.docker/key.pem")
--tlsverify Use TLS and verify the remote
-v, --version Print version information and quit
[...]
Something is calling docker -f ... , preumably it meant to call docker <subcommand> -f
What you expected to happen:
Expected it to succeed.
How to reproduce it (as minimally and precisely as possible):
As above.
Anything else we need to know?:
This works with other Kubernetes versions from 1.14 through 1.20, so this is something 1.21-specific.
Environment:
- kind version: (use
kind version): kind v0.10.0 go1.15.7 linux/amd64 - Kubernetes version: (use
kubectl version): 1.21.0 - Docker version: (use
docker info): Server Version: 20.10.5-ce - OS (e.g. from
/etc/os-release): openSUSE Tumbleweed 20210330
Edit: strace indicates the problematic command is:
[pid 83376] execve("/usr/bin/docker", ["docker", "buildx", "build", "-f", "/tmp/kubernetes-v1.21.0/build/server-image/kube-apiserver/Dockerfile", "--platform", "linux/amd64", "--load", "--pull", "-t", "k8s.gcr.io/kube-apiserver-amd64:v1.21.0", "--build-arg", "BASEIMAGE=k8s.gcr.io/build-image/go-runner:v2.3.1-go1.16.1-buster.0", "--build-arg", "SETCAP_IMAGE=k8s.gcr.io/build-image/setcap:buster-v1.4.0", "--build-arg", "BINARY=kube-apiserver", "/tmp/kubernetes-v1.21.0/_output/release-stage/server/linux-amd64/kubernetes/server/bin/kube-apiserver.dockerbuild"], 0x556108f33680 /* 125 vars */ <unfinished ...>
ie
docker buildx build \
-f '/tmp/kubernetes-v1.21.0/build/server-image/kube-apiserver/Dockerfile' \
--platform linux/amd64 \
--load \
--pull \
-t 'k8s.gcr.io/kube-apiserver-amd64:v1.21.0' \
--build-arg 'BASEIMAGE=k8s.gcr.io/build-image/go-runner:v2.3.1-go1.16.1-buster.0' \
--build-arg 'SETCAP_IMAGE=k8s.gcr.io/build-image/setcap:buster-v1.4.0' \
--build-arg 'BINARY=kube-apiserver' \
'/tmp/kubernetes-v1.21.0/_output/release-stage/server/linux-amd64/kubernetes/server/bin/kube-apiserver.dockerbuild'Running that command by itself also has the same problem.
Edit 2: Okay, the problem is my distro-provided docker CLI does not have the buildx plugin. docker foo build -f produces this confusing error when foo can't be found as a plugin.
Is this something that can be fixed by kind (given it's new with 1.21 and was not needed for 1.20 and earlier) ? If not, it needs to be documented somewhere that this docker CLI plugin is required to build node images.
Activity
neolit123 commentedon Apr 8, 2021
yes, the plugin is required.
kind executes the kubernetes/kuberentes scripts for building the images and they use buildx.
https://github.com/kubernetes/kubernetes/search?q=buildx
i think for the time being the requirement should be documented in the kind node image docs.
neolit123 commentedon Apr 8, 2021
/remove-kind bug
/kind documentation
Arnavion commentedon Apr 8, 2021
Okay, so once I installed the buildx plugin I was able to build the node image, however it fails to start:
Click here
journalctlinside the container reveals:neolit123 commentedon Apr 8, 2021
related to #1969
see some of the linked kubernetes/kubernetes issues in there.
Arnavion commentedon Apr 8, 2021
I don't have this issue deploying a v1.20 cluster with kind 0.10.0, for which the
--cgroup-root=/kubeletparameter is also given to kubelet.Arnavion commentedon Apr 8, 2021
I guess v1.21 changed to require cgroups v2? That's the only thing that makes sense.
If that's the case, then yes my distro doesn't have the file it's looking for. The kubelet service unit has an
ExecStartPre(injected via/etc/systemd/system/kubelet.service.d/10-kubeadm.conf):which means the script is entirely skipped on cgroup v1 hosts. If the script was run, it would've produced a more descriptive error:
If v1.21 requires cgroups v2, it would be useful to run that script unguarded so that the error can be seen in the journal.
neolit123 commentedon Apr 8, 2021
could you please log a separate issue and provide the details there?
we can keep this one for the node image documentation.
Arnavion commentedon Apr 8, 2021
Done. #2189
BenTheElder commentedon Apr 9, 2021
1.21 requires KIND @ HEAD due to the breaking change in upstream Kubernetes.
Similarly the build requirement for buildx comes entirely from Kubernetes upstream.
Cgroups v2 is not required, and that script is only for v2.
BenTheElder commentedon Apr 9, 2021
The breaking cgroups change is kubeadm defaulting kubelet to systemd cgroup driver instead of cgroupsfs in 1.21 which is an action required change for cluster administrators.
Kind handles this, but not in v0.10 as when v0.10 was released this was not necessary / known.
17 remaining items