/assign @bowei

/lgtm
/approve

/ok-to-test

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

Does this need to be updated for kubeadm as well?

/milestone v1.12

/sig network

Does this need to be updated for kubeadm as well?

for ref:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/constants/constants.go#L302

i see the multi arch images are already up at gcr, but there are no manifest lists in place.

also this PR only bumps for amd64:

image: k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.11

@neolit123 if i remember right since CoreDNS feature was marked GA and kube-dns was kinda deprecated ... the build harness was not updated to generate manifests:
https://github.com/kubernetes/dns/tree/master/images/dnsmasq

@mkumatag can you please confirm?

kube-dns is not deprecated yet.

@bowei sorry i used wrong words :)

@neolit123 if i remember right since CoreDNS feature was marked GA and kube-dns was kinda deprecated ... the build harness was not updated to generate manifests:
https://github.com/kubernetes/dns/tree/master/images/dnsmasq

@mkumatag can you please confirm?

you are right! kube-dns is not yet multi-arched.

kube-dns not being multi-arch can break for a user that wants to switch from core-dns to kube-dns on a cluster that is not only amd64 via the CoreDNS feature gate.

two options:

• add schema2s for kube-dns until it's deprecated.
• not support anything but amd64 and document it. this is worse because bug reports can spawn and these are not that obvious to debug.

What does multi-arched mean? We build kube-dns for amd64 arm arm64 ppc64le s390x.

@prameshj @MrHohn Here's a note from @neolit123 on slack

neolit123 [9:11 PM]
there is something fishy about the manifest lists for kube-dns 1.14.12.

https://console.cloud.google.com/gcr/images/google-containers/GLOBAL/k8s-dns-kube-dns?gcrImageListsize=50
https://console.cloud.google.com/gcr/images/google-containers/GLOBAL/k8s-dns-kube-dns-amd64?gcrImageListsize=50
the manifest list for 1.14.12 is there, but say the amd64 1.14.12 is missing.

@dims I think we were confused about how the multi-arched images work.

https://pantheon.corp.google.com/gcr/images/google-containers/GLOBAL/k8s-dns-kube-dns contains:

• Manifest list which includes all supprted archs (amd64, arm, arm64, ppc64le, s390x)
• Docker Manifest for each arch.

Is that not enough? Do we need to push images to k8s-dns-kube-dns-{arch} as well?

@MrHohn
please have a look at how the kube-apiserver manifest list and related images are defined at GCR:
https://console.cloud.google.com/gcr/images/google-containers/GLOBAL/kube-apiserver?gcrImageListsize=50

those are correct.

Okay, seems like kube-apiserver pushes the same image to both --- kube-apiserver and kube-apiserver-{arch}. We will do the same for kube-dns then.

thanks @MrHohn yes, all the individual {arch} images and the manifest (without the arch)

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

@dims Do we need the individual {arch} images to support docker pull commands like k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.12 ?
I had tried downloading images without the suffix and it worked. Wasn't sure whether to push the individual images, so followed example of coredns which has only one image name, but the other arch images have different tags. How do we decide when the individual images are needed and when they aren't?

We will do the same for kube-dns then.

@dims @neolit123 All the individual {arch} images should have been pushed now. e.g. https://pantheon.corp.google.com/gcr/images/google-containers/GLOBAL/k8s-dns-kube-dns-amd64

@prameshj @MrHohn thanks!

@prameshj we are in that weird zone where we are getting things to work but not really told everyone to switch over to manifests. So for now since there are many tools other than kubeadm, we should just publish what we used to publish before AND the new manifest. when we get to v1.13 we can make a stronger statement around requiring manifests all the time (may be).

Yea, coredns was ahead of us if i remember right. So let's follow what i outlined above.

( cc @mkumatag )

@MrHohn
thank you, i have just verified that the 1.14.12 images and manifests are all there!

@MrHohn The image is not correctly built... Similar issue with #67027 and #65253.

Please use docker 18.06 to build the image if you want multi-arch support.

The image is not correctly built...

That is unfortunately :/ Working with @Random-Liu now to repush the images.

Thanks @Random-Liu @MrHohn !

@Random-Liu thanks for the heads up on the issue, indeed!
i'm working on a e2e test to verify the manifests and the "digest vs size" verification is currently missing. i need to add that.

@neolit123 Thanks a lot!

@Random-Liu thanks for the heads up on the issue, indeed!
i'm working on a e2e test to verify the manifests and the "digest vs size" verification is currently missing. i need to add that.

we already have a check for making sure we are using docker 18.06 version - https://github.com/kubernetes/kubernetes/blob/master/test/images/image-util.sh#L100

@mkumatag
yep, the e2e was agreed upon regardless as an extra verification step.

@MrHohn
could you please update us when this is done, or perhaps drop a note in #sig-release ?
thank you.

More updates:

1. 1.14.12 images haven't be re-pushed yet and is pending on the discussion internally regarding whether or not we should re-push images with the same label.
2. Rebuild docker images to pick up Alpine 3.8.1 dns#263 just came in and we are building images with a new label (1.14.13).

Likely we will need another PR to bump kube-dns to 1.14.13, which should be correctly built and includes the RCE fix.

@MrHohn

1.14.12 images haven't be re-pushed yet and is pending on the discussion internally regarding whether or not we should re-push images with the same label.

there was a similar discussion for etcd and it was decided to bump revisions:
#59664
https://github.com/kubernetes/kubernetes/pull/59664/files#diff-ff03ae6284132dbd066dc35d350259c9R37

kubernetes/dns#263 just came in and we are building images with a new label (1.14.13).
Likely we will need another PR to bump kube-dns to 1.14.13, which should be correctly built and includes the RCE fix.

sounds good.

Likely we will need another PR to bump kube-dns to 1.14.13, which should be correctly built and includes the RCE fix.

Please get that submitted ASAP today!!!

Sent #68900.