Skip to content

Patch does not apply server-side defaults to resulting object #42764

New issue
New issue
Closed
Closed
Patch does not apply server-side defaults to resulting object#42764
Assignees
liggittwojtek-t
Labels
kind/bugCategorizes issue or PR as related to a bug.release-blockersig/api-machineryCategorizes an issue or PR as relevant to SIG API Machinery.
Milestone
 
v1.6
@skriss

Description

@skriss
skriss
opened on Mar 8, 2017

Found when looking into DNS configMap upgrade-test errors

@bowei - can you take a look at the following and let me know if this is an issue for the 1.6 release or not? Thanks!

There are two tests failing around DNS configMap in the following test job:

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.5-gci-1.6-upgrade-cluster-new/18

For context, this job is creating a 1.5 cluster, upgrading it to 1.6, then running the 1.6 E2E tests on it to ensure it acts like a fresh 1.6 cluster.

Activity

skriss

skriss commented on Mar 8, 2017

@skriss
skriss
on Mar 8, 2017
ContributorAuthor

/assign @bowei

k8s-ci-robot
assigned
bowei
on Mar 8, 2017
liggitt

liggitt commented on Mar 9, 2017

@liggitt
liggitt
on Mar 9, 2017
Member

Is this a dupe of #42746 or are the nodes upgraded as well?

skriss

skriss commented on Mar 9, 2017

@skriss
skriss
on Mar 9, 2017
ContributorAuthor

@liggitt not a dupe - this test job upgrades nodes as well, and the failure mode is different

skriss

skriss commented on Mar 9, 2017

@skriss
skriss
on Mar 9, 2017
ContributorAuthor

@liggitt @bowei is someone looking at this? should it be routed to someone else?

bowei

bowei commented on Mar 9, 2017

@bowei
bowei
on Mar 9, 2017
Member

Looking --

bowei

bowei commented on Mar 9, 2017

@bowei
bowei
on Mar 9, 2017
Member

Let keep this as a 1.6 issue for now. The run does not seem to have incorporated the ConfigMap change, it should be independent, but I am still investigating why the failure occurred.

MrHohn

MrHohn commented on Mar 10, 2017

@MrHohn
MrHohn
on Mar 10, 2017
Member

From the failure above, seems like kube-dns pod was not upgraded to the newer version after upgrade:

I0308 01:22:10.127] [k8s.io] DNS configMap federations 
...
I0308 01:22:10.281] Mar  8 01:22:10.281: INFO: Using DNS pod: kube-dns-4101612645-2xknf
...
I0308 01:24:17.655] Mar  8 01:24:17.653: INFO: kube-dns-4101612645-2xknf started at 2017-03-07 23:31:28 -0800 PST (0+4 container statuses recorded)
I0308 01:24:17.655] Mar  8 01:24:17.653: INFO: 	Container dnsmasq ready: true, restart count 0
I0308 01:24:17.655] Mar  8 01:24:17.653: INFO: 	Container dnsmasq-metrics ready: true, restart count 0
I0308 01:24:17.655] Mar  8 01:24:17.653: INFO: 	Container healthz ready: true, restart count 0
I0308 01:24:17.655] Mar  8 01:24:17.653: INFO: 	Container kubedns ready: true, restart count 0

Note that we already removed the healthz container from kube-dns in v1.6.

MrHohn

MrHohn commented on Mar 10, 2017

@MrHohn
MrHohn
on Mar 10, 2017 · edited by MrHohn
Member

Found more hints from the equivalent gce upgrade test: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster-new/8

Apparently addon-manager is not working properly and it failed to update kube-dns (logs file). The cause for its failures was that kubectl apply hit some weird validation errors as below:

INFO: == Reconciling with deprecated label ==
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"addonmanager.kubernetes.io/mode\":\"Reconcile\",\"k8s-app\":\"kube-dns\",\"kubernetes.io/cluster-service\":\"true\"},\"name\":\"kube-dns\",\"namespace\":\"kube-system\"},\"spec\":{\"selector\":{\"matchLabels\":{\"k8s-app\":\"kube-dns\"}},\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"10%\",\"maxUnavailable\":0}},\"template\":{\"metadata\":{\"annotations\":{\"scheduler.alpha.kubernetes.io/critical-pod\":\"\"},\"labels\":{\"k8s-app\":\"kube-dns\"}},\"spec\":{\"containers\":[{\"args\":[\"--domain=cluster.local.\",\"--dns-port=10053\",\"--config-dir=/kube-dns-config\",\"--v=2\"],\"env\":[{\"name\":\"PROMETHEUS_PORT\",\"value\":\"10055\"}],\"image\":\"gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1\",\"livenessProbe\":{\"failureThreshold\":5,\"httpGet\":{\"path\":\"/healthcheck/kubedns\",\"port\":10054,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":60,\"successThreshold\":1,\"timeoutSeconds\":5},\"name\":\"kubedns\",\"ports\":[{\"containerPort\":10053,\"name\":\"dns-local\",\"protocol\":\"UDP\"},{\"containerPort\":10053,\"name\":\"dns-tcp-local\",\"protocol\":\"TCP\"},{\"containerPort\":10055,\"name\":\"metrics\",\"protocol\":\"TCP\"}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/readiness\",\"port\":8081,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":3,\"timeoutSeconds\":5},\"resources\":{\"limits\":{\"memory\":\"170Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"70Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/kube-dns-config\",\"name\":\"kube-dns-config\"}]},{\"args\":[\"-v=2\",\"-logtostderr\",\"-configDir=/etc/k8s/dns/dnsmasq-nanny\",\"-restartDnsmasq=true\",\"--\",\"-k\",\"--cache-size=1000\",\"--log-facility=-\",\"--server=/cluster.local/127.0.0.1#10053\",\"--server=/in-addr.arpa/127.0.0.1#10053\",\"--server=/ip6.arpa/127.0.0.1#10053\"],\"image\":\"gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1\",\"livenessProbe\":{\"failureThreshold\":5,\"httpGet\":{\"path\":\"/healthcheck/dnsmasq\",\"port\":10054,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":60,\"successThreshold\":1,\"timeoutSeconds\":5},\"name\":\"dnsmasq\",\"ports\":[{\"containerPort\":53,\"name\":\"dns\",\"protocol\":\"UDP\"},{\"containerPort\":53,\"name\":\"dns-tcp\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"150m\",\"memory\":\"20Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/etc/k8s/dns/dnsmasq-nanny\",\"name\":\"kube-dns-config\"}]},{\"args\":[\"--v=2\",\"--logtostderr\",\"--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A\",\"--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A\"],\"image\":\"gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1\",\"livenessProbe\":{\"failureThreshold\":5,\"httpGet\":{\"path\":\"/metrics\",\"port\":10054,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":60,\"successThreshold\":1,\"timeoutSeconds\":5},\"name\":\"sidecar\",\"ports\":[{\"containerPort\":10054,\"name\":\"metrics\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"10m\",\"memory\":\"20Mi\"}}}],\"dnsPolicy\":\"Default\",\"serviceAccountName\":\"kube-dns\",\"tolerations\":[{\"key\":\"CriticalAddonsOnly\",\"operator\":\"Exists\"}],\"volumes\":[{\"configMap\":{\"name\":\"kube-dns\",\"optional\":true},\"name\":\"kube-dns-config\"}]}}}}\n"},"creationTimestamp":null,"labels":{"addonmanager.kubernetes.io/mode":"Reconcile"}},"spec":{"template":{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/tolerations":null},"creationTimestamp":null},"spec":{"containers":[{"$patch":"delete","name":"dnsmasq-metrics"},{"$patch":"delete","name":"healthz"},{"args":["-v=2","-logtostderr","-configDir=/etc/k8s/dns/dnsmasq-nanny","-restartDnsmasq=true","--","-k","--cache-size=1000","--log-facility=-","--server=/cluster.local/127.0.0.1#10053","--server=/in-addr.arpa/127.0.0.1#10053","--server=/ip6.arpa/127.0.0.1#10053"],"image":"gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1","livenessProbe":{"httpGet":{"path":"/healthcheck/dnsmasq","port":10054}},"name":"dnsmasq","resources":{"requests":{"memory":"20Mi"}},"volumeMounts":[{"mountPath":"/etc/k8s/dns/dnsmasq-nanny","name":"kube-dns-config"}]},{"args":["--domain=cluster.local.","--dns-port=10053","--config-dir=/kube-dns-config","--v=2"],"image":"gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1","livenessProbe":{"httpGet":{"path":"/healthcheck/kubedns","port":10054}},"name":"kubedns","volumeMounts":[{"mountPath":"/kube-dns-config","name":"kube-dns-config"}]},{"args":["--v=2","--logtostderr","--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A","--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A"],"image":"gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1","livenessProbe":{"failureThreshold":5,"httpGet":{"path":"/metrics","port":10054,"scheme":"HTTP"},"initialDelaySeconds":60,"successThreshold":1,"timeoutSeconds":5},"name":"sidecar","ports":[{"containerPort":10054,"name":"metrics","protocol":"TCP"}],"resources":{"requests":{"cpu":"10m","memory":"20Mi"}}}],"serviceAccountName":"kube-dns","tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"}],"volumes":[{"configMap":{"name":"kube-dns","optional":true},"name":"kube-dns-config"}]}}},"status":null}
to:
&{0xc4200ac3c0 0xc420041490 kube-system kube-dns /etc/kubernetes/addons/dns/kubedns-controller.yaml 0xc420a4b2a8 0xc420a4bce0 79456 false}
for: "/etc/kubernetes/addons/dns/kubedns-controller.yaml": Deployment.apps "kube-dns" is invalid: [spec.template.spec.containers[2].terminationMessagePolicy: Required value: must be 'File' or 'FallbackToLogsOnError', spec.template.spec.containers[2].imagePullPolicy: Required value]
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"addonmanager.kubernetes.io/mode\":\"Reconcile\",\"k8s-app\":\"kube-dns\",\"kubernetes.io/cluster-service\":\"true\",\"kubernetes.io/name\":\"KubeDNS\"},\"name\":\"kube-dns\",\"namespace\":\"kube-system\"},\"spec\":{\"clusterIP\":\"10.0.0.10\",\"ports\":[{\"name\":\"dns\",\"port\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"port\":53,\"protocol\":\"TCP\"}],\"selector\":{\"k8s-app\":\"kube-dns\"}}}\n"},"creationTimestamp":null,"labels":{"addonmanager.kubernetes.io/mode":"Reconcile"}},"spec":{"ports":[{"port":53,"targetPort":null},{"port":53,"targetPort":null}]},"status":null}
to:
&{0xc420857140 0xc420478e70 kube-system kube-dns /etc/kubernetes/addons/dns/kubedns-svc.yaml 0xc4209733d8 0xc4209735d0 144 false}
for: "/etc/kubernetes/addons/dns/kubedns-svc.yaml": Service "kube-dns" is invalid: spec.ports[0].targetPort: Invalid value: 0: must be between 1 and 65535, inclusive

And I could not reproduce these errors locally with v1.6.0-beta.1 kubectl against either v1.5.3 or v1.6+ cluster.

@kubernetes/sig-cli-bugs

Updates:
FYI the kubectl binary this addon-manager used is built with commit 17375fc (earlier than v1.6.0-beta.1) but I could not reproduce the error with this binary either. So it feels like the validation errors happened on the apiserver side?

mengqiy

mengqiy commented on Mar 10, 2017

@mengqiy
mengqiy
on Mar 10, 2017 · edited by mengqiy
Member

I think this issue have been fixed by #42489. And it is in 1.6.0-beta.2.
The root cause is #42488

EDIT: This only fix the second error

Service "kube-dns" is invalid: spec.ports[0].targetPort: Invalid value: 0: must be between 1 and 65535, inclusive
MrHohn

MrHohn commented on Mar 10, 2017

@MrHohn
MrHohn
on Mar 10, 2017 · edited by MrHohn
Member

Updates: I reproduced the first error with kubectl v1.6.0-beta.2 and the latest k8s cluster when try to kubectl apply -f an older version kube-dns deployment against the latest one:

$ ./kubectl-v1.6.0-beta.2 apply -f kubedns-controller-gke1.5.yaml 
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
The Deployment "kube-dns" is invalid: 
* spec.template.spec.containers[3].terminationMessagePolicy: Required value: must be 'File' or 'FallbackToLogsOnError'
* spec.template.spec.containers[4].terminationMessagePolicy: Required value: must be 'File' or 'FallbackToLogsOnError'

Another update: And with kubectl v1.5.2:

$ ./kubectl-v1.5.2 apply -f kubedns-controller-gke1.5.yaml                                                                                                           
The Deployment "kube-dns" is invalid: 
* spec.template.spec.containers[3].terminationMessagePolicy: Required value: must be 'File' or 'FallbackToLogsOnError'
* spec.template.spec.containers[4].terminationMessagePolicy: Required value: must be 'File' or 'FallbackToLogsOnError'
liggitt

liggitt commented on Mar 10, 2017

@liggitt
liggitt
on Mar 10, 2017
Member

huh, that almost seems like defaults aren't getting applied after the patch is applied

bowei

bowei commented on Mar 10, 2017

@bowei
bowei
on Mar 10, 2017
Member

@ymqytw -- as this seems to be an issue with the API object patch mechanism, I am going to assign to you for now... You can assign it back if this continues to fail with the fixes mentioned above.

/assign @ymqytw

k8s-ci-robot
assigned
mengqiy
on Mar 10, 2017
liggitt

liggitt commented on Mar 10, 2017

@liggitt
liggitt
on Mar 10, 2017 · edited by liggitt
Member

@wojtek-t is it possible the switch to unstructured.DefaultConverter.FromUnstructured in applyPatchToObject dropped the defaulting step of conversion?

22 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.release-blockersig/api-machineryCategorizes an issue or PR as relevant to SIG API Machinery.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions
    Patch does not apply server-side defaults to resulting object · Issue #42764 · kubernetes/kubernetes