@dhawal55 There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

/sig cli

Same issue report in kubernetes/kubectl#27. I believe Strategic Merge Patch works kubernetes/kubectl#27 (comment). It seems not a cli issue.
I think this an api machinery issue. @kubernetes/sig-api-machinery-misc

Do we have the same issue in 1.7?

cc/ @kubernetes/kubernetes-release-managers

Yes.

seems according to the doc, only "Changes to the Init Container spec are limited to the container image field.", IIUC, kubernetes/kubectl#27 is trying to update the image field, if @mengqiy you are not working on this, I can help take a look at the issue.

@xingzhou Thanks for triaging the issue.

@erictune, I think the root cause of this issue is from these lines of code here. While when we try to update a pod object, we need to convert from v1.Pod to api.Pod, and the code in conversion function always overwrite the values in pod.Spec.InitContainers with the value of annotation pod.beta.kubernetes.io/init-containers. Therefore, when user changes the values in pod.Spec.InitContainers, it won't work.

One solution is to move the logic in the conversion function to pod's storage methods like create/update, in this case, at least we can realize that user is modifying the fields in pod.Spec.InitContainers. What's your idea on this?

@xingzhou Is there any workaround?

@JulianWei you can just go back to specifying your init containers using the beta annotation (pod.beta.kubernetes.io/init-containers), it works correctly. When this bug is fixed, you will be able to use the stable InitContainers field. A bit annoying, but easy to fix :)

@MartinodF, thank you

I don't know if this is relevant or not, but it appears that if you never used annotations at all and have only ever used initContainers in your yaml, apply doesn't seem to touch that, either.

The beta annotations do not work for us. We've even tried to delete our daemonset and and only use the beta annotations but changes don't get noticed.

Is there any way this could be fixed for 1.7.x? Due to this bug, on 1.7 and earlier, the only reliable way to use init containers is to use the pod.beta.kubernetes.io/init-containersannotation, not the newer initContainers field. But in 1.8 (as of 6ec80ea), that annotation is going away. That means that a given pod spec can't be compatible with both 1.7 (where the annotation is required) and 1.8 (where the annotation is unsupported), without duplicating the init containers.

Fixing this in 1.7.x -- so the initContainers field is supported for updates -- would make the migration smoother, as the initContainers field could be used always. Is it too late to get this narrow problem fixed in the next 1.7 release?

Unfortunately, the annotations have to take precedence as long as 1.5 kubelets are supported (which is through 1.7.x), or updates from those kubelets would not correctly update the objects. 1.8 is the earliest that behavior could be dropped, and was why 6ec80ea was possible.

That conflict between honoring the earliest API representation (via annotations) to preserve skewed compatibility and having an API that isn't insane to use is why annotations are no longer used as an alpha/beta-stage mechanism for new fields.

fixed in #51816

So do I understand correctly that currently the safe way is to always duplicate the init container between the annotation and the actual field, and then after upgrading to 1.8 drop the annotation?

So do I understand correctly that currently the safe way is to always duplicate the init container between the annotation and the actual field, and then after upgrading to 1.8 drop the annotation?

Correct. Pre-1.8, the information is duplicated in two places in the object, with the annotation taking precedence. 1.8+, only the field is honored. You should set the field for forward compatibility, and set the annotation if you want your changes to be effective against a pre-1.8 server.

Rather than manually duplicating the same content to multiple places, I've found that if you explicitly set both the alpha and the beta annotations to null, you can get the desired behavior out of the system in 1.7: initContainer updates will actually take effect and the actual API-level annotations will simply track the initContainers field.

Thanks @clhodapp -- that's super clever, and it works on 1.6 as well.

@clhodapp I'm working with @kevingessner and found today that your fix worked only for existing deployments and not for new ones on 1.6.
Just in case anyone else gets stuck on this, the error message is misleading (Error from server (BadRequest): error when creating "STDIN": Deployment in version "v1beta1" cannot be handled as a Deployment: unexpected end of JSON input) and the config still passes fine with --dry-run. So not backwards compatible for us, unfortunately.

Thanks for posting your finding, @linse. It sounds like the only real fix here is updating to 1.8 then.

I am still encountering this issue with Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.4-gke.0", GitCommit:"c08fe9548257472151a8fa298bd6c3e9ea1b8c09", GitTreeState:"clean", BuildDate:"2018-06-06T11:34:07Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

I have tried setting the alpha and beta annotations to null.

@linse, @clhodapp, @Noah-Huppert Actually, there is a way to avoid Deployment in version "v1beta1" cannot be handled as a Deployment: unexpected end of JSON input)
If there is no deployment created yet, you can remove:

annotations:
  pod.beta.kubernetes.io/init-containers: null
  pod.alpha.kubernetes.io/init-containers: null

and left only initContainers:, deployment will have been created without errors and init container will be there.
And if you already have deployment in your namespace you can left:

annotations:
  pod.beta.kubernetes.io/init-containers: null
  pod.alpha.kubernetes.io/init-containers: null

with initContainers: in your descriptor an your deployment will have been updated along with the init container.
Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.12", GitCommit:"3bda299a6414b4866f179921610d6738206a18fe", GitTreeState:"clean", BuildDate:"2017-12-29T09:00:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.12", GitCommit:"3bda299a6414b4866f179921610d6738206a18fe", GitTreeState:"clean", BuildDate:"2017-12-29T08:39:49Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}