init-container definition (non-anno.) persisting after removal from deployment #45627

gladiatr72 · 2017-05-11T00:46:53Z

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

initContainers, init-containers, init containers

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

bug

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T22:51:55Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:22:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

I was pleased to see the graduation of init-containers to GA. Unfortunately, there are two problems with how k8s is dealing with them:

Our use case involves a Django application component. The idea is to use init containers to deal with pre-flight checks and handle any pre-launch processing that needs to happen before the application comes online (typical use as per the docs)

Since these processes make use of the same image:version, it is necessary for the containers to run the same version of the app image.

Defining the init-containers in .template.spec.initContainers works with the initial deployment; however, any change to the deployment spec file within .template.spec.initContainers is not properly processed. Any subsequent applications to the deployment will cause the initial init-container definitions to run, but, of course, they're running the wrong version. This problem affects the entire init-container element definition, not just image.

Another interesting feature of this bug is that despite not having defined the init-containers in .template.metadata.annotations, the running deployment shows that the .template.spec.initContainers definitions have had their JSON equivalents packed into both pod.alpha.kubernetes.io/init-containers and pod.beta.kubernetes.io/init-containers annotations.

NOTE: At this point in my investigation, my original spec file has its init-containers defined only in template.spec.initContainers

At this point I'm thinking the API must be pre-processnigtemplate.spec.initContainers to create the annotation strings--not a typical GA feature feature, but whatever... I'll roll with it, right?

I decided to convert the init-container defs to JSON and see if the deployment problem goes away.

NOTE: At this point .template.spec.InitContainers is removed from the spec file

Defining and subsequently updating the init-containers as beta annotations work just like in 1.5. Functionally, I'm in good shape; however, I ask the API for its view of the deployment spec and .template.spec.initContainers is back!

Environment:

Cloud provider or hardware configuration:
gce
OS (e.g. from /etc/os-release):
Debian 8

What happened:

see above

What you expected to happen:

I expected the support of the beta annotation to be the piece that was duct-taped onto the new spot on the API for init-containers rather than the other way around.

How to reproduce it (as minimally and precisely as possible):

Create a deployment that uses a versioned image for an init-container with the definitions under .template.spec.initContainers[]

Once the deployment is verified to work, update the spec file's init-container definition to use a different version of the image.

After deployment, kubectl describe deployment thing and gaze in dismay.

Convert init-container definition to annotation JSON string. Redeploy. Functionally happy.

Anything else we need to know:

It appears part of this pig's lipstick is that when I either view or attempt to directly edit the deployment spec, the annotation is converted back into YAML for my viewing pleasure (with the alpha and beta init-container annotations in the template metadata). If I try to delete said YAML, kube-api doesn't complain; however, another kubectl get deployment -o yaml shows the bits that I just deleted!

Needless to say, if someone else comes along to interact with this deployment or its child pods, their first instinct will be to (try to) interact with the YAML structure, not the nightmarish JSON strings.

The text was updated successfully, but these errors were encountered:

gladiatr72 · 2017-05-11T00:57:19Z

Another observation: after creating a sparkly new deployment, the API returns a structure that includes the .templates.spec.initContainers bits even though the deployment spec file does not include one.

Whoever spent time making it look like this was done needs to be stared at balefully for at least a few minutes.

liggitt · 2017-05-11T03:23:36Z

cc @smarterclayton

xiangpengzhao · 2017-05-12T15:55:49Z

Made a simple digging into the code and found the root cause might be here: SetInitContainersAndStatuses.

@smarterclayton would give a detailed explanation.

smarterclayton · 2017-05-12T20:08:06Z

@erictune re beta annotations.

Unfortunately, part of supporting beta annotations is that they must override and take priority over the fields. Otherwise they would break if someone set init annotations. We will drop the annotations in 1.X (I was hoping 1.7, but not sure what guarantee we established, so 1.8 if not 1.7)

liggitt · 2017-05-12T20:15:44Z

kubelets can skew two versions older than the apiserver, so 1.8, probably

jagosan · 2017-05-12T22:02:20Z

/sig apps

gladiatr72 · 2017-05-16T17:59:01Z

That's all very interesting, but that doesn't mitigate init-containers being essentially broken in 1.6 (relative to the 1.6 documentation page)

To reiterate:

defining init-containers using GA notation works for the initial deployment.
- If any part of the GA notation is modified, subsequent deployments do not trigger an update
- if any part of the GA notation is modified, subsequent deployments must involve manual deletion of the annotation strings before the GA notation-based modifications will trigger an update
- if init-containers are defined using annotation strings in the original deployment doc, the initial and subsequent deployment trigger updates as expected only if init-container configuration continues to be maintained via annotation strings
deployments/pods with annotation-defined init-containers return both the annotation strings and a GA notation initContainers structure
- the GA notation initContainers structure is completely non-functional unless both the alpha and beta init-container strings are deleted manually for each deployment update.

this works (1.6.x)

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: thing
  namespace: thing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thing
      env: prod
  template:
    metadata:
      labels:
        app: thing
        env: prod
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
            "name": "migrate",
            "image": "gcr.io/repo-000000/thing:1.0.6",
            "command": ["sh", "-c", "/usr/local/bin/python /code/manage.py migrate"],
            "env": [
              {
                "name": "DJANGO_SETTINGS_MODULE",
                "value": "thing.settings.prod"
              },
              {
                "name": "PGPASSWORD",
                "valueFrom": {
                  "secretKeyRef": {
                    "name": "pg",
                    "key": "pass"
                  }
                }
              }
            ]
          },
          {
            "name": "reindex",
            "image": "gcr.io/repo-000000/thing:1.0.6",
            "command": ["sh", "-c", "/usr/local/bin/python /code/manage.py rebuild_index --noinput || sleep 1000"],
            "env": [
              {
                "name": "DJANGO_SETTINGS_MODULE",
                "value": "thing.settings.prod"
              },
              {
                "name": "PGPASSWORD",
                "valueFrom": {
                  "secretKeyRef": {
                    "name": "pg",
                    "key": "pass"
                  }
                }
              },
              {
                "name": "DJANGO_DEBUG",
                "value": ""
              }
            ]
          }
        ]'
    spec:
      containers:
        -
          name: gunicorn
          image: gcr.io/repo-000000/thing:1.0.6
          env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: pg
                  key: pass
            - name: DJANGO_SETTINGS_MODULE
              value: "thing.settings.prod"
            - name: SITE_URL
              value: "https://new.thing.domain.com"
            - name: DJANGO_DEBUG
              value: "True"
            - name: REDIS_DB
              value: "5"
            - name: REDIS_PORT
              value: "6379"
          ports:
            - containerPort: 8000

... and produces

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    annotations:
      deployment.kubernetes.io/revision: "4"
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"name":"thing","namespace":"thing"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"thing","env":"prod"}},"template":{"metadata":{"annotations":{"pod.beta.kubernetes.io/init-containers":"[ { \"name\": \"migrate\", \"image\": \"gcr.io/repo-000000/thing:1.0.5\", \"command\": [\"sh\", \"-c\", \"/usr/local/bin/python /code/manage.py migrate\"], \"env\": [ { \"name\": \"DJANGO_SETTINGS_MODULE\", \"value\": \"#
    creationTimestamp: 2017-05-15T04:16:13Z
    generation: 4
    labels:
      app: thing
      env: prod
    name: thing
    namespace: thing
    resourceVersion: "22792740"
    selfLink: /apis/extensions/v1beta1/namespaces/thing/deployments/thing
    uid: 3b9ec6c8-3925-11e7-aa37-42010af00091
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: thing
        env: prod
    strategy:
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 1
      type: RollingUpdate
    template:
      metadata:
        annotations:
          pod.alpha.kubernetes.io/init-containers: '[{"name":"migrate","image":"gcr.io/repo-000000/thing:1.0.5","command":["sh","-c","/usr/local/bin/python
            /code/manage.py migrate"],"env":[{"name":"DJANGO_SETTINGS_MODULE","value":"thing.settings.prod"},{"name":"PGPASSWORD","valueFrom":{"secretKeyRef":{"name":"pg","key":"pass"}}}],"resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"reindex","image":"gcr.io/repo-000000/thing:1.0.5","command":["sh","-c","/usr/local/bin/python
            /code/manage.py rebuild_index --noinput || sleep 1000"],"env":[{"name":"DJANGO_SETTINGS_MODULE","value":"thing.settings.prod"},{"name":"PGPASSWORD","valueFrom":{"secretKeyRef":{"name":"pg","key":"pass"}}},{"name":"DJANGO_DEBUG"}],"resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}]'
          pod.beta.kubernetes.io/init-containers: '[{"name":"migrate","image":"gcr.io/repo-000000/thing:1.0.5","command":["sh","-c","/usr/local/bin/python
            /code/manage.py migrate"],"env":[{"name":"DJANGO_SETTINGS_MODULE","value":"thing.settings.prod"},{"name":"PGPASSWORD","valueFrom":{"secretKeyRef":{"name":"pg","key":"pass"}}}],"resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"reindex","image":"gcr.io/repo-000000/thing:1.0.5","command":["sh","-c","/usr/local/bin/python
            /code/manage.py rebuild_index --noinput || sleep 1000"],"env":[{"name":"DJANGO_SETTINGS_MODULE","value":"thing.settings.prod"},{"name":"PGPASSWORD","valueFrom":{"secretKeyRef":{"name":"pg","key":"pass"}}},{"name":"DJANGO_DEBUG"}],"resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}]'
        creationTimestamp: null
        labels:
          app: thing
          env: prod
      spec:
        containers:
        - env:
          - name: PGPASSWORD
            valueFrom:
              secretKeyRef:
                key: pass
                name: pg
          - name: DJANGO_SETTINGS_MODULE
            value: thing.settings.prod
          - name: SITE_URL
            value: https://new.thing.domain.com
          - name: DJANGO_DEBUG
            value: "True"
          - name: REDIS_DB
            value: "5"
          - name: REDIS_PORT
            value: "6379"
          image: gcr.io/repo-000000/thing:1.0.5
          imagePullPolicy: IfNotPresent
          name: gunicorn
          ports:
          - containerPort: 8000
            protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        dnsPolicy: ClusterFirst

magically generated structure that does nothing

        initContainers:
        - command:
          - sh
          - -c
          - /usr/local/bin/python /code/manage.py migrate
          env:
          - name: DJANGO_SETTINGS_MODULE
            value: thing.settings.prod
          - name: PGPASSWORD
            valueFrom:
              secretKeyRef:
                key: pass
                name: pg
          image: gcr.io/repo-000000/thing:1.0.5
          imagePullPolicy: IfNotPresent
          name: migrate
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        - command:
          - sh
          - -c
          - /usr/local/bin/python /code/manage.py rebuild_index --noinput || sleep
            1000
          env:
          - name: DJANGO_SETTINGS_MODULE
            value: thing.settings.prod
          - name: PGPASSWORD
            valueFrom:
              secretKeyRef:
                key: pass
                name: pg
          - name: DJANGO_DEBUG
          image: gcr.io/repo-000000/thing:1.0.5
          imagePullPolicy: IfNotPresent
          name: reindex
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 30

  status:
    conditions:
    - lastTransitionTime: 2017-05-15T04:16:13Z
      lastUpdateTime: 2017-05-15T04:16:13Z
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
    observedGeneration: 4
    replicas: 1
    unavailableReplicas: 1
    updatedReplicas: 1
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""

cknowles · 2017-07-11T03:34:54Z

Following the above advice from @gladiatr72, we've tried to delete the init container annotations prior to new deploys and thought it might help someone facing the same issue. Our init containers don't change so frequently and we just started using them a bit more so we needed a workaround for these issues on k8s 1.6.4.

If we only delete the annotations, they seems to get restored immediately from the GA notation. At the point in time we wish to clear the init containers they reference the old versions. It seems the only way for us to update init containers on k8s 1.6.4 is to delete both the annotations and the GA notation:

✗ kubectl patch deployment --type=json <DEPLOYMENT_NAME> \
-p='[{"op": "remove", "path": "/spec/template/metadata/annotations/pod.beta.kubernetes.io~1init-containers"}, 
  {"op": "remove", "path": "/spec/template/metadata/annotations/pod.alpha.kubernetes.io~1init-containers"},
  {"op": "remove", "path": "/spec/template/spec/initContainers"}]`

deployment "<DEPLOYMENT_NAME>" patched

✗ kubectl get deployment <DEPLOYMENT_NAME> -o yaml

<result does not have any init containers so ready to be updated>

To clarify, deleting only the annotations like this command below does not work for us because afterwards DEPLOYMENT_NAME still has the annotations and GA notation present despite kubectl saying it has been patched:

✗ kubectl patch deployment --type=json <DEPLOYMENT_NAME> \
-p='[{"op": "remove", "path": "/spec/template/metadata/annotations/pod.beta.kubernetes.io~1init-containers"}, 
  {"op": "remove", "path": "/spec/template/metadata/annotations/pod.alpha.kubernetes.io~1init-containers"}}]`

deployment "<DEPLOYMENT_NAME>" patched

✗ kubectl get deployment <DEPLOYMENT_NAME> -o yaml

<result still has both alpha and beta init container annotations plus the GA notation>

liggitt · 2017-07-11T03:39:36Z

the population of the annotations happens during conversion, and can be removed in 1.8 (since the oldest supported kubelets will be at 1.6, which recognizes the API fields)

cknowles · 2017-07-11T03:44:52Z

@liggitt Thanks, understood. Looking forward to it! Hopefully the above can help someone needing a workaround before 1.8.

liggitt · 2017-09-06T16:50:25Z

fixed by #51816

k8s-ci-robot added the sig/apps Categorizes an issue or PR as relevant to SIG Apps. label May 12, 2017

0xmichalis added the kind/technical-debt label May 21, 2017

0xmichalis added this to the next-candidate milestone May 21, 2017

liggitt mentioned this issue May 23, 2017

Propose a feature to troubleshoot running pods kubernetes/community#649

Merged

6 tasks

liggitt mentioned this issue Aug 15, 2017

Kubernetes v1.6.4 -> v1.7.0 update breaks statefulsets #48327

Closed

liggitt mentioned this issue Sep 1, 2017

Remove deprecated init-container in annotations #51816

Merged

liggitt closed this as completed Sep 6, 2017

gladiatr72 mentioned this issue Mar 26, 2018

feature request: cluster semver via .Release object helm/helm#3765

Closed

ClementGautier mentioned this issue Aug 5, 2021

Add support for initContainers jenkinsci/kubernetes-plugin#254

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

init-container definition (non-anno.) persisting after removal from deployment #45627

init-container definition (non-anno.) persisting after removal from deployment #45627

gladiatr72 commented May 11, 2017 •

edited

Loading

gladiatr72 commented May 11, 2017

liggitt commented May 11, 2017

xiangpengzhao commented May 12, 2017

smarterclayton commented May 12, 2017

liggitt commented May 12, 2017

jagosan commented May 12, 2017

gladiatr72 commented May 16, 2017 •

edited

Loading

cknowles commented Jul 11, 2017 •

edited

Loading

liggitt commented Jul 11, 2017

cknowles commented Jul 11, 2017

liggitt commented Sep 6, 2017

init-container definition (non-anno.) persisting after removal from deployment #45627

init-container definition (non-anno.) persisting after removal from deployment #45627

Comments

gladiatr72 commented May 11, 2017 • edited Loading

gladiatr72 commented May 11, 2017

liggitt commented May 11, 2017

xiangpengzhao commented May 12, 2017

smarterclayton commented May 12, 2017

liggitt commented May 12, 2017

jagosan commented May 12, 2017

gladiatr72 commented May 16, 2017 • edited Loading

cknowles commented Jul 11, 2017 • edited Loading

liggitt commented Jul 11, 2017

cknowles commented Jul 11, 2017

liggitt commented Sep 6, 2017

gladiatr72 commented May 11, 2017 •

edited

Loading

gladiatr72 commented May 16, 2017 •

edited

Loading

cknowles commented Jul 11, 2017 •

edited

Loading