Environment variables named specific things not getting passed on from deployment to pods #25585

drashkov · 2016-05-13T19:51:07Z

Deployment is defined in a json file with several environment varibles. Deployment is deployed using "kubectl apply -f deployment.json". Pods get most of the environment variables defined, but not all.

Specifically, the variables defined in the deployment.json are named:
{ "name": "DB_HOSTS", "value": "10.0.0.1" }, { "name": "SLACK_ENDPOINT", "value": "ERRORNOT" }, { "name": "LACK_ENDPOINT", "value": "ERRORNOT" }, { "name": "SLACK", "value": "ERRORNOT" }, { "name": "CK_ENDPOINT", "value": "ERRORNOT" }, { "name": "ACK_ENDPOINT", "value": "ERRORNOT" }, { "name": "K_ENDPOINT", "value": "ERRORNOT" }, { "name": "CASSANDRA_LOGGING_LEVEL", "value": "ERROR" }, { "name": "ENABLE_GETIP", "value": "https://www.url.com" }

the environment variables that appear in a pod are:

`
ENABLE_GETIP

CASSANDRA_LOGGING_LEVEL

K_ENDPOINT

ACK_ENDPOINT

CK_ENDPOINT

DB_HOSTS
`

That is, ENV variables named "SLACK_ENDPOINT", "LACK_ENDPOINT" and "SLACK" never get passed on to the pods.

The text was updated successfully, but these errors were encountered:

drashkov · 2016-05-20T23:21:13Z

It also won't accept "randomshit" as an environment varible. When I add that as an environment variable to an existing deployment and run "kubctl apply -f deployment.json" nothing happens.

This is on version v1.2.4 / 3eed1e3

thockin · 2016-05-23T00:54:27Z

Ehh? Can you post a full but minimized yaml that I can run to reproduce it?

On Fri, May 20, 2016 at 4:21 PM, drashkov notifications@github.com wrote:

It also won't accept "randomshit" as an environment varible. When I add
that as an environment variable to an existing deployment and run "kubctl
apply -f deployment.json" nothing happens.

This is on version v1.2.4 / 3eed1e3
3eed1e3

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#25585 (comment)

drashkov · 2016-05-24T21:59:12Z

We use JSON files, here is the full one. "SLACK_ENDPOINT" does not appear as an environment variable in the pods. Doing "kubectl apply -f .json" does not create new pods, and does not pass these variables to the pods. If I change the number of replicas and add a new pod, that pod will be created, but won't have that varible.

{

    "apiVersion": "extensions/v1beta1",

    "kind": "Deployment",

    "metadata": {

        "name": "test-api-deployment"

    },

    "spec": {

        "replicas": 3,

        "template": {

            "metadata":{

                "labels": {

                    "app": "test-api"

                }

            },

            "spec": {

                "containers": [{

                    "image": "gcr.io/<repository>/test_api:1.3.5",

                    "name": "test-api",

                    "env": [

                        {

                            "name": "SLACK_ENDPOINT",

                            "value": "https://slack.com"

                        },

                        {

                            "name": "API_DB_HOSTS",

                            "value": "10.0.0.83"

                        },

                        {

                            "name": "CASSANDRA_LOGGING_LEVEL",

                            "value": "ERROR"

                        }]

                }]

            }

        }

    }

}

pwittrock · 2016-05-25T21:22:53Z

I am not able to reproduce on GKE at 1.2.4 when exec'ing into one of the Pods and printing the environment.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}

$ kubectl exec test-api-deployment-2124663613-029ak -i -t sh
/ # env
...
SLACK_ENDPOINT=https://slack.com
...

This is the yaml I used - note the image is different:

{

    "apiVersion": "extensions/v1beta1",

    "kind": "Deployment",

    "metadata": {

        "name": "test-api-deployment"

    },

    "spec": {

        "replicas": 3,

        "template": {

            "metadata":{

                "labels": {

                    "app": "test-api"

                }

            },

            "spec": {

                "containers": [{

                    "image": "gcr.io/google_containers/serve_hostname:v1.4",

                    "name": "test-api",

                    "env": [

                        {

                            "name": "SLACK_ENDPOINT",

                            "value": "https://slack.com"

                        },

                        {

                            "name": "API_DB_HOSTS",

                            "value": "10.0.0.83"

                        },

                        {

                            "name": "CASSANDRA_LOGGING_LEVEL",

                            "value": "ERROR"

                        }]

                }]

            }

        }

    }

}

What are the symptoms you are seeing? Can you provide yaml using standard public image along with the specific symptoms you are seeing?

gflarity · 2016-05-25T21:43:59Z

@pwittrock did you create the deployment first with out the env vars, then add them via an apply?

drashkov · 2016-05-25T22:56:03Z

Thanks for looking into it @pwittrock, I managed to reproduce it with the yaml you posted.

Here's what I did:

Saved the yaml you provided as 'og.yaml'
Deleted the "SLACK_ENDPOINT" env var from that file and saved that as "mod.yaml" (no other changes)

$ kubectl apply -f og.yaml
deployment "test-api-deployment" created
$ kubectl exec test-api-deployment-2124663613-9tl0u env |grep SLACK
SLACK_ENDPOINT=https://slack.com

This creates the deployment with all env vars as expected

$ kubectl apply -f mod.yaml
$ kubectl exec test-api-deployment-2413078913-547h0 env |grep SLACK

This re-creates a new set of pods and they don't have the SLACK_ENDPOINT var, as expected.

$ kubectl apply -f og.yaml ; watch kubectl get pods

This does not re-create the pods, so the env var is not passed along. kubectl does not return errors.

pwittrock · 2016-05-26T22:12:59Z

cc @janetkuo
cc @adohe

pwittrock · 2016-05-26T22:19:04Z

@drashkov Thanks, I was able to reproduce the issue.

It looks like the config with the ENV does get stored in the last-applied-configurations annotation:

kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"test-api-deployment","creationTimestamp":null},"spec":{"replicas":3,"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"test-api","app2":"test-api2"}},"spec":{"containers":[{"name":"test-api","image":"gcr.io/google_containers/serve_hostname:v1.4","env":[{"name":"SLACK_ENDPOINT","value":"https://slack.com"},{"name":"SLACK_ENDPOINT2","value":"https://slack.com"},{"name":"CASSANDRA_LOGGING_LEVEL","value":"ERROR"}],"resources":{}}]}},"strategy":{}},"status":{}}'

But does not get put into the pods:

  containers:
  - env:
    - name: SLACK_ENDPOINT
      value: https://slack.com
    - name: CASSANDRA_LOGGING_LEVEL
      value: ERROR
    image: gcr.io/google_containers/serve_hostname:v1.4
    imagePullPolicy: IfNotPresent
    name: test-api
    resources:
      requests:
        cpu: 100m

janetkuo · 2016-05-26T23:31:04Z

After a few tries I found that if you add the SLACK_ENDPOINT env as the last one you'll be able to add it. Definitely a bug.

janetkuo · 2016-05-26T23:48:04Z

The generated three way merge patch wasn't correct (the annotation is good, but look at the spec part):
patch = {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"kind\":\"Deployment\",\"apiVersion\":\"extensions/v1beta1\",\"metadata\":{\"name\":\"test-api-deployment\",\"creationTimestamp\":null},\"spec\":{\"replicas\":3,\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"test-api\"}},\"spec\":{\"containers\":[{\"name\":\"test-api\",\"image\":\"gcr.io/google_containers/serve_hostname:v1.4\",\"env\":[{\"name\":\"API_DB_HOSTS\",\"value\":\"10.0.0.83\"},{\"name\":\"SLACK_ENDPOINT\",\"value\":\"https://slack.com\"},{\"name\":\"CASSANDRA_LOGGING_LEVEL\",\"value\":\"ERROR\"}],\"resources\":{}}]}},\"strategy\":{}},\"status\":{}}"},"creationTimestamp":null},

"spec":{"template":{"spec":{"containers":[{"env":[{"name":"CASSANDRA_LOGGING_LEVEL","value":"ERROR"}],"name":"test-api"}]}}}}

janetkuo · 2016-05-26T23:49:01Z

It only recognizes the last env as diff

adohe-zz · 2016-05-27T12:50:53Z

Opened a PR #26418 ptal.

drashkov · 2016-05-27T14:21:33Z

Thanks everyone.

@janetkuo

Automatic merge from submit-queue fix strategy patch diff list issue fixes #25585 @janetkuo @pwittrock ptal.

j3ffml added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. team/ux labels May 14, 2016

pwittrock added the priority/backlog Higher priority than priority/awaiting-more-evidence. label May 26, 2016

pwittrock added kind/bug Categorizes issue or PR as related to a bug. 0 - Backlog labels May 26, 2016

janetkuo added area/app-lifecycle area/kubectl labels May 26, 2016

adohe-zz mentioned this issue May 27, 2016

Fix strategic merge diff list diff bug #26418

Merged

bgrant0607 mentioned this issue May 28, 2016

kubectl apply results in incorrect/corrupted config if a fieldRef env var is added #26229

Closed

bgrant0607 assigned adohe-zz May 28, 2016

bgrant0607 added this to the v1.3 milestone May 28, 2016

bgrant0607 added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels May 28, 2016

k8s-github-robot closed this as completed in #26418 May 30, 2016

k8s-github-robot pushed a commit that referenced this issue May 30, 2016

Merge pull request #26418 from AdoHe/kubectl_env

1958427

Automatic merge from submit-queue fix strategy patch diff list issue fixes #25585 @janetkuo @pwittrock ptal.

adohe-zz mentioned this issue Jun 8, 2016

Deployment does not update when adding environment variables #26944

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Environment variables named specific things not getting passed on from deployment to pods #25585

Environment variables named specific things not getting passed on from deployment to pods #25585

drashkov commented May 13, 2016 •

edited

Loading

drashkov commented May 20, 2016

thockin commented May 23, 2016

drashkov commented May 24, 2016 •

edited by pwittrock

Loading

pwittrock commented May 25, 2016

gflarity commented May 25, 2016 •

edited

Loading

drashkov commented May 25, 2016 •

edited by pwittrock

Loading

pwittrock commented May 26, 2016

pwittrock commented May 26, 2016

janetkuo commented May 26, 2016 •

edited

Loading

janetkuo commented May 26, 2016 •

edited

Loading

janetkuo commented May 26, 2016 •

edited

Loading

adohe-zz commented May 27, 2016

drashkov commented May 27, 2016

Environment variables named specific things not getting passed on from deployment to pods #25585

Environment variables named specific things not getting passed on from deployment to pods #25585

Comments

drashkov commented May 13, 2016 • edited Loading

drashkov commented May 20, 2016

thockin commented May 23, 2016

drashkov commented May 24, 2016 • edited by pwittrock Loading

pwittrock commented May 25, 2016

gflarity commented May 25, 2016 • edited Loading

drashkov commented May 25, 2016 • edited by pwittrock Loading

pwittrock commented May 26, 2016

pwittrock commented May 26, 2016

janetkuo commented May 26, 2016 • edited Loading

janetkuo commented May 26, 2016 • edited Loading

janetkuo commented May 26, 2016 • edited Loading

adohe-zz commented May 27, 2016

drashkov commented May 27, 2016

drashkov commented May 13, 2016 •

edited

Loading

drashkov commented May 24, 2016 •

edited by pwittrock

Loading

gflarity commented May 25, 2016 •

edited

Loading

drashkov commented May 25, 2016 •

edited by pwittrock

Loading

janetkuo commented May 26, 2016 •

edited

Loading

janetkuo commented May 26, 2016 •

edited

Loading

janetkuo commented May 26, 2016 •

edited

Loading