@fiatala There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-* for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

/sig storage

I'm not sure I understand what you gain by using $(NAME) instead of metadata.name directly.

To use metadata.name you need the extra scaffolding to support a map instead of a string - note that the key was changed from subPath to subPathFrom to align with ENV value and valueFrom pattern.
If you support $(NAME), then it's still a string and can use the variable substitution work that is already in place for other sections of the file.

Ah, I see. I misread your example as saying that subPathFrom was the existing solution and you wanted to move to subPath with an env var.

Are there other cases where we plumb pod env vars to pod spec fields today?
@lavalamp wdyt about this? I'm wary of accidentally creating a templating solution.

You can use expanded env vars in command line and args lines only. So that's how you can create a unique subpath using an init container with a command line symbolic link but that solution is messy.

I'd rather not selectively allow env var expansion depending on the subfield of a config, that would be pretty confusing to users. I'd prefer the subPathFrom solution (downward api based) if you just want this for subPaths. Ofc we'd have to caution users not to reference fields that don't contain valid filenames.

@mtaufen , $() expansion is already implemented for args and env vars, how expanding it for another field makes it confusing?

subPathFrom doesn't work very well as it doesn't allow things like:

subPath: $(POD_NAMESPACE)/$(POD_NAME).dir which is a real use case and pain point at least for us

If you only extend it to work in the subPath case, people will expect it to work everywhere and be confused that it doesn't. It would be interesting to have env var expansion work in arbitrary fields of the container spec, and I think it's worth discussing with a larger audience. I'd recommend writing a brief design doc in a PR against the community repo (see https://github.com/kubernetes/community/tree/master/contributors/design-proposals). It is a significant API decision, IMO.

General templating has been discussed a lot before, and we still don't have an implemented solution:

• Config template parameterization #11492
• https://github.com/kubernetes/community/blob/master/contributors/design-proposals/templates.md
• Add a simple template API object that assists with config parameterization enhancements#35

But since this is a far more specific use-case, and you're only resolving variables for containers, it might get more traction.

By adding the subpath doesnt in my opinion make it more confusing because it is already very restricted to only 2 elements. The current way of using an init container is much more confusing for an end user

@thockin @lavalamp @smarterclayton @devin-donnelly thoughts?

Im trying to do the same as what this issue is talking about.

But I dint understand @kevtaylor's comments about using an init-container to solve this. I have a init-container that can reference the pod's name(using env) and create a subdirectory in NFS specific to the pod , but how do I mount that subdirectory into my container. Should I use a postStart script to symlink the directory?

@rajasaur

      - name: prep-logs
        image: alpine:3.5
        command:
        - /bin/sh
        - -xc
        - >
          LOGDIR=/logs/${POD_NAMESPACE}/${POD_NAME};
          mkdir -p ${LOGDIR}
          && ln -sfv ${LOGDIR} /workdir/logs
          && chmod -R ugo+wr ${LOGDIR}
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - mountPath: /logs
          name: logs
        - mountPath: /workdir
          name: workdir

  volumeMounts:
            # Inside init container we created symlink  /workdir/logs -> /logs/$POD_NAME
            # where /logs is a logs EBS volume mounted on a host *UNDER_THE_SAME_NAME* == /logs
            # when starting main app container, it looks like we re mounting workdir to $appLogdir,
            # but because of *subpath* what actually happens is that symlink /workdir/logs is mounted
            # instead. 
            # Trick is: *WHEN MOUNTING SYMLINK, DOCKER FOLLOWS IT AND MOUNTS IT'S TARGET INSTEAD*
            # which is in our case a log directory on a /logs EBS volume
            - mountPath: /containers/log/folder
              name: workdir
              subPath: logs

 volumes:
        - name: workdir
          emptyDir: {}
        - name: logs
          hostPath:
            path: /logs

Thanks @kevtaylor . I tried a similar approach earlier which led to an error of "file already exists" because the "Trick" that you mentioned wasnt happening in my case , i.e. once the symlink was created in the init container, when actually doing the volumemount subPath, it tried to create a path called "logs" (the subPath argument) and failed with "File already exists" error. Ill retry with your configuration and see what happens.

I worked around it with a postScript that does something like

        lifecycle:
           postStart:
              exec:
                 command: ["/bin/sh", "-c", "ln -s /workdir/logs/$PODNAME /container/logs"]

and exposing PODNAME as env variable.

@rajasaur Thanks for the update - much messing about can be alleviated with an environmental expansion - hence my PR #49388

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

@kubernetes/sig-storage-feature-requests
/cc @msau42 @jsafrane

Sorry, the trick will no longer work due to #60813

We will have to reconsider if/how to support this feature request as the subpath handling is quite complicated now

@msau42 Which trick are you referring to - the symbolic link in the init container ?
We found that out by trying to go above 1.9.3 and they all failed
But we believe that the PR removes all that need anyway, so not sure why the PR won't work as the e2e tests within it are passing

Yes

@kevtaylor actually the trick could still work if you use a hard link instead of a symlink. As long as the link you're creating is within the same volume, then we will allow it.

Anyway, regarding this feature, we need to have further API discussions because as @mtaufen points out, this is straddling the line with templating and could have broader implications. I can bring it up in the next sig storage meeting.

For this particular use case of logging, have you considered using a distributed logging pipeline with fluentd and elasticsearch?

Yes. It isnt quite that simple. Our legacy apps write a number of log files not just single streamed stdout so we need to isolate them into a file system structure and get splunk to log them. If we have multiple pods running on a single instance then the file systems have to be uniquely partioned.

Also it isn't really straddling templating. There is already environment expansion in command and args. This is just extending that same principle into the volume mount. It's really quite a small pr change to be quite fair.

This is definitively needed, pods for the same pod-template/deployment could land on the same host (even though the scheduler will try to avoid this), and this would result in a path clash w/o this option. Also some logging is so voluminous that splunk/elk/fluentd would just be too costly.

@msau42 Has there been any progress on this? We are totally blocked on moving from k8s 1.9.3 until we get some sort of resolution. Thanks

We didn't have time in yesterday's sig meeting to discuss this unfortunately.

Are you not able to use the workaround of hard links or relative symlinks in the init container?

Given that we just had a huge vulnerability in the subpath feature, and this feature is also touching the same area, I want to be more conservative and at the very minimum, have this feature go through alpha/beta/ga feature gating, and have the design go through security review as well.

@msau42 I do not want to use links or otherwise as this is an entire bodge. We have 100s of apps and I do not want to start redeploying our entire product base by changing our init containers especially as they took us a lifetime to make work in the first place

@kevtaylor can you create an issue in the features repo for this, and add it to the SIG-storage planning spreadsheet so that we can track it for this quarter?

You can put me and @jsafrane (sorry I'm volunteering you) as the reviewers.

@msau42 I think I did this correctly but I have created a request and added into Q2 2018. Thanks

cc @andyzhangx can you review this feature to see if it can be supported in windows?

@msau42 this proposoal is not os specific, so it should be ok for windows. Subpath is already supported on windows, what we need to to is ensure that in code implementation there is no os specific handling(like os path handling) in common code, just loop me in the code review, thanks.