Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for environment variable resolution in httpGet probe paths #128900

Open
bachmanity1 opened this issue Nov 21, 2024 · 18 comments · May be fixed by #128957
Open

Add support for environment variable resolution in httpGet probe paths #128900

bachmanity1 opened this issue Nov 21, 2024 · 18 comments · May be fixed by #128957
Assignees
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@bachmanity1
Copy link

bachmanity1 commented Nov 21, 2024

Currently, Kubernetes does not support resolving environment variables in httpGet probe paths, which limits configuration flexibility.

Expected behavior:

  • Allow environment variable interpolation like path: "$(CONTEXT_PATH)/health/"
  • Support standard shell-like variable substitution syntax

Sample configuration:

       readinessProbe:
          httpGet:
            path: "$(CONTEXT_PATH)/actuator/health/readiness"
            port: 8080
            scheme: HTTP
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 21, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@bachmanity1
Copy link
Author

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 21, 2024
@bachmanity1
Copy link
Author

If the relevant SIG believes this feature request is worthwhile, I can begin working on it.

@kannon92
Copy link
Contributor

/kind feature

@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 21, 2024
@pacoxu
Copy link
Member

pacoxu commented Nov 21, 2024

IIUC, a simple workaround is using exec probe.

  • There are alternative solutions, but I am not opposed to this requirement.

@bachmanity1
Copy link
Author

IIUC, a simple workaround is using exec probe.

Yes, that's correct, but it would mean the image needs to have curl (or a similar tool) included, which isn’t always the case. Plus, using exec would make the probe configuration a bit more verbose.

@saschagrunert
Copy link
Member

Exec probes are also a bit more resource intensive compared to the other ones. I kinda like the idea of supporting the predefined environment variables, but it needs to apply to all other types of probes as well.

@bachmanity1
Copy link
Author

bachmanity1 commented Nov 21, 2024

I’m considering an approach like the one below to resolve environment variables in the path:

exec := exec.New()
cmd := exec.Command("echo", pathToProcess)
buff := bytes.Buffer{}
cmd.SetStdout(&buff)
cmd.Run()
processedPath := buff.String()

However, I’m concerned this might not be safe, as it could open the door to remote shell execution attacks (though exec probes have a similar risk). Additionally, running a shell before using httpGet could end up being just as resource-intensive as exec probes. An alternative might be to fetch environment variables directly from the container object, but there’s a chance that some variables in the path might not be present in the container spec. Additionally, values for certain variables, like the pod name, cannot be derived from the container spec.

@saschagrunert
Copy link
Member

An alternative might be to fetch environment variables directly from the container object

That's how we should do it from my point of view.

@bachmanity1
Copy link
Author

That's how we should do it from my point of view.

I think I’m leaning toward the first approach since, as mentioned, fetching environment variables directly from the container object wouldn’t work for all cases.

@sftim
Copy link
Contributor

sftim commented Nov 22, 2024

We should use Kubernetes-style environment variable interpolation, I think

See https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config for an example.

@saschagrunert
Copy link
Member

@bachmanity1 the issue is that the kubelet calls the http probes directly, while exec'ing into the container comes with a reasonable overhead. Keeping those constraints allows us only to work with the data we have at the manifest level.

@bachmanity1
Copy link
Author

Thanks for the feedback! I’m planning to open a PR in a couple of days using Kubernetes-style environment variable interpolation.

@bachmanity1
Copy link
Author

/assign @bachmanity1

@bachmanity1
Copy link
Author

Hi @pacoxu, @saschagrunert, and @sftim, thank you for your feedback! I've opened a PR and linked it above. Could you take a look when you have a moment? Thanks!

@aojea
Copy link
Member

aojea commented Dec 16, 2024

Exec probes are also a bit more resource intensive compared to the other ones. I kinda like the idea of supporting the predefined environment variables, but it needs to apply to all other types of probes as well.

@saschagrunert they are not for the kubelet, http and tcp probes steal resources from the kubelet that is more problematic, actually for constrained kubelet that fails network probes the workaround is to move to use exec probes #89898 (comment)

@liggitt
Copy link
Member

liggitt commented Dec 17, 2024

reinterpreting existing fields in new ways isn't really ok... existing httpGet probes with environment-looking substrings would behave differently

see kubernetes/enhancements#559 / #71351 for a similar addition to support envvars in subpath mounts, which required a new field that explicitly supported envvar expansion

@liggitt
Copy link
Member

liggitt commented Dec 17, 2024

if this was added, I would expect the same env values and format used to expand other aspects of the pod (container args, etc)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants