Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate DNS proxy port bind issues #1272

Closed
bleggett opened this issue Aug 21, 2024 · 5 comments
Closed

Investigate DNS proxy port bind issues #1272

bleggett opened this issue Aug 21, 2024 · 5 comments
Assignees

Comments

@bleggett
Copy link
Contributor

bleggett commented Aug 21, 2024

Reference: https://istio.slack.com/archives/C37A4KAAD/p1724148926299219

@bleggett bleggett changed the title DNS proxy likely does not need to bind to localhost Investigate DNS proxy port bind issues Aug 21, 2024
@bleggett bleggett self-assigned this Aug 21, 2024
@howardjohn
Copy link
Member

Sorry for the vague comment on slack - I am not sure I agree, I would actually prefer if we bound to localhost for 15001 almost as well.

The only legitimate place to connect to these is from the local pod. if we expose it externally, we can/will have users able to do curl pod-ip:15053. This is bad. We have a hack in the 15001 listener to explicitly block these, but would prefer if we don't.

At the very least, we would need to add a similar check for DNS

@bleggett
Copy link
Contributor Author

bleggett commented Aug 21, 2024

The only legitimate place to connect to these is from the local pod. if we expose it externally, we can/will have users able to do curl pod-ip:15053. This is bad.
We have a hack in the 15001 listener to explicitly block these, but would prefer if we don't.

Yes, agreed on both. If I avoid localhost, a prerequisite of doing so would be mitigating this without code hacks. DNS lookups don't happen from the pod netns anyway, so this should be preventable with an iptables rule.

From the perspective of the iptables rules we really don't care what the bound address is, I'm mostly trying to understand how it can happen for localhost binds specifically.

@yuval-k
Copy link
Contributor

yuval-k commented Aug 23, 2024

this error looks similar kubernetes/ingress-nginx#2904
basically, while ipv6 is enabled, lo device doesn't have an ipv6 address assigned to it.

@bleggett
Copy link
Contributor Author

bleggett commented Aug 23, 2024

this error looks similar kubernetes/ingress-nginx#2904 basically, while ipv6 is enabled, lo device doesn't have an ipv6 address assigned to it.

Which might explain why restarts fix it but retries do not, as presumably lo is created/assigned addresses once, on pod creation.

Seems like options are

  • Dynamically do not create a V6 localhost listener in-pod, if the interface doesn't have a V6 address, regardless of V6 flags (seems potentially bad/confusing for people -but we could loudly warn).
  • Move the DNS listener off of localhost and bind to any/unspecified like the other handlers (which hides the problem) - in theory this should work/I'm trying to get this working.

We could just do the former tho, if we want? I guess it's not so much different from the kernel support detection we do already.

@bleggett
Copy link
Contributor Author

Closing as this turned out to be an issue on our end (which using UNSPECIFIED addr would have elided, but no need to do so, as long as it isn't needed to work around platform quirks)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants