See also:

1. net.Listen listens on all available unicast and anycast IP addresses regarding the IP family #88458
2. kube-proxy: We should be able to override bind addresses for healthz and metrics  #108737
3. kube-proxy: Inconsistent behaviors about disabling health check server and metrics server #123559

/sig network

I think that's doubtful, 0.0.0.0 and :: also bind to both families in kube-apiserver

How does kube-apiserver work for headlthz-bind-address? I think if it defaults to "0.0.0.0:10256", we can access to the service with ipv4 or ipv6 address.

when a server is told to listen on 0.0.0.0 that means "listen on every available network interface".

Just FYI, I tested to "scope" ipv6 localhost with [::%lo]:10256. It is accepted by go (but not by kube-proxy), but alas, the scope is simply ignored for non-link-local addresses 😞

@sanmai-NL we are planning to add v1alpha2 kube-proxy configuration which will accept a dual-stack CIDR pair of healthz and metrics addresses.

(we are moving towards CIDR to allow users to provide the network instead of just address, healthz and metrics server will be exposed on nodesIPs which belong to that CIDR range)
(ref: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/784-kube-proxy-component-config#following-fields-will-be-changed)

The wanted feature here seems to be to accept connects on lo only. A good way is to bind the listen socket to the device using the SO_BINDTODEVICE sockopt. Now, I don't propose this as a solution in this case, but the function may come in handy in some other cases, so I figured out how it's done in go.

// Tcp server that binds to "lo"
package main

import (
	"context"
	"log"
	"net"
	"os"
	"syscall"
)

func main() {
	if len(os.Args) < 2 {
		log.Fatal("No address")
	}
	lconfig := net.ListenConfig{
		Control: listenControl,
	}
	listener, err := lconfig.Listen(context.TODO(), "tcp", os.Args[1])
	if err != nil {
		log.Fatal(err)
	}
	defer listener.Close()
	conn, err := listener.Accept()
	if err != nil {
		log.Fatal(err)
	}
	conn.Write([]byte("Bye...\n"))
	defer conn.Close()
}
func listenControl(network, address string, c syscall.RawConn) error {
	return c.Control(setIf)
}
func setIf(fd uintptr) {
	if err := syscall.BindToDevice(int(fd), "lo"); err != nil {
		log.Fatal(err)
	}
}

./tcp-server :7000
nc -v 127.4.5.6 7000  # works
nc -v ::1 7000        # works
nc -v 172.17.0.1 7000 # Connection refused

As you can see, any address on lo works, but not local address on another interface (172.17.0.1 is on docker0)

Um, come to think of it, a device may be better than two addresses here. Unless the addresses are "any" or loopback, they must be different on all nodes. While this can be done, a common configmap makes it hard. An option might be something like:

healthzBindAddress: "[::]:10256"
healthzBindInterface: "lo"

Um, come to think of it, a device may be better than two addresses here. Unless the addresses are "any" or loopback, they must be different on all nodes. While this can be done, a common configmap makes it hard. An option might be something like:

healthzBindAddress: "[::]:10256"
healthzBindInterface: "lo"

kubernetes/enhancements#4337 (comment)

Thanks for the pointer. Actually I think an interface may be better than a CIDR. Interfaces are more stable. It may be easier to guess the interface than the CIDR some infra-structure provider will use for nodes.

Thanks for the pointer. Actually I think an interface may be better than a CIDR. Interfaces are more stable. It may be easier to guess the interface than the CIDR some infra-structure provider will use for nodes.

But people can assign non-loopback addresses to a loopback interface and this would break.

A network interface can be named variously depending on node OS config or hot-swapping NICs. They are less stable than, e.g., a pair of loopback CIDR blocks ::1/128,127.0.0.1/8.

And CIDRs can change because of DHCP lease, or a new /64 segment for SLAAC. But, I agree with Dan that CIDRs are a more "more-kube-proxy-like approach".

And CIDRs can change because of DHCP lease

IPs will change because of the DHCP lease not CIDR, right?
CIDR will be configured with the DHCP server itself, no?

@uablrek For the use case I mentioned, CIDR subnets are stable. I think working with devices and supporting zone indexes in IPv6 addresses is interesting and potentially useful, but the requested functionality shouldn't be designed around that.

@shaneutt: The label(s) triage/accept cannot be applied, because the repository doesn't have them.

/assign @danwinship

/triage accepted

I feel like we have danced around this area a bunch of times. Can we clarify what the real requirment is and implement it one time that we can use everywhere?

E.g. write a k/utils library that offers "multi-listen" and takes a list of IPs and/or CIDRs, finds all matching local IPs, opens sockets on all of them and presents a facade of a single socket.

The documentation suggests that the bind address is singular and that it's either of the IPv4 or IPv6 family. I think that's doubtful, 0.0.0.0 and :: also bind to both families in kube-apiserver.

The documentation is correct, just misleading. If you leave the value unset, then kube-proxy behaves as though you had said either --healthz-bind-address 0.0.0.0:10256 or --healthz-bind-address [::]:10256, depending on whether the cluster is ipv4-primary or ipv6-primary. On Linux, this is sort of pointless because they both have the same effect, but on Windows it actually only binds to one family.

I'm not sure anyone really cares a lot about listening on both 127.0.0.1 and ::. You can always listen on just 127.0.0.1, even in a single-stack IPv6 cluster, because you don't need any external IPv4 routing to use 127.0.0.1.

In discussion on the v1alpha2 stuff, we decided the use case for overriding this is "I only want to serve health/metrics on the IP that faces the load balancer / prometheus / whatever". To match the way --nodeport-addresses works, we decided to let the user specify a CIDR, and have kube-proxy pick a local IP from in that CIDR. That way you can specify a single config for all your nodes.

we decided to let the user specify a CIDR, and have kube-proxy pick a local IP from in that CIDR

That pattern makes sense to me, but is a single family sufficient? IOW, does --healthz-bind-address=10.9.8.0/24,2601:8675::3099/64 need to work?

That pattern makes sense to me, but is a single family sufficient? IOW, does --healthz-bind-address=10.9.8.0/24,2601:8675::3099/64 need to work?

All of the following are valid combinations

--health-bind-address=10.9.8.0/24,2601:8675::3099/64
--health-bind-address=10.9.8.0/24
--health-bind-address=2601:8675::3099/64

but IIRC the plan is to allow users to configure it via v1alpha2 config, not via command line flags.

Do we have a socketmux Go package already, or is thst something we'll invent?

Do we have a socketmux Go package already, or is thst something we'll invent?

We can do that, we will need to listen on multiple IPs (technically 0.0.0.0/0 is a valid CIDR). That might be handy.

Also would love to have your feedback on #121830 (comment)

/area kube-proxy
see #128303