Thanks for reporting @airstand I've also seen this issue.

We've seen the same issues happening. This also makes running preemptible clusters on GKE impossible, because you'll experience cluster-wide DNS outage when a node is swapped for a new one.

In the mean time, we've updated the kube-dns-autoscaler ConfigMap to scale the dns deployment to a minimum of two pods, instead of one.

We are not able to change ConfigMap in case we are using kops to deploy the whole cluster.

@airstand How did you fail to change the kube-dns-autoscaler ConfigMap? I don't think kops caused this.

cc @kubernetes/dns-maintainers

I'm wondering; is there any downside to just scaling kube-dns to as many nodes as possible, to have the highest possible availability, besides from a minor increase in required resources?

We've set it to two, but I'm still not sure if this is enough, as I now see one of the pods not starting because DNSMasq doesn't have any inodes available (an issue we've been seeing for some time now, but are unable to find its cause), and the second kube-dns pod was just deleted because its preemptible node was terminated, again causing a cluster-wide DNS outage.

Ideally (at least for now), we'd scale the kube-dns pods to the amount of machines.

I'm unsure however if having two dns pods on the same machine can cause problems, since there is no guarantee (since these aren't DaemonSets, and pod anti-affinity isn't available yet) that the pods won't end up on the same machine.

@JeanMertz -- Can you post the log for DNSmasq inode problem?

Apart from resource utilization, there should be no problem with running more copies of the DNS service.

I would not recommend use of pre-emptable nodes for running important services such as DNS however.

@JeanMertz I also want to see what the inode problem is?

Btw.. is a great idea to have kube-dns deployed on each node.

@bowei @airstand I also replied here about the inode error: #32526 (comment)

I would not recommend use of pre-emptable nodes for running important services such as DNS however.

Fair point. Until now, we really haven't modified any "default" Kubernetes resources in the kube-system namespace, but since we started doing this anyway, by updating the kube-dns-autoscaler ConfigMap, we might as well update the kube-dns deployment to add an affinity to the non-preemptible machines we still have running.

Looks like inter-pod anti-affinity is going to beta in 1.6 (#25319)?

BTW, you won't be able to update the kube-dns deployment if it's managed by Addon-manager (#36411).

@MrHohn thanks for those links.

I'm guessing if this issue is converted to a PR, it would be nice if anti-affinity is added to the kube-dns deployment as well (given both features land in 1.6).

Also, regarding your link about the Addon-manager, does this immutability also apply to the configmap used by the deployment to configure itself? It obviously isn't right now (since we scaled our kube-dns to more than one pod), but I hope that won't change in the future without this issue being fixed.

I'm guessing if this issue is converted to a PR, it would be nice if anti-affinity is added to the kube-dns deployment as well (given both features land in 1.6).

Note that pod anti-affinity is alpha in 1.5 and can be specified as an annotation (scheduler.alpha.kubernetes.io/affinity) in the pod template.
https://kubernetes.io/docs/user-guide/node-selection/

@Kargakis Thanks, but we're running on GKE, so that's not a solution to us unfortunately 😢

@JeanMertz Yeah, make sense to me to put anti-affinity into the kube-dns deployment.

The tricky point about kube-dns-autoscaler ConfigMap is that it is managed by kube-dns-autoscaler instead of Addon-manager. So it is allowed to be modified, and will be re-created with the template value if it is deleted. I believe this pattern will not change for kube-dns-autoscaler.

@JeanMertz, @airstand -- adding anti-affinity to the kube-dns pod spec would be great target for community contribution :-)

@thockin that's not what we are seeing on GKE.

This is what is currently configured on our cluster:

$ k get no --no-headers | wc -l
       5

$ k get --namespace=kube-system -ojson cm kube-dns-autoscaler | jq -r .data.linear
{"coresPerReplica":256,"min":2,"nodesPerReplica":16}

but this is after we manually set the min value to 2, it was set to 1 ever since we booted this (and one other) cluster on GKE.

@thockin Sure, I will change it back.

I would like to see some way of giving kube-dns pods affinity, because as @bowei said, having kube-dns on a preemptible node is not a good idea. In our current setup we have 3 preemptible nodes and 2 regular nodes, and kube-dns often lands on a preemptible node.

/sig cluster-lifecycle

/sig network

Hello,

What is the status of this one ? On my 2 GKE clusters running 1.7.6 I still have a kube-dns deployment configured with only 1 replica.
My kube-dns-autoscaler configmap is still:

{"coresPerReplica":256,"min":1,"nodesPerReplica":16}

@rvrignaud We've updated the default parameters to ensure at least 2 replicas when cluster has >= 2 nodes by #42065. Your clusters are using the old default parameters likely due to #45851 --- we don't update those params if cluster is upgraded from an older version.

For getting the latest default params for the confimap, you could:

• Delete kube-dns-autoscaler configmap in kube-system namespace and dns-horizontal-autoscaler pod will recreate a new configmap with default params. OR
• Manually edit that confimap. Please take https://github.com/kubernetes/kubernetes/blob/v1.7.6/cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml#L47 as reference.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale