Currently the ttl is set to: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/controller_utils.go#L44

In theory it can be infinite, but if by some chance (user error/bug) a pod gets orphaned the controller will wait 5m before giving up. It can give up much sooner in most situations. One way aroud this would be to set the timeout to something like: numPods/(watches per second) + padding.

Since we currently don't have an accurate way to track watch latency we need to calculate these value empirically. The comment indicates that watches per second is 10 but it should be set based on the theoretical limit of pods in the cluster, which depends on the number of nodes and allowed pods per node.

This probably involves opening up a watch on all nodes in the rc manager and only paying attention to adds/deletes (since node udpates are really frequent). It also involves determining watches per second for buckets like 100 nodes, 500 nodes, 1000 nodes etc.

If we did this, we'd notice a wedged RC in under 10s in the simple case of 3 nodes and a few orphaned pods.

This would be nice to have for 1.2, I'll try to get to it but anyone with cycles should jump in.