Forked from #260. Aka leader election.

Applications can use their own master-election/lock services, such as etcd or zookeeper. However, a problem remains, which is how clients of those services find the current master. Some applications, such as databases and key-value stores, provide their own client libraries or special protocols for this. Clients of HTTP APIs could potentially use load balancing plus redirects.

The question is should we do anything to make this easier?

We could:

• assign an IP address and DNS name to the current master
• reflect the current mastership status via the cluster API and via the downward API (Container downward/upward API umbrella issue #386)
  • applications that need true exclusion could use resourceVersion as the "lock sequence number" for their own concurrency control
• create lifecycle hooks for acquiring and losing mastership

If we decide to support this, I think the main design decision is whether the application should participate directly in the master election process. We have an etcd-based master-election implementation. We could expose an API for master election that would be independent of the underlying key-value store, but which would couple the application to that API. Or, we could potentially use container status and/or readiness probes (#620) to decide when to transfer mastership to another instance. Using readiness probes (or a variant thereof) would have the nice property that the application could use that mechanism to influence which instance was elected master, such as in the case that the application just wanted to take advantage of the addressing/naming mechanism.