Related to #5476, #2003, #1007, #5620, and other issues.

The PR creation rate and number of contributors are both growing. In order to scale the effort, we need to better document the patterns we want additions to the system and API to follow, so that we can better parallelize the design and review effort. I'm filing this issue to solicit important principles and patterns to document. I volunteer to create the actual document and/or presentation slides.

What do you think isn't sufficiently covered by existing design docs and/or isn't sufficiently discoverable? What principles/patterns do you find yourself explaining over and over again? What properties would you be unhappy to have violated/degraded/changed by new PRs?

I'm thinking more about reasonably general issues rather than implementation-oriented invariants that apply to very specific/narrow parts of the code.

Examples (some areas, some principles):

• Declarative APIs, desired state and current state
• Level-based operation: Functionality must be level-based, meaning the system must operate correctly given the desired state and the current/observed state, regardless of how many intermediate state updates may have been missed. Edge-based behavior must be just an optimization.
• Status must be 100% reconstructable by observation. Any history kept must be just an optimization and not required for correct operation.
• Phase, Reason, Message, Conditions, etc. vs. a single state machine with lots of states
• Composable, a la carte functionality
• Transparent, no glass to break
• More generally, Eric Raymond's 17 UNIX rules
• Only the apiserver should communicate with etcd/store, and not other components (scheduler, kubelet, etc.).
• Compromising a single node shouldn't compromise the cluster (not that we've achieved that yet)
• Components should continue to do what they're told in the absence of new instructions.
• Open world assumption: continually verify assumptions and gracefully adapt to external events and/or actors (example: we allow users to kill pods under control of a replication controller; it just replaces them)
• Don't assume a component's decisions will not be overridden or rejected, nor for the component to always understand why. For example, etcd may reject writes. Kubelet may reject pods. The scheduler may not be able to schedule pods.
• When to use labels vs. annotations vs. object references
• Watch vs. polling
• Bootstrapping-related principles

Let's brainstorm first, and argue and prioritize afterward.

@smarterclayton @derekwaynecarr @thockin @erictune @lavalamp