Is the decision to unbind a pod from a host (delete it) a decision a node controller makes (delete all pods on node, then delete node)? Or is it reactive by the system from the node being deleted (node deleted, therefore some process needs to clean up pods)? The former seems like annotations and a custom node controller would solve (delete all pods immediately that have forgiveness true). The latter seems like it would have to be supported by the system itself.

This is probably a separate issue but I don't know where it is - should a node have a concept of being unavailable for scheduling?

@smarterclayton Yes, a node should have a concept of being unavailable for scheduling, and the system (a node controller) should also be responsible for removing pods from non-functional nodes -- see #1366. This separates concerns of pod controlling vs. node controlling (for the most part). Both control systems will get more and more sophisticated over time.

Sketch of forgiveness from #3058:

// goes in Pod
    ForgivenessPolicy *ForgivenessPolicy `json:"forgivenessPolicy,omitempty"`

type ForgivenessPolicy struct {
    Thresholds []ForgivenessThreshold `json:"thresholds"`
}
type ForgivenEvent string
const (
    ForgivenDead ForgivenEvent = "Dead"
    ForgivenNotReady ForgivenEvent = "NotReady"
    ForgivenDeadNode ForgivenEvent = "DeadNode"
    ForgivenUnreachableNode ForgivenEvent = "UnreachableNode"
)
type ForgivenessThreshold struct {
    Event ForgivenEvent `json:"event"`
    ContiguousSeconds int `json:"contiguousSeconds,omitempty"`
    Rate int `json:"rate,omitempty"`
    RatePeriodSeconds int `json:"ratePeriodSeconds,omitempty"`
}

OpenShift implemented a "manage-node" command which models an evacuate action. Currently it's up to the admin to decide whether the pods will be moved, but a forgiveness value on the pod could be the input to that command (just like the automatic evacuation implemented in node controller). A node with a pod with a long forgiveness policy is also less transient, so there is potentially value in making scheduling decisions based on forgiveness (where increasing the forgiveness of a given node has a cost, but scheduling a pod with a lower forgiveness is free). That's probably more a model for more traditional operations teams, and is certainly not a high priority.

Do we want to simplify forgiveness to try to get it for 1.1? As a general policy value it's useful to have in the system.

I don't know if it would be high enough priority to commit to for 1.1, but I could imagine the first step would just be a bool (Pod is treated normally, i.e. deleted by NodeController after standard timeout, vs. Pod stays bound to the node forever unless manually deleted or node is deleted)..

How about a duration?

We want to dramatically reduce the fail detecting time for certain
workloads on the cluster, which implies some way of distinguishing pods on
an unhealthy node that should be aggressively considered down (such that
the replication controller gets the earliest chance to proceed).

Maybe there's a more tactical way to manage that, but that does seem
something that would let admins start to manage SLA more effectively that
fits into the long term pattern.

That seems reasonable to me, I was just suggesting the simplest thing we could do. What you suggested would also make it a subset of what @bgrant0607 suggested in his comment above, so
#1574 (comment)
you'd just have a ContiguousSeconds and basically nothing else. (As opposed to the bool which doesn't really fit into what he suggested).

Not sure about prioritization for 1.1 though.

If we added the value, but did not leverage it in node controller (or made
it the simplest possible expression, such as changing the evacuate behavior
to be the min forgiveness), would we be in a bad spot?

As far as prioritization it's something we'd be interested in implementing,
so it's really a question of API design and scope of change.

Is there any doc that has a couple of examples of concrete real world applications and what forgiveness values we recommend for Taints and Tolerations?

We have a few apps running that are "must never be evicted" where the user
has chosen to turn off node eviction today. I have fewer examples of
30 except a few production dbs that would prefer to only be
moved if absolutely necessary because restore >30 min

On Oct 29, 2016, at 10:08 AM, Eric Tune notifications@github.com wrote:

Is there any doc that has a couple of examples of concrete real world
applications and what forgiveness values we recommend for Taints and
Tolerations?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1574 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pxJruBotk6j9s0vX4xKZBhw6DyGgks5q41NBgaJpZM4CqrbH
.

Yeah I think there are three categories

• want to be evicted sooner than the default 5 minutes -- we have an example of this from a GKE customer running a session-based application where every container handles a set of users, and if the node becomes unavailable that set of users can't be served until a replacement container is created on a new node (they requested 1 minute timeout)
• want to be evicted after more than 5 minutes but less than infinite -- I think Clayton answered this one; unreplicated apps that take a long time to rebuild their state if restarted on a different node
• never want to be evicted -- anything that runs as a DaemonSet today falls into this category (we can get rid of the hacky code that handles DaemonSet specially with respect to evictions, and just give them infinite tolerations); sounds like Clayton has other examples (I would think anything that is imported from a legacy VM world where the app can't handle "moving" once started without operator intervention and can't be rewritten to be a PersistentSet would fall into this category)

Anything taking to legacy network attached storage (iscsi, fibrechannel,
nfs over SAN) will probably want to be using stateful set + infinite
forgiveness for quite a while until we implement volume locking and have a
fencer in place. That's most mission critical data stores I'm aware of -
Oracle, SAP, etc. Some of those will be "nodes as pets" where there is
extrinsic HA applied to the node and eviction would be impractical.

In terms of general "not being surprised by the platform" there are a class
of existing workloads that should have very high forgiveness until they
have time to verify they can be moved.

On Oct 29, 2016, at 2:52 PM, David Oppenheimer notifications@github.com
wrote:

Yeah I think there are three categories

• want to be evicted sooner than the default 5 minutes -- we have an
  example of this from a GKE customer running a session-based application
  where every container handles a set of users, and if the node becomes
  unavailable that set of users can't be served until a replacement container
  is created on a new node (they requested 1 minute timeout)
• want to be evicted after more than 5 minutes but less than infinite --
  I think Clayton answered this one; unreplicated apps that take a long time
  to rebuild their state if restarted on a different node
• never want to be evicted -- anything that runs as a DaemonSet today
  falls into this category (we can get rid of the hacky code that handles
  DaemonSet specially with respect to evictions, and just give them infinite
  tolerations); sounds like Clayton has other examples (I would think
  anything that is imported from a legacy VM world where the app can't handle
  "moving" once started and can't be rewritten to be a PersistentSet would
  fall into this category)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1574 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p2jWRQ5cv8IHLjj7oSg6tUcLbPnaks5q45XkgaJpZM4CqrbH
.