Real clocks or physical time are not perfectly accurate even time synchronization is on. Some kind of virtual time could be better.

cc @smarterclayton

The node should have access to the clock skew of the master via the date header returned on responses. It's likely that in the future we'd want a story to deal with node/master clock skew consistently. We have ve said you must run your cluster with low time skew.

I think in the short term the deployment controller should create a virtual clock (like the node controller and namespace controller do) for dealing with timestamps. The deployment controller can at least bracket between creation time and pod accepted by kubelet time and start the clock when it observes the pod. I think we want to remove any time.Now() comparisons and replace them with observed clock skew from the master's reference frame.

xref: #6159

An alternative to virtual clocks is minReadySeconds on the pod level (@bgrant0607 has already asked for it).

Note that the node controller already had to deal with this. IIRC, it didn't actually use the timestamps in the condition, but started its own timers when observing transitions.

The kubelet, namespace, and node controller also do that for graceful
deletion - starting timers when they observe the state change, but not
treating the timestamp in the object as meaningful.

On Tue, Sep 27, 2016 at 10:59 PM, Brian Grant notifications@github.com
wrote:

Note that the node controller already had to deal with this. IIRC, it
didn't actually use the timestamps in the condition, but started its own
timers when observing transitions.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#29229 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p5OkpUvUjF8yI8xXKlTj8KpvprNbks5qudgOgaJpZM4JQLjm
.

We need a virtual clock in the deployment controller also for deployment conditions. See #35691 (comment) for more context.

I like the option of a private timer in the deployment controller.

We think we have seen this in the wild. Too late to fix in 1.5. Targeting 1.6.

FWIW, the availability check "moved" in the replica set controller, meaning we should add the virtual clock in the replica set controller if we want to solve this by using a virtual clock. I guess all controllers that deal with Pods should use virtual clocks in the long term.

An alternative to virtual clocks is minReadySeconds on the pod level

I've got kubernetes/community#194 up for review (proposal for adding minReadySeconds to pods) and I have a preliminary buggy working implementation of it locally but the concerns raised by @thockin regarding the ready condition being ambiguous already got me thinking of following the simplest solution possible ie. using a virtual clock in the replica set controller.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close