caches have a lifetime of 10 minutes, and can be cleared client-side.

it's not perfect, but this is no different than how kubectl behaved with dynamic discovery changes due to registered/unregistered TPRs in prior releases, so I don't consider it a release blocker.

What are you doing to trigger this problems? kubectl get foo check its cache, fails, re-primes the cache, checks again, succeeds, all silently to the user.

What should other clients that are not kubectl do? This is a breaking change for kubefed for example. I am going to add the release-blocker label back, until we decide it is not for clients that are not kubectl.

Btw, I forgot to mention that kubefed uses kubectl's cmd libraries.

again, this behavior is no different than how prior releases handled dynamically added/removed APIs

@liggitt Do we understand why this cause ci-kubernetes-e2e-gce-federation failing #47737?

To my knowledge, nothing has changed in this area anytime recently. Didn't the Federation job just start failing a few days ago?

If the autoregister-completion health check is not done, the /healthz will return 500. can we let kubectl check the /healthz before visiting the discovery endpoints?

I0623 02:53:30.095497       5 wrap.go:42] GET /healthz: (571.071µs) 500
...[-]autoregister-completion failed: reason withheld\n

This seems like much ado about a problem we've had since 1.2 (I think) when we got TPRs. The discovery doc changes in response to those and clients already had to deal with it. The same thing happens on upgrades and even server restarts since we added API groups years ago.

Claiming this as a new problem is a mischaracterisation since we've had it for years. In addition, the exact case described here is incorrect (kubectl live checks on a miss).

@deads2k I am not sure about when we introduced the issue here. But what you described for upgrade is matching what I observed during upgrade test debugging for 1.7.

The only reason we mark this as the release blocker is the federation team think this is the root cause for #47737. We need to find the solution / workaround / document as the known issue for federation / upgrade / etc.. The long term fixing is not required here.

@deads2k I am not sure about when we introduced the issue here. But what you described for upgrade is matching what I observed during upgrade test debugging for 1.7.

@dchen1107 Then I would avoid trying to make rushed API changes to discovery or behavior modifications to kubectl. The aggregation behavior people seem focused on merged in March with this pull here #42911 .

This is a behavior change on the server that exposes a caching bug (which itself is probably both a server and client bug).

There is no reason for apiserver to return a partial list of built-in resources at any point in its startup stack. I consider that a bug, it is exposing internal setup details to the world for no good reason. Since it is breaking clients and we aren't sure the etag is set correctly to allow caches to work, I think we need to fix the server bug.

Caching for X amount of time is also clearly really wrong now that we are super dynamic, the server must present an ETag and the client needs to use the If-None-Match header to do it right.

In the Burndown meeting, the decision was not to fix API server, only document it as a known issue.

the kubectl and kubefed bugs related to cache usage will be fixed in 1.7 (and picked back to 1.6.x)

kubectl:
master: #48016 (merged)
1.7 pick: #48067 (merged)
1.6 pick: #48070 (merged)

kubefed:
master: #48077 (merged)
1.7 pick: #48100 (merged)
1.6 pick: #48101 (merged)

xref: v1.7 known issue: #46733

@liggitt Given that the pr's you've linked to have all merged, should this issue continue blocking 1.7? If not, please remove the 'approved-for-milestone' label.

The kubectl api-versions and kubefed cache usage has been fixed for 1.7

issue tracking caching headers / etag support is #44957