Handle innocuous error messages during control plane startup #2168
Comments
|
@klingerf should the klog setting be |
|
Note that the connection refused lines are at the error level. Error-level logging from client-go may be valuable after successful startup. If possible we should consider dynamically changing the log-level once the server is fully initialized. For reference, here's an example error we see at startup: |
|
@grampelberg If |
grampelberg
changed the title
|
Updated to reflect the problem statement and a more sustainable solution. |
|
For this, what about just hitting the pods |
|
@dadjeibaah That makes sense to me, especially since that's what we're already using for the proxy's readiness checks. I wanted to weigh in with a bit of additional info though. I think this issue is more complicated to fix than it seems on the surface. The easiest fix is to suppress these logs messages entirely, unless Alternatively, if we're worried about debuggability when the logs are suppressed, then we can introduce some sort of delay in the controller components that waits for their sidecar proxies to initialize. Frankly that feels a bit anti-linkerd to me. The whole idea behind proxy injection is that your applications should be unaware that they're running with a sidecar, and in my mind, that includes the control plane. Fwiw, other projects have hit this issue too, and there's been some push for container startup/shutdown ordering supported natively by Kubernetes. The original issue for that was tracked in kubernetes/kubernetes#65502, and it has now been converted to a KEP, here: https://github.com/kubernetes/enhancements/blob/master/keps/sig-apps/sidecarcontainers.md I'm not sure what happens now, but assumedly there will be some support for this in future versions of Kubernetes, and my recommendation would be to wait for that instead of applying application-level fixes at this time. |
siggy
added
priority/P0
…2221) The controller logs innocuous messages when control plane proxies aren't ready to route requests during startup from each control plane component. i.e. tap, public-api and proxy-api. Setting the log level in the control plane to `INFO` would not hide these log messages and would still show up on control plane startup. This PR modifies `klogs` initial flag set to route innocuous logs to `/dev/null` if the controller log level is set to INFO. If set to debug, we output all loglines to stderr. Fixes #2171 #2168 Signed-off-by: Dennis Adjei-Baah <dennis@buoyant.io>
|
Closed by #2221. |
What problem are you trying to solve?
When the controller containers start up, they log a lot of connection refused lines. This is a timing issue between the proxy and the controller container. As the proxy is not yet ready to receive traffic, the connections get retried.
Unfortunately, this scares users and can provide misleading information around debugging. In the past, it was not possible to opt-out of these lines because of glog.
How should the problem be solved?
The underlying problem here is that a
Syncis kicked off before the proxy is ready to forward traffic. This feels like a general linkerd problem, and possibly something that could help our users out.I would like to see a way for sidecars to know when they can start sending traffic out. It feels like this could be a readiness probe for the proxy. Then, the controller code could wait for the probe to pass and continue starting.
Alternatives considered
The text was updated successfully, but these errors were encountered: