-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Manually implement client.Wait
backoffs
#4200
Conversation
cd792e9
to
d09ed9d
Compare
This reverts commit 1aef766. Signed-off-by: Justin Chadwell <me@jedevc.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The automatic retries for transient errors and recover gracefully when the service is unavailable temporarily looks good enough but I wonder if we should consider a maximum retry limit as well.
I think the |
Looks good 👍 |
d09ed9d
to
8b6fb78
Compare
When calling client.Wait, we want to avoid the default backoff behavior, because we want to achieve a quick response back once the server becomes active. To do this, without modifying the entire client's exponential backoff configuration, we can use conn.ResetConnectBackoff, while attempting to reconnect every second. Here are some common scenarios: - Server is listening: the call to Info succeeds quickly, and we return. - Server is listening, but is behind several proxies and so latency is high: the call to Info succeeds slowly (up to minConnectTimeout=20s), and we return. - Server is not listening and gets "connection refused": the call to Info fails quickly, and we wait a second before retrying. - Server is not listening and does not respond (e.g. firewall dropping packets): the call to Info fails slowly (by default after minConnectTimeout=20s). After the call fails, we wait a second before retrying. Signed-off-by: Justin Chadwell <me@jedevc.com>
8b6fb78
to
f1d7f2e
Compare
🛠️ Fixes regression #4164, while preserving the spirit of #4015.
◀️ Reverts #4015.
When calling
client.Wait
, we want to avoid the default backoff behavior, because we want to achieve a quick response back once the server becomes active.To do this, without modifying the entire client's exponential backoff configuration, we can use
conn.ResetConnectBackoff
, while attempting to reconnect every second.Here are some common scenarios under the new
client.Wait
implementation:Info
succeeds quickly, and we return.Info
succeeds slowly (up tominConnectTimeout=20s
), and we return.minConnectTimeout
, thenclient.Wait
will never return."connection refused"
: the call toInfo
fails quickly, and we wait a second before retrying.Info
fails slowly (by default afterminConnectTimeout=20s
). After the call fails, we wait a second before retrying.This PR is split into two commits:
client.Wait
.