Summary

This PR allows a small set of connections (10 reserved in the proposal) to be made to the websocket network, even if at full connection capacity. This solves the Operational Excellence risk that is currently in place, as the public /statusendpoint used for health service checks will fail responding if a server is operating at maximum peer capacity. This subsequently creates a risk that a server may be rebooted / on-call rotation may be receiving a paging alarm; both outcomes which should not happen if it's a matter of a server operating at max peer limit.

The 10 reserved connections will be used from the reserved FD pool of 256. In my testing an algod node uses 56 open connections if running on an M2 Mac from within GoLand, meaning there's a lot of headroom. 10 reserved connection limit was chosen as multiple load balancers / status check scripts may be in front of an algod node.

The underlying issue was discussed in #5766, but with no remediation of underlying issue (health service endpoint being unreliable under load), and where proposed solutions with tokens / limits would not work for the newly introduced public health service endpoint.

FAQ

Q: Why not introduce a new IncomingGossipLimit parameter?
A: This could be done, however it would still mean that additional headroom somehow would have to be permitted, as changing semantics of existing configuration may leave node runners behind, as they may not adopt new configuration parameters. I see no great way to change semantics of the current parameter without breaking backwards compatibility.

Q: Don't all servers have plenty of headroom with regards to peer limit?
A: This may or may not be the case depending on the network that the relays are operating on.

Q: Could this not be done on a different service endpoint?
A: To fit into where the health service endpoint is running, and which is part of the gossip service, we need to allow connections to that specific endpoint.

Q: Are there other places where the existing limit is enforced?
A: Yes. Existing limit is enforced by the checkIncomingConnectionLimits function in wsNetwork.go, which happens later in the code path after the rejectingLimitListener has been executed.

Q: Will this cause an issue with file descriptors?
A: It should not. There is a pool of 256 FD's reserved for short-lived operations, and where up to 10 additional FD may be used in a short-lived fashion.

Q: Why not make the change in rejectingLimitListener.go?
A: This runs before we know the request route, and as such would require parsing HTTP payloads before middleware can access it.

Q: Will this introduce breaking changes as some requests may be accepted and not served?
A: This will change the response mechanism for up to 10 requests that may be accepted before being dropped. requestTracker do already limit and respond with HTTP failure codes (4xx / 5xx) if a client is being throttled. Responding with a 503 does not change client behavior in my testing.

Q: Why make a change in requestTracker.go?
A: The exception path in requestTracker.go is to enforce limits for other services. The gossip protocol currently checks connection limits, however other services would be able to service requests without the check in requst tracker.

Q: Why not use HealthServiceStatusPath?
A: This leads to a circular dependency. Would love to extract paths across all services if so makes sense and put into a route path configuration. For now opted for duplication to reduce amount of changes, as this is the first change where routes are checked directly in the requestTracker. If additional route checks are needed I think refactoring at that point in time would be advisable. Current PR operates with the 2 > 0 principle in mind - it's better to have an improvement, even if duplicated, vs no improvement.

Test Plan

Setup a relay server by configuration NetAddress, and set "IncomingConnectionsLimit": 0.

Make requests to

• /v1/{genesis}/gossip. Expected fail with HTTP response code 503
• /v1/{genesis}/block/1. Expected fail with HTTP response code 503
• /status. Expected success with HTTP response code 200

Set "IncomingConnectionsLimit": 1.
Make requests to

• /v1/{genesis}/gossip. Expected response code 412 as pre-conditions are not fulfilled.
• /v1/{genesis}/block/1. Expected response code 404 as block does not exist
• /status. Expected success with HTTP response code 200