config: make Tx dedupe cache maxSize configurable #5419
Conversation
Thanks for the catch. I generated it locally to ensure that config tests pass locally but forgot to add the new file to my commit. Fixed now |
|
Please rebase - I think the e2e test failure is fixed in master |
Codecov Report
@@ Coverage Diff @@
## master #5419 +/- ##
==========================================
+ Coverage 55.39% 55.40% +0.01%
==========================================
Files 447 447
Lines 63313 63312 -1
==========================================
+ Hits 35072 35080 +8
+ Misses 25849 25842 -7
+ Partials 2392 2390 -2
... and 5 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| @@ -171,13 +171,11 @@ func MakeTxHandler(opts TxHandlerOpts) (*TxHandler, error) { | |||
| streamVerifierDropped: make(chan *verify.UnverifiedTxnSigJob), | |||
| } | |||
|
|
|||
| // use defaultBacklogSize = approx number of txns in a full block as a parameter for the dedup cache size | |||
There was a problem hiding this comment.
Your PR describes what change is being made, but could you also include why? it would be helpful if we ever need to refer back to this change. This comment describes a reasonable estimation, so why 10x it?
(I know we've discussed it plenty in team settings, but I want to make sure it's recorded)
Summary
Allow node runners to set custom max size for the incoming transaction deduplication. Setting it higher could decrease the CPU load and the amount of egress data on the networking size at the cost of extra memory.
This PR introduces a new config
TxIncomingFilterMaxSizeset to 500k. This is just under a 10x increase from the previous value which depended on the TxBacklogSize the default for which is 26000 * 2. At periods of high transaction volume it's possible to see many more transactions than the two block estimate and this change allows individual operators to set appropriate limits based on the traffic they observe and the memory overhead that they have.Test Plan
Existing tests