-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(api): introduce mempool cache #1460
Merged
RomanBrodetski
merged 25 commits into
main
from
rb-pla-672-optimize-db-reads-for-getfilterchanges
Mar 21, 2024
Merged
feat(api): introduce mempool cache #1460
RomanBrodetski
merged 25 commits into
main
from
rb-pla-672-optimize-db-reads-for-getfilterchanges
Mar 21, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RomanBrodetski
commented
Mar 19, 2024
…nsactionFilter` filter
RomanBrodetski
force-pushed
the
rb-pla-672-optimize-db-reads-for-getfilterchanges
branch
from
March 19, 2024 16:56
e5bd818
to
a020bb2
Compare
popzxc
previously approved these changes
Mar 20, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. A few minor nits, feel free to ignore.
popzxc
reviewed
Mar 20, 2024
popzxc
reviewed
Mar 20, 2024
perekopskiy
reviewed
Mar 20, 2024
perekopskiy
approved these changes
Mar 21, 2024
RomanBrodetski
deleted the
rb-pla-672-optimize-db-reads-for-getfilterchanges
branch
March 21, 2024 17:26
This was referenced Mar 21, 2024
github-merge-queue bot
pushed a commit
that referenced
this pull request
Mar 22, 2024
🤖 I have created a release *beep* *boop* --- ## [22.0.0](core-v21.1.0...core-v22.0.0) (2024-03-21) ### ⚠ BREAKING CHANGES * Use protocol version v22 as latest ([#1432](#1432)) ### Features * add docs for Dal ([#1273](#1273)) ([66ceb0b](66ceb0b)) * adds debug_traceBlockByNumber.callFlatTracer ([#1413](#1413)) ([d2a5e36](d2a5e36)) * **api:** introduce mempool cache ([#1460](#1460)) ([c5d6c4b](c5d6c4b)) * **commitment-generator:** `events_queue` shadow mode ([#1138](#1138)) ([9bb47fa](9bb47fa)) * **contract-verifier:** Allow sc code reverification ([#1455](#1455)) ([5a37b42](5a37b42)) * **database:** add an optional master replica max connections settings ([#1472](#1472)) ([e5c8127](e5c8127)) * **db:** Configurable maximum number of open RocksDB files ([#1401](#1401)) ([b00c052](b00c052)) * **en:** Check recipient contract and function selector in consistency checker ([#1367](#1367)) ([ea5c684](ea5c684)) * Follow-up for DAL split ([#1464](#1464)) ([c072288](c072288)) * **node_framework:** Ergonomic improvements ([#1453](#1453)) ([09b6887](09b6887)) * **node_framework:** Only store each wiring layer once ([#1468](#1468)) ([4a393dc](4a393dc)) * **node_framework:** Support for preconditions and oneshot tasks ([#1398](#1398)) ([65ea881](65ea881)) * **node-framework:** Add eth sender layer ([#1390](#1390)) ([0affdf8](0affdf8)) * **node-framework:** Add housekeeper layer ([#1409](#1409)) ([702e739](702e739)) * Remove batch dry_run ([#1076](#1076)) ([b82d093](b82d093)) * Separate Prover and Server DAL ([#1334](#1334)) ([103a56b](103a56b)) * **storage:** Make the storage caches task cancellation aware ([#1430](#1430)) ([ab532bb](ab532bb)) * support running consensus from snapshot (BFT-418) ([#1429](#1429)) ([f9f4d38](f9f4d38)) * **tx-sender:** Limit concurrent tx submissions ([#1473](#1473)) ([4bdf3ca](4bdf3ca)) * Use protocol version v22 as latest ([#1432](#1432)) ([1757412](1757412)) * **vm:** Prestate tracer implementation ([#1306](#1306)) ([c36be65](c36be65)) ### Bug Fixes * **core:** drop correct index of proof_generation_details table during database migration ([#1199](#1199)) ([76a6821](76a6821)) * **dal:** correct `tx_index_in_l1_batch` in `get_vm_events_for_l1_batch` ([#1463](#1463)) ([8bf37ac](8bf37ac)) * **node_framework:** Fix main node example ([#1470](#1470)) ([ac4a744](ac4a744)) * **protocol:** Remove verifier address from protocol upgrade ([#1443](#1443)) ([90dee73](90dee73)) * **reth:** use reth instead of geth ([#1410](#1410)) ([bd98dc7](bd98dc7)) * **verified sources fetcher:** Use correct `contact_name` for `SolSingleFile` ([#1433](#1433)) ([0764227](0764227)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
montekki
pushed a commit
that referenced
this pull request
Mar 22, 2024
This introduces an in-memory cache on ws API servers that caters for `eth_newPendingTransactionFilter` filter. Currently for each `eth_getFilterChanges` request to such filter, we do a DB query. This query dominates the DB load of ws DBs. The cache is updated preiodically, loading the new mempool transactions to a `Deque`. To return all the transactions that happened after a particular timestamp, we binsearch this data strucute - avoiding additional DB calls. to relieve some load from the read replica DB.
montekki
pushed a commit
that referenced
this pull request
Mar 22, 2024
🤖 I have created a release *beep* *boop* --- ## [22.0.0](core-v21.1.0...core-v22.0.0) (2024-03-21) ### ⚠ BREAKING CHANGES * Use protocol version v22 as latest ([#1432](#1432)) ### Features * add docs for Dal ([#1273](#1273)) ([66ceb0b](66ceb0b)) * adds debug_traceBlockByNumber.callFlatTracer ([#1413](#1413)) ([d2a5e36](d2a5e36)) * **api:** introduce mempool cache ([#1460](#1460)) ([c5d6c4b](c5d6c4b)) * **commitment-generator:** `events_queue` shadow mode ([#1138](#1138)) ([9bb47fa](9bb47fa)) * **contract-verifier:** Allow sc code reverification ([#1455](#1455)) ([5a37b42](5a37b42)) * **database:** add an optional master replica max connections settings ([#1472](#1472)) ([e5c8127](e5c8127)) * **db:** Configurable maximum number of open RocksDB files ([#1401](#1401)) ([b00c052](b00c052)) * **en:** Check recipient contract and function selector in consistency checker ([#1367](#1367)) ([ea5c684](ea5c684)) * Follow-up for DAL split ([#1464](#1464)) ([c072288](c072288)) * **node_framework:** Ergonomic improvements ([#1453](#1453)) ([09b6887](09b6887)) * **node_framework:** Only store each wiring layer once ([#1468](#1468)) ([4a393dc](4a393dc)) * **node_framework:** Support for preconditions and oneshot tasks ([#1398](#1398)) ([65ea881](65ea881)) * **node-framework:** Add eth sender layer ([#1390](#1390)) ([0affdf8](0affdf8)) * **node-framework:** Add housekeeper layer ([#1409](#1409)) ([702e739](702e739)) * Remove batch dry_run ([#1076](#1076)) ([b82d093](b82d093)) * Separate Prover and Server DAL ([#1334](#1334)) ([103a56b](103a56b)) * **storage:** Make the storage caches task cancellation aware ([#1430](#1430)) ([ab532bb](ab532bb)) * support running consensus from snapshot (BFT-418) ([#1429](#1429)) ([f9f4d38](f9f4d38)) * **tx-sender:** Limit concurrent tx submissions ([#1473](#1473)) ([4bdf3ca](4bdf3ca)) * Use protocol version v22 as latest ([#1432](#1432)) ([1757412](1757412)) * **vm:** Prestate tracer implementation ([#1306](#1306)) ([c36be65](c36be65)) ### Bug Fixes * **core:** drop correct index of proof_generation_details table during database migration ([#1199](#1199)) ([76a6821](76a6821)) * **dal:** correct `tx_index_in_l1_batch` in `get_vm_events_for_l1_batch` ([#1463](#1463)) ([8bf37ac](8bf37ac)) * **node_framework:** Fix main node example ([#1470](#1470)) ([ac4a744](ac4a744)) * **protocol:** Remove verifier address from protocol upgrade ([#1443](#1443)) ([90dee73](90dee73)) * **reth:** use reth instead of geth ([#1410](#1410)) ([bd98dc7](bd98dc7)) * **verified sources fetcher:** Use correct `contact_name` for `SolSingleFile` ([#1433](#1433)) ([0764227](0764227)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
6 tasks
github-merge-queue bot
pushed a commit
that referenced
this pull request
Mar 26, 2024
## What ❔ Small changes after #1460: - `MempoolCache` is moved to the core crate as per its domain. - Some minor things like dropping DB connections earlier. ## Why ❔ Marginally simplifies maintenance. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. - [x] Spellcheck has been run via `zk spellcheck`. - [x] Linkcheck has been run via `zk linkcheck`.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Mar 28, 2024
🤖 I have created a release *beep* *boop* --- ## [12.2.0](prover-v12.1.0...prover-v12.2.0) (2024-03-28) ### Features * **api:** introduce mempool cache ([#1460](#1460)) ([c5d6c4b](c5d6c4b)) * **commitment-generator:** `events_queue` shadow mode ([#1138](#1138)) ([9bb47fa](9bb47fa)) * Drop prover tables in core database ([#1436](#1436)) ([0d78122](0d78122)) * Follow-up for DAL split ([#1464](#1464)) ([c072288](c072288)) * **prover:** export prover traces through OTLP ([#1427](#1427)) ([16dce75](16dce75)) * **prover:** File-info tool to help prover debugging ([#1216](#1216)) ([9759907](9759907)) * Separate Prover and Server DAL ([#1334](#1334)) ([103a56b](103a56b)) * support running consensus from snapshot (BFT-418) ([#1429](#1429)) ([f9f4d38](f9f4d38)) ### Bug Fixes * **api:** Fix API server shutdown flow ([#1425](#1425)) ([780f6b0](780f6b0)) * **prover:** Remove FriProtocolVersionId ([#1510](#1510)) ([6aa51b0](6aa51b0)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Roman Brodetski <rb@matterlabs.dev>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
This introduces an in-memory cache on ws API servers that caters for
eth_newPendingTransactionFilter
filter. Currently for eacheth_getFilterChanges
request to such filter, we do a DB query. This query dominates the DB load of ws DBs.The cache is updated preiodically, loading the new mempool transactions to a
Deque
. To return all the transactions that happened after a particular timestamp, we binsearch this data strucute - avoiding additional DB calls.Why ❔
to relieve some load from the read replica DB.