[quorum store] support for pulling OptQS payload with refactoring #14106

ibalajiarun · 2024-07-24T16:08:32Z

Description

This PR adds support for pulling optimistic batches in Quorum Store Proof Manager and returning OptQuorumStore payloads.

Refactors the payload clients including UserPayloadClient trait and implementations and PayloadClient trait and implementations to use a PayloadPullParams struct to manage the increasing number of parameters that have to be passed around.

Introduces PayloadTxnsSize struct to help manage transactions count and bytes together instead of two different things. We use it all throughout QS and it would be nice to have a unified quantity for both quantities. The struct also support arithmetic operations like add, sub.

trunk-io · 2024-07-24T16:08:35Z

⏱️ 2h 56m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-smoke-tests	25m	🟥
general-lints	24m	🟩 🟩 🟩 🟩 🟩 (+9 more)
rust-cargo-deny	23m	🟩 🟩 🟩 🟩 🟩 (+9 more)
check-dynamic-deps	16m	🟩 🟩 🟩 🟩 🟩 (+9 more)
rust-smoke-tests	15m	🟥
rust-move-tests	7m	🟩
rust-move-tests	7m	🟩
rust-move-tests	6m	🟩
rust-move-tests	6m	🟩
rust-move-tests	6m	🟩
rust-move-tests	6m	🟩
semgrep/ci	6m	🟩 🟩 🟩 🟩 🟩 (+9 more)
rust-move-tests	3m	🟩
rust-move-tests	3m	🟩
rust-move-tests	3m	🟩
file_change_determinator	3m	🟩 🟩 🟩 🟩 🟩 (+9 more)
rust-move-tests	3m	🟩
file_change_determinator	3m	🟩 🟩 🟩 🟩 🟩 (+9 more)
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
permission-check	52s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	42s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	41s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	41s	🟩 🟩 🟩 🟩 🟩 (+9 more)
rust-move-tests	28s	⬜

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

bchocho

LGTM

github-actions · 2024-08-14T19:27:16Z

✅ Forge suite `compat` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `62c7fc77f133afb9fc811296d0b7cc5700addc8b`

Compatibility test results for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 62c7fc77f133afb9fc811296d0b7cc5700addc8b (PR)
1. Check liveness of validators at old version: 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5
compatibility::simple-validator-upgrade::liveness-check : committed: 8451.17 txn/s, latency: 3905.75 ms, (p50: 2600 ms, p90: 6900 ms, p99: 20200 ms), latency samples: 293580
2. Upgrading first Validator to new version: 62c7fc77f133afb9fc811296d0b7cc5700addc8b
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7293.87 txn/s, latency: 3920.61 ms, (p50: 4000 ms, p90: 4400 ms, p99: 4700 ms), latency samples: 136120
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6845.74 txn/s, latency: 4240.61 ms, (p50: 4300 ms, p90: 4500 ms, p99: 6300 ms), latency samples: 263420
3. Upgrading rest of first batch to new version: 62c7fc77f133afb9fc811296d0b7cc5700addc8b
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6899.89 txn/s, latency: 4024.83 ms, (p50: 4600 ms, p90: 4800 ms, p99: 5200 ms), latency samples: 129440
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6558.07 txn/s, latency: 4398.22 ms, (p50: 4500 ms, p90: 4900 ms, p99: 6100 ms), latency samples: 249060
4. upgrading second batch to new version: 62c7fc77f133afb9fc811296d0b7cc5700addc8b
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10629.42 txn/s, latency: 2672.61 ms, (p50: 2900 ms, p90: 3100 ms, p99: 3300 ms), latency samples: 192100
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 9954.94 txn/s, latency: 3324.38 ms, (p50: 3100 ms, p90: 4600 ms, p99: 6400 ms), latency samples: 330660
5. check swarm health
Compatibility test for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 62c7fc77f133afb9fc811296d0b7cc5700addc8b passed
Test Ok

github-actions · 2024-08-14T19:27:34Z

✅ Forge suite `realistic_env_max_load` success on `62c7fc77f133afb9fc811296d0b7cc5700addc8b`

two traffics test: inner traffic : committed: 12126.45 txn/s, latency: 3284.64 ms, (p50: 3100 ms, p90: 3600 ms, p99: 4500 ms), latency samples: 4610740
two traffics test : committed: 100.03 txn/s, latency: 2708.47 ms, (p50: 2600 ms, p90: 3400 ms, p99: 5000 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.235, avg: 0.220", "QsPosToProposal: max: 0.369, avg: 0.341", "ConsensusProposalToOrdered: max: 0.323, avg: 0.317", "ConsensusOrderedToCommit: max: 0.757, avg: 0.665", "ConsensusProposalToCommit: max: 1.080, avg: 0.982"]
Max round gap was 1 [limit 4] at version 2599707. Max no progress secs was 7.554624 [limit 15] at version 2599707.
Test Ok

github-actions · 2024-08-14T19:30:03Z

✅ Forge suite `framework_upgrade` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `62c7fc77f133afb9fc811296d0b7cc5700addc8b`

Compatibility test results for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 62c7fc77f133afb9fc811296d0b7cc5700addc8b (PR)
Upgrade the nodes to version: 62c7fc77f133afb9fc811296d0b7cc5700addc8b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1143.93 txn/s, submitted: 1146.19 txn/s, failed submission: 2.26 txn/s, expired: 2.26 txn/s, latency: 2646.86 ms, (p50: 2100 ms, p90: 4700 ms, p99: 9900 ms), latency samples: 101280
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1153.95 txn/s, submitted: 1157.61 txn/s, failed submission: 3.66 txn/s, expired: 3.66 txn/s, latency: 2695.61 ms, (p50: 2100 ms, p90: 4800 ms, p99: 10500 ms), latency samples: 100780
5. check swarm health
Compatibility test for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 62c7fc77f133afb9fc811296d0b7cc5700addc8b passed
Upgrade the remaining nodes to version: 62c7fc77f133afb9fc811296d0b7cc5700addc8b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1196.06 txn/s, submitted: 1198.78 txn/s, failed submission: 2.72 txn/s, expired: 2.72 txn/s, latency: 2587.35 ms, (p50: 2100 ms, p90: 4500 ms, p99: 8100 ms), latency samples: 105600
Test Ok

ibalajiarun force-pushed the balaji/ops-qs-consensus branch from 82d2073 to b120d19 Compare July 25, 2024 02:04

ibalajiarun force-pushed the balaji/opt-qs-proposal branch 2 times, most recently from a1ea209 to 6178210 Compare July 26, 2024 16:37

ibalajiarun changed the title ~~wip~~ [quorum store] support for pulling OptQS payload with refactoring Jul 26, 2024

ibalajiarun force-pushed the balaji/opt-qs-proposal branch 2 times, most recently from 0f6afa1 to dbb150e Compare July 26, 2024 17:03

ibalajiarun marked this pull request as ready for review July 26, 2024 17:08

ibalajiarun requested review from bchocho, sasha8, gelash, zekun000, JoshLind and gregnazario as code owners July 26, 2024 17:08

ibalajiarun force-pushed the balaji/ops-qs-consensus branch from b120d19 to 18906bf Compare July 29, 2024 02:28

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from dbb150e to 8807647 Compare July 29, 2024 02:28

ibalajiarun force-pushed the balaji/ops-qs-consensus branch from 18906bf to 4f1bcd7 Compare July 29, 2024 03:51

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from 8807647 to 1c5d7e7 Compare July 29, 2024 03:51

ibalajiarun force-pushed the balaji/ops-qs-consensus branch from 4f1bcd7 to df5a0b2 Compare July 29, 2024 04:27

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from 1c5d7e7 to 53a61ce Compare July 29, 2024 04:27

ibalajiarun force-pushed the balaji/ops-qs-consensus branch from df5a0b2 to f2d5ac1 Compare July 29, 2024 04:34

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from 53a61ce to 67e3629 Compare July 29, 2024 04:34

ibalajiarun force-pushed the balaji/ops-qs-consensus branch from f2d5ac1 to 835d377 Compare July 29, 2024 05:33

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from 67e3629 to 6ec55f4 Compare July 29, 2024 05:34

ibalajiarun force-pushed the balaji/ops-qs-consensus branch from 835d377 to 1cf52da Compare July 29, 2024 14:31

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from 6ec55f4 to 36d7ce1 Compare July 29, 2024 14:57

ibalajiarun force-pushed the balaji/ops-qs-consensus branch from 44ab923 to 05dc221 Compare July 29, 2024 20:25

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from 36d7ce1 to e72ae81 Compare July 29, 2024 20:25

Base automatically changed from balaji/ops-qs-consensus to main July 29, 2024 20:58

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from e72ae81 to 48ff620 Compare July 29, 2024 21:39

ibalajiarun requested a review from bchocho August 13, 2024 21:28

bchocho approved these changes Aug 13, 2024

View reviewed changes

lint

e6f0287

This comment has been minimized.

Sign in to view

unit tests

e06b9ac

ibalajiarun enabled auto-merge (squash) August 13, 2024 22:15

lint

66fbce7

This comment has been minimized.

Sign in to view

fix

62c7fc7

ibalajiarun force-pushed the balaji/opt-qs-proposal branch from d4e11ba to 62c7fc7 Compare August 14, 2024 18:58

This comment has been minimized.

Sign in to view

ibalajiarun merged commit 3df964b into main Aug 14, 2024
46 checks passed

ibalajiarun deleted the balaji/opt-qs-proposal branch August 14, 2024 19:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[quorum store] support for pulling OptQS payload with refactoring #14106

[quorum store] support for pulling OptQS payload with refactoring #14106

ibalajiarun commented Jul 24, 2024 •

edited

Loading

trunk-io bot commented Jul 24, 2024 •

edited

Loading

bchocho left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Aug 14, 2024

github-actions bot commented Aug 14, 2024

github-actions bot commented Aug 14, 2024

[quorum store] support for pulling OptQS payload with refactoring #14106

[quorum store] support for pulling OptQS payload with refactoring #14106

Conversation

ibalajiarun commented Jul 24, 2024 • edited Loading

Description

trunk-io bot commented Jul 24, 2024 • edited Loading

bchocho left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Aug 14, 2024

✅ Forge suite compat success on 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 62c7fc77f133afb9fc811296d0b7cc5700addc8b

github-actions bot commented Aug 14, 2024

✅ Forge suite realistic_env_max_load success on 62c7fc77f133afb9fc811296d0b7cc5700addc8b

github-actions bot commented Aug 14, 2024

✅ Forge suite framework_upgrade success on 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 62c7fc77f133afb9fc811296d0b7cc5700addc8b

ibalajiarun commented Jul 24, 2024 •

edited

Loading

trunk-io bot commented Jul 24, 2024 •

edited

Loading

✅ Forge suite `compat` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `62c7fc77f133afb9fc811296d0b7cc5700addc8b`

✅ Forge suite `realistic_env_max_load` success on `62c7fc77f133afb9fc811296d0b7cc5700addc8b`

✅ Forge suite `framework_upgrade` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `62c7fc77f133afb9fc811296d0b7cc5700addc8b`