Integrating Nested Prover Storage Management #1197

citizen-stig · 2023-12-01T12:18:25Z

Description

Main Changes

sov_db::StateDB and sov_db::NativeDB are both using DbSnapshot. This means that fork management is supported, as each fork can have its own instance of ProverStorage that points correctly to previous storages and DB
StorageManager trait has bee changed to HierarchicalStorageManager trait, that reflects DA driven nature of creating storage snapshots.
stf-runner uses new implementation of ProverStorageManager that creates snapshots based on DA blocks and maintains all forking related machinery.

What is not included

Runner does not handle actual reorgs and back-trackings. In order to keep PR managable to review, this is exlcluded, assuming that 2 currently included DAs (celestia and mock) have instant finality: sov-stf-runner should handle DA head change. #1217
RPC endpoint should point to head storage, instead of finalized storage #1218
- RPC endpoints are looking only on finalized state. Because of that extra method get_finalized_storage added to HierarchicalStorageManager, so it can give storage without attaching it to a block. max_out_version method added to state_db, so this "finalized" db can query latest version. All this should be removed in favour of updating RPC storage with each block progression
- LedgerDB works on top of rocksdb directly and not snapshots:
- BatchBuilder works on top of rocksdb directly

OtherChanges

Spec for DefaultContex have declared concrete SnapshotManager for Storage:
```
type Storage = ProverStorage<DefaultStorageSpec, sov_prover_storage_manager::SnapshotManager>;
```
- This prevents generic from bubbling up everywhere, but then require sov_prover_storage_manager to become dev-dependency almost everywhere where storage is needed. Even though, there's NoopQueryManager from sov-schema-db, it cannot be used outside, because of this spec declration. Another option would be to declare DefaultTestContext where NoopQueryManager would be used, which result in simpler testing and no need to creating temporary folders everywhere.
MockDa has hard coded genesis header at height 0.
MockBlockHeader now have from_height constructor which always gives coherent and deterministic headers. Convenient in tests
Method with_config from Storage trait is removed, as all storages are built externally
Stf::PreState and Stf::ChangeSet are the same type C::Storage in case of app template.
FrozenDbSnapshot is renamed to ReadOnlyDbSnapshot
demo-rollup has initialize_logging function, so it is easy to plug in test/bench to get same logging

Important traits

These 2 traits have been aded or updated and it is important to review naming/documentation:

/// A trait to make nested calls to several [`SchemaBatch`]s and eventually [`crate::DB`]
pub trait QueryManager {
    /// Iterator over key-value pairs in reverse lexicographic order in given [`Schema`]
    type Iter<'a, S: Schema>: Iterator<Item = (SchemaKey, SchemaValue)>
    where
        Self: 'a;
    /// Iterator with given range
    type RangeIter<'a, S: Schema>: Iterator<Item = (SchemaKey, SchemaValue)>
    where
        Self: 'a;
    /// Get a value from parents of given [`SnapshotId`]
    /// In case of unknown [`SnapshotId`] return `Ok(None)`
    fn get<S: Schema>(
        &self,
        snapshot_id: SnapshotId,
        key: &impl KeyCodec<S>,
    ) -> anyhow::Result<Option<S::Value>>;

    /// Returns an iterator over all key-value pairs in given [`Schema`] in reverse lexicographic order
    /// Starting from given [`SnapshotId`]
    fn iter<S: Schema>(&self, snapshot_id: SnapshotId) -> anyhow::Result<Self::Iter<'_, S>>;
    /// Returns an iterator over all key-value pairs in given [`Schema`] in reverse lexicographic order
    /// Starting from given [`SnapshotId`], where largest returned key will be less or equal to `upper_bound`
    fn iter_range<S: Schema>(
        &self,
        snapshot_id: SnapshotId,
        upper_bound: SchemaKey,
    ) -> anyhow::Result<Self::RangeIter<'_, S>>;
}

/// Storage manager, that supports tree-like hierarchy of snapshots
/// So different rollup state can be mapped to DA state 1 to 1, including chain forks.
pub trait HierarchicalStorageManager<Da: DaSpec> {
    /// Type that can be consumed by `[crate::state_machine::stf::StateTransitionFunction]` in native context.
    type NativeStorage;
    /// Type that is produced by `[crate::state_machine::stf::StateTransitionFunction]`.
    type NativeChangeSet;

    /// Creates storage based on given Da block header,
    /// meaning that at will have access to previous blocks state in same fork.
    fn create_storage_on(
        &mut self,
        block_header: &Da::BlockHeader,
    ) -> anyhow::Result<Self::NativeStorage>;

    /// Snapshots that points directly to finalized storage.
    /// Won't be saved if somehow 'saved'
    fn create_finalized_storage(&mut self) -> anyhow::Result<Self::NativeStorage>;

    /// Adds [`Self::NativeChangeSet`] to the storage.
    /// [`DaSpec::BlockHeader`] must be provided for efficient consistency checking.
    fn save_change_set(
        &mut self,
        block_header: &Da::BlockHeader,
        change_set: Self::NativeChangeSet,
    ) -> anyhow::Result<()>;

    /// Finalizes snapshot on given block header
    fn finalize(&mut self, block_header: &Da::BlockHeader) -> anyhow::Result<()>;

Linked Issues

Related to Data Availability re-orgs handling tracking issue #1023

Testing

All tests have been updated.

Docs

Documentation has been updated.

Benchmarking

Baseling Nightly

Run 1

+--------------------------+-------------------+
| Blocks                   | 100               |
+--------------------------+-------------------+
| Txns per Block           | 10000             |
+--------------------------+-------------------+
| Processed Txns (Success) | 1000000           |
+--------------------------+-------------------+
| Total                    | 138.068591118s    |
+--------------------------+-------------------+
| Apply Block              | 135.286691131s    |
+--------------------------+-------------------+
| Txns per sec (TPS)       | 7242.776882870865 |
+--------------------------+-------------------+

Run 2

+--------------------------+------------------+
| Blocks                   | 100              |
+--------------------------+------------------+
| Txns per Block           | 10000            |
+--------------------------+------------------+
| Processed Txns (Success) | 1000000          |
+--------------------------+------------------+
| Total                    | 142.025576523s   |
+--------------------------+------------------+
| Apply Block              | 139.150302183s   |
+--------------------------+------------------+
| Txns per sec (TPS)       | 7040.98532448525 |
+--------------------------+------------------+

This PR

Fork length 1 block

+----------------------------------+-------------------------+
| Blocks                           | 100                     |
+----------------------------------+-------------------------+
| Transactions per block           | 10000                   |
+----------------------------------+-------------------------+
| Processed transactions (success) | 1000000                 |
+----------------------------------+-------------------------+
| Total                            | 2m 17s 431ms 318us 43ns |
+----------------------------------+-------------------------+
| Apply block                      | 2m 9s 278ms 541us 253ns |
+----------------------------------+-------------------------+
| Transactions per sec (TPS)       | 7276.4                  |
+----------------------------------+-------------------------+

Fork length 3 blocks

+----------------------------------+--------------------------+
| Blocks                           | 100                      |
+----------------------------------+--------------------------+
| Transactions per block           | 10000                    |
+----------------------------------+--------------------------+
| Processed transactions (success) | 1000000                  |
+----------------------------------+--------------------------+
| Total                            | 2m 22s 944ms 176us 445ns |
+----------------------------------+--------------------------+
| Apply block                      | 2m 14s 493ms 87us 909ns  |
+----------------------------------+--------------------------+
| Transactions per sec (TPS)       | 6995.7                   |
+----------------------------------+--------------------------+

Fork lenght 50 blocks

+----------------------------------+--------------------------+
| Blocks                           | 100                      |
+----------------------------------+--------------------------+
| Transactions per block           | 10000                    |
+----------------------------------+--------------------------+
| Processed transactions (success) | 1000000                  |
+----------------------------------+--------------------------+
| Total                            | 2m 57s 980ms 435us 593ns |
+----------------------------------+--------------------------+
| Apply block                      | 2m 52s 11ms 782us 429ns  |
+----------------------------------+--------------------------+
| Transactions per sec (TPS)       | 5618.6                   |
+----------------------------------+--------------------------+

codecov · 2023-12-07T21:28:13Z

Codecov Report

Merging #1197 (847f43b) into nightly (d2b13ca) will increase coverage by 0.3%.
The diff coverage is 94.6%.

Additional details and impacted files

Files	Coverage Δ
adapters/mock-da/src/service.rs	`99.0% <100.0%> (-0.4%)`	⬇️
adapters/mock-da/src/types/mod.rs	`84.4% <100.0%> (+0.7%)`	⬆️
full-node/db/sov-db/src/ledger_db/mod.rs	`90.8% <ø> (ø)`
full-node/db/sov-db/src/schema/tables.rs	`89.0% <100.0%> (ø)`
full-node/db/sov-schema-db/src/iterator.rs	`98.8% <100.0%> (+<0.1%)`	⬆️
full-node/db/sov-schema-db/src/lib.rs	`83.5% <ø> (ø)`
full-node/db/sov-schema-db/src/schema_batch.rs	`100.0% <100.0%> (ø)`
full-node/db/sov-schema-db/src/test.rs	`100.0% <100.0%> (ø)`
...sov-prover-storage-manager/src/snapshot_manager.rs	`99.8% <100.0%> (+0.6%)`	⬆️
full-node/sov-sequencer/src/batch_builder.rs	`97.1% <100.0%> (+<0.1%)`	⬆️
... and 28 more

... and 3 files with indirect coverage changes

adapters/mock-da/src/service.rs

full-node/sov-prover-storage-manager/src/snapshot_manager.rs

full-node/sov-stf-runner/src/runner.rs

full-node/sov-prover-storage-manager/src/lib.rs

full-node/db/sov-schema-db/src/iterator.rs

* Concrete Snapshot Manager in default context * Updating sov-db to point to snapshots * Initialize ProverStorage inside NewProverStorageManager * Remove old storage manager * Update MockDa to have block zero finalized in the beginning * Adding more logging to sov-prover-storage-manager * Rename `FrozenDbSnapshot` to `ReadOnlyDbSnapshot` * Remove arbitrary from sov-db * Fixes in benchmarks * Add range iterator to `SchemaBatch`

citizen-stig self-assigned this Dec 1, 2023

citizen-stig force-pushed the nikolai/fork_manager_integration_stage_1 branch from 546765a to 5eec43d Compare December 6, 2023 12:26

citizen-stig changed the title ~~WIP: Integrating Fork Management~~ Integrating Nested Prover Storage Management Dec 8, 2023

citizen-stig marked this pull request as ready for review December 8, 2023 13:18

citizen-stig requested review from preston-evans98 and bkolad December 8, 2023 13:18

citizen-stig mentioned this pull request Dec 8, 2023

arbitrary for sov-db #1206

Open