[indexer] table info service backup support #14373

larry-aptos · 2024-08-21T22:07:41Z

Description

In this PR, we introduce the backup of db index_indexer_async_v2_db, i.e., async table info db.

Design

There are two major loops:

async indexing loop: fetch transactions, index table info. If the backup is enabled, snapshot at the end of epoch.
backup loop: if backup is enabled, backup the snapshot to GCS.

Example Config

indexer_table_info:
    parser_task_count: 10
    parser_batch_size: 100
    table_info_service_mode:
        Backup:
            example-gcs-bucket-name

Breaking Change

This change is a breaking change around the configuration. It's not released yet and the only impact is our internal deployment. We need to update the configuration for the next deployment.

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

tested locally, looks good

GCS:

Key Areas to Review

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-08-21T22:07:45Z

⏱️ 2h 23m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
general-lints	19m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-cargo-deny	19m	🟩 🟩 🟩 🟩 🟩 (+6 more)
check-dynamic-deps	11m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	6m	⬜
rust-move-tests	5m	⬜
semgrep/ci	5m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-move-tests	3m	⬜
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	32s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	31s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	29s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	28s	🟩 🟩 🟩 🟩 🟩 (+6 more)

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

ecosystem/indexer-grpc/indexer-grpc-table-info/src/runtime.rs

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

config/src/config/indexer_table_info_config.rs

ecosystem/indexer-grpc/indexer-grpc-table-info/src/runtime.rs

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

config/src/config/indexer_table_info_config.rs

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

larry-aptos · 2024-08-28T18:21:48Z

i am working on:

split the snapshot and the backup logic apart.
backup-related stuff should be optional for readability.

grao1991 · 2024-08-28T23:31:58Z

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

+                let backup_restore_operator = backup_restore_operator.clone();
+                let context = self.context.clone();
+                let _task = tokio::spawn(async move {
+                    let backup_restore_operator = backup_restore_operator.clone();


you don't need to clone so many times.

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

grao1991 · 2024-08-29T23:28:58Z

config/src/config/indexer_table_info_config.rs

+    /// Backup service mode with GCS bucket name.
+    Backup(String),
+    /// Restore service mode with GCS bucket name.
+    Restore(String),


how do you plan to use Restore mode?

That's a good question.

Restore mode is for people to use when bootstrap a new node. It can help when the node lags/has issues and the rest of the world has already pruned the data.

IndexingOnly - no backup or restore, async db only.

ecosystem/indexer-grpc/indexer-grpc-table-info/README.md

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

larry-aptos · 2024-08-30T22:41:05Z

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

+            let file_name = path.file_name().unwrap().to_string_lossy();
+            if path.is_dir()
+                && file_name.starts_with(&target_snapshot_directory_prefix)
+                && !file_name.ends_with(".tmp")


is it safe to assume that snapshot is done if tmp doesn't show up in the filename? cc @grao1991

I don't know? @msmouse any idea?

What? You mean rocksdb adds a ".tmp" suffix while taking a checkpoint? That's great design (moving / renaming folder should be assumed atomic)

if rocksdb didn't do it, we should do it manually in your use case.

Yea then i'll use the tmp for now to determine the snapshot is done.

grao1991 · 2024-09-03T19:21:15Z

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

+            let file_name = path.file_name().unwrap().to_string_lossy();
+            if path.is_dir()
+                && file_name.starts_with(&target_snapshot_directory_prefix)
+                && !file_name.ends_with(".tmp")


I don't know? @msmouse any idea?

grao1991 · 2024-09-03T19:24:18Z

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs

+        }
+    }
+
+    backup_restore_operator


I'm not sure if you need to do something here to handle the case when multiple jobs uploading the backup at the same time. Can you double check?

Do you mean multiple backup services are running? No; my thought is to check it on the deployment side. If we do want to check within the program(runtime?), we can either use some distributed locking or some coarse method like, checking metadata versioning. wdyt?

I feel we want to have multiple jobs to be able to backup things to ensure a fair reliability. I don't have preference on how they corporate. Either doing it on client side or on server side are ok.
We can leave it as a TODO for future PR.

(I'd just add a random suffix to the file name so we never write to the same remote file and we can observe the conflicting behavior better)

Thanks! I added a to-do as follow-up to check if concurrent jobs are running. One of the concurrent scenarios for now is multiple full nodes running with backup enabled.

github-actions · 2024-09-05T21:36:52Z

✅ Forge suite `realistic_env_max_load` success on `0f8bfabef29261ba93dcd265aa39426297c771ae`

two traffics test: inner traffic : committed: 12250.94 txn/s, latency: 3242.28 ms, (p50: 3000 ms, p70: 3300, p90: 3600 ms, p99: 7500 ms), latency samples: 4658120
two traffics test : committed: 99.98 txn/s, latency: 2658.72 ms, (p50: 2500 ms, p70: 2600, p90: 3000 ms, p99: 10600 ms), latency samples: 1700
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.245, avg: 0.221", "QsPosToProposal: max: 0.623, avg: 0.471", "ConsensusProposalToOrdered: max: 0.329, avg: 0.320", "ConsensusOrderedToCommit: max: 0.533, avg: 0.490", "ConsensusProposalToCommit: max: 0.856, avg: 0.810"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.30s no progress at version 1622193 (avg 0.23s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.55s no progress at version 1622191 (avg 7.54s) [limit 15].
Test Ok

github-actions · 2024-09-05T21:37:29Z

✅ Forge suite `framework_upgrade` success on `d1bf834728a0cf166d993f4728dfca54f3086fb0` ==> `0f8bfabef29261ba93dcd265aa39426297c771ae`

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 0f8bfabef29261ba93dcd265aa39426297c771ae (PR)
Upgrade the nodes to version: 0f8bfabef29261ba93dcd265aa39426297c771ae
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1241.04 txn/s, submitted: 1243.59 txn/s, failed submission: 2.55 txn/s, expired: 2.55 txn/s, latency: 2790.75 ms, (p50: 2300 ms, p70: 2800, p90: 5200 ms, p99: 6800 ms), latency samples: 107080
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1071.79 txn/s, submitted: 1074.32 txn/s, failed submission: 2.53 txn/s, expired: 2.53 txn/s, latency: 2971.99 ms, (p50: 2400 ms, p70: 3000, p90: 5600 ms, p99: 7400 ms), latency samples: 93300
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 0f8bfabef29261ba93dcd265aa39426297c771ae passed
Upgrade the remaining nodes to version: 0f8bfabef29261ba93dcd265aa39426297c771ae
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1216.82 txn/s, submitted: 1220.18 txn/s, failed submission: 3.36 txn/s, expired: 3.36 txn/s, latency: 2657.92 ms, (p50: 2400 ms, p70: 2700, p90: 4200 ms, p99: 7200 ms), latency samples: 101380
Test Ok

github-actions · 2024-09-05T21:38:09Z

✅ Forge suite `compat` success on `d1bf834728a0cf166d993f4728dfca54f3086fb0` ==> `0f8bfabef29261ba93dcd265aa39426297c771ae`

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 0f8bfabef29261ba93dcd265aa39426297c771ae (PR)
1. Check liveness of validators at old version: d1bf834728a0cf166d993f4728dfca54f3086fb0
compatibility::simple-validator-upgrade::liveness-check : committed: 9892.99 txn/s, latency: 3232.83 ms, (p50: 2300 ms, p70: 2500, p90: 3300 ms, p99: 29300 ms), latency samples: 399720
2. Upgrading first Validator to new version: 0f8bfabef29261ba93dcd265aa39426297c771ae
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6850.21 txn/s, latency: 4137.06 ms, (p50: 4700 ms, p70: 5000, p90: 5100 ms, p99: 5300 ms), latency samples: 127440
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6581.09 txn/s, latency: 4664.68 ms, (p50: 4700 ms, p70: 4900, p90: 5200 ms, p99: 7200 ms), latency samples: 246800
3. Upgrading rest of first batch to new version: 0f8bfabef29261ba93dcd265aa39426297c771ae
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7164.07 txn/s, latency: 3960.53 ms, (p50: 4400 ms, p70: 4600, p90: 4700 ms, p99: 4900 ms), latency samples: 138820
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6216.83 txn/s, latency: 4778.85 ms, (p50: 4800 ms, p70: 5100, p90: 5400 ms, p99: 6900 ms), latency samples: 236720
4. upgrading second batch to new version: 0f8bfabef29261ba93dcd265aa39426297c771ae
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 9123.52 txn/s, latency: 2972.32 ms, (p50: 2900 ms, p70: 3200, p90: 4600 ms, p99: 5800 ms), latency samples: 166120
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 9522.18 txn/s, latency: 3359.11 ms, (p50: 2800 ms, p70: 3100, p90: 7500 ms, p99: 9100 ms), latency samples: 317480
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 0f8bfabef29261ba93dcd265aa39426297c771ae passed
Test Ok

larry-aptos force-pushed the larry-table-info-service branch from 8daf2f6 to d663816 Compare August 21, 2024 22:11

larry-aptos commented Aug 23, 2024

View reviewed changes

ecosystem/indexer-grpc/indexer-grpc-table-info/src/runtime.rs Outdated Show resolved Hide resolved

larry-aptos commented Aug 23, 2024

View reviewed changes

ecosystem/indexer-grpc/indexer-grpc-table-info/src/runtime.rs Outdated Show resolved Hide resolved

larry-aptos commented Aug 23, 2024

View reviewed changes

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs Outdated Show resolved Hide resolved

larry-aptos commented Aug 23, 2024

View reviewed changes

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs Show resolved Hide resolved

larry-aptos commented Aug 23, 2024

View reviewed changes

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs Outdated Show resolved Hide resolved

larry-aptos commented Aug 23, 2024

View reviewed changes

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs Outdated Show resolved Hide resolved

larry-aptos commented Aug 23, 2024

View reviewed changes

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs Outdated Show resolved Hide resolved

larry-aptos changed the title ~~Larry table info service~~ [indexer] table info service backup support Aug 23, 2024

larry-aptos marked this pull request as ready for review August 23, 2024 23:24

larry-aptos requested review from JoshLind and gregnazario as code owners August 23, 2024 23:24

larry-aptos requested review from jillxuu, grao1991 and a team and removed request for gregnazario and JoshLind August 23, 2024 23:24

larry-aptos force-pushed the larry-table-info-service branch from ab39429 to 84f816c Compare August 24, 2024 00:04

grao1991 requested a review from msmouse August 24, 2024 00:10

grao1991 reviewed Aug 24, 2024

View reviewed changes

config/src/config/indexer_table_info_config.rs Outdated Show resolved Hide resolved

ecosystem/indexer-grpc/indexer-grpc-table-info/src/runtime.rs Outdated Show resolved Hide resolved

ecosystem/indexer-grpc/indexer-grpc-table-info/src/table_info_service.rs Outdated Show resolved Hide resolved

larry-aptos requested a review from grao1991 August 26, 2024 08:20

grao1991 reviewed Aug 26, 2024

View reviewed changes

larry-aptos requested a review from grao1991 August 28, 2024 22:53

larry-aptos force-pushed the larry-table-info-service branch 2 times, most recently from f2aa2fc to 5823faa Compare August 28, 2024 23:02

add backup support.

ecf8017

larry-aptos force-pushed the larry-table-info-service branch from 5823faa to ecf8017 Compare August 28, 2024 23:51

grao1991 reviewed Aug 29, 2024

View reviewed changes

address comment.

77bc976

larry-aptos requested a review from grao1991 August 30, 2024 04:07

msmouse reviewed Aug 30, 2024

View reviewed changes

larry-aptos added 2 commits August 30, 2024 13:34

fix.

9e3be5e

remove the CI.

64b1b1d

larry-aptos commented Aug 30, 2024

View reviewed changes

fix the lint.

1968971

grao1991 reviewed Sep 3, 2024

View reviewed changes

add todo.

ffe6c82

larry-aptos requested review from msmouse and grao1991 September 4, 2024 20:04

grao1991 approved these changes Sep 5, 2024

View reviewed changes

jillxuu approved these changes Sep 5, 2024

View reviewed changes

larry-aptos enabled auto-merge (squash) September 5, 2024 18:59

This comment has been minimized.

Sign in to view

Merge branch 'main' into larry-table-info-service

0f8bfab

This comment has been minimized.

Sign in to view

larry-aptos merged commit 78c815b into main Sep 5, 2024
46 checks passed

larry-aptos deleted the larry-table-info-service branch September 5, 2024 21:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[indexer] table info service backup support #14373

[indexer] table info service backup support #14373

larry-aptos commented Aug 21, 2024 •

edited

Loading

trunk-io bot commented Aug 21, 2024 •

edited

Loading

larry-aptos commented Aug 28, 2024 •

edited

Loading

grao1991 Aug 28, 2024

larry-aptos Aug 30, 2024

grao1991 Aug 29, 2024

larry-aptos Aug 30, 2024 •

edited

Loading

larry-aptos Aug 30, 2024

grao1991 Sep 3, 2024

msmouse Sep 3, 2024

larry-aptos Sep 4, 2024

grao1991 Sep 3, 2024

grao1991 Sep 3, 2024

larry-aptos Sep 3, 2024

grao1991 Sep 3, 2024

msmouse Sep 3, 2024

larry-aptos Sep 4, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 5, 2024

github-actions bot commented Sep 5, 2024

github-actions bot commented Sep 5, 2024

[indexer] table info service backup support #14373

[indexer] table info service backup support #14373

Conversation

larry-aptos commented Aug 21, 2024 • edited Loading

Description

Design

Example Config

Breaking Change

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Key Areas to Review

Checklist

trunk-io bot commented Aug 21, 2024 • edited Loading

larry-aptos commented Aug 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larry-aptos Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 5, 2024

✅ Forge suite realistic_env_max_load success on 0f8bfabef29261ba93dcd265aa39426297c771ae

github-actions bot commented Sep 5, 2024

✅ Forge suite framework_upgrade success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 0f8bfabef29261ba93dcd265aa39426297c771ae

github-actions bot commented Sep 5, 2024

✅ Forge suite compat success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 0f8bfabef29261ba93dcd265aa39426297c771ae

larry-aptos commented Aug 21, 2024 •

edited

Loading

trunk-io bot commented Aug 21, 2024 •

edited

Loading

larry-aptos commented Aug 28, 2024 •

edited

Loading

larry-aptos Aug 30, 2024 •

edited

Loading

✅ Forge suite `realistic_env_max_load` success on `0f8bfabef29261ba93dcd265aa39426297c771ae`

✅ Forge suite `framework_upgrade` success on `d1bf834728a0cf166d993f4728dfca54f3086fb0` ==> `0f8bfabef29261ba93dcd265aa39426297c771ae`

✅ Forge suite `compat` success on `d1bf834728a0cf166d993f4728dfca54f3086fb0` ==> `0f8bfabef29261ba93dcd265aa39426297c771ae`