-
Notifications
You must be signed in to change notification settings - Fork 490
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Algod: support state proofs recoverability #4803
Merged
Merged
Changes from 16 commits
Commits
Show all changes
125 commits
Select commit
Hold shift + click to select a range
598705d
Persist StateProof builders on disk (#4553)
Aharonee fae555c
State Proof Recoverability: state proof verification tracking (#4570)
almog-t 9ebdbce
fix tests: use better function for waiting
7f11ab3
Algod: state-proof key deletion safety (#4601)
algonathan 3bac343
merged with avm box changes to master
e734ed7
fixed tracker db version
dc631d2
fixed lint error
194b7ac
updated msgp_gen
7d55071
built with latest dev tools
13c0fb4
fixed comment typo
311bc30
added missing dot
e569879
Merge pull request #4735 from almog-t/merge-recoverability-master
id-ms 0bd34b8
StateProofs: Use stateproof tracker (#4733)
id-ms 2005611
CR: shorter names for verification data in dcc
8845eb4
CR: renamed non db insert to append
a4dae19
renamed mutex
ec8dda1
CR: now using db queries pointer
788b50f
CR: using nested struct in tracker.go
02edfdf
renamed TargetRound to LastAttestedRound
f9cc17f
fixed potential segfault
4e01f81
renamed stateProofVerificationData to SPVerificationContext
2a6bc73
renamed internal stateproofverification struct to spVerification
30dd781
another rename for stateproofverificationcontext
e70d7d8
refactored deletion data usage
f000773
full renamed of stateproofverificationdata to stateproofverificationc…
eefc0f9
ran msgp again
f4456ef
stateproofround -> lastattestedround
c9dd1af
more stateproofround -> lastattestedroun
478d94c
CR: various renamings
687258f
StateProofs: Sp verification-tracker last lookup cache (#4780)
algonathan af27896
merged and renamed relevant variables in feature branch
f6959af
CR: shortened dcc variables
e0432f4
Merge pull request #4769 from almog-t/spv-tracking-cr-fixes
almog-t 30de42d
Merge branch 'master' into merge-sp-recovery-with-master
056719b
Merge pull request #4793 from algoidan/merge-sp-recovery-with-master
id-ms 8180823
StateProofs: Limit builders memory usage and reduce StateProofs signa…
Aharonee 7409b73
Merge branch 'master' into merge-sp-recovery-with-master
7a2c94d
Merge pull request #4801 from algoidan/merge-sp-recovery-with-master
id-ms e6ed9eb
make state proof unit-tests synced
6dd8f40
remove printf
a85fcda
fix state proof message tests
75f6ac5
test cache limit on sig handler.
3a6982e
remove panic from test
1722455
fix race problem
d65a96b
refactor AllSigs test
50f009f
reduce log spamming
c9bd815
add signature broadcast test
9c7bf73
Revert "reduce log spamming"
1b35e37
handling mem issue
ae2b236
fix Broadcast test
4726d1d
reduce memory consumption in tests
d337e8a
CR fixed
7b9b68e
test builder cache context
7c8bce1
Merge pull request #4817 from algoidan/sp-recovery-test-refactor
id-ms d661705
Stateproof Recoverability: verification tracking catchpoint support (…
almog-t 88d857e
fix merge issues
eab1fff
moving stateproofverification tracker's sql function into a file.
1e4544d
some refactoring
2ba5c97
refactoring
43a72f0
Merge pull request #4863 from algoidan/merge-sp-recoverability-from-m…
id-ms e5d48e4
merge with master
ebc6ef2
Merge pull request #4888 from algoidan/merge-sp-recoverability-from-m…
id-ms 3c004a5
Fix catchup stateproof race (#4887)
almog-t 0f58463
merge changes from master
b1905b4
Merge pull request #4950 from algoidan/merge-sp-recvoery-with-master
id-ms fe354d8
Apply StateProof transactions on catchup (#4949)
Aharonee 8278b53
StateProofs: Fix CR comments (#4958)
id-ms 42caa30
merge with master
b3fc3f5
Merge pull request #4959 from algoidan/merge-sp-recoverability-with-m…
id-ms c1ad50c
Fix flaky unit test
Aharonee 40f1168
Merge branch 'master' into merge-sp-recoverbility-with-master
9f18d39
Merge pull request #5020 from algoidan/merge-sp-recoverbility-with-ma…
id-ms 7e6d874
Add recoverability catchpoint e2e (#4961)
almog-t 98d5a4f
Refactor insertSigToBuilder and handleSig, avoid unused allocations (…
Aharonee cd7c667
StateProof Recoverability: Fix CR comments (#5030)
id-ms 816449f
change sp interval in short tests (#5081)
id-ms ede87b2
fix broken nightly tests
96d9c3b
merge with master
57ba79b
refactor sp tracker to support new store
2272129
use prepare statement
5b899b6
Merge pull request #5086 from algoidan/merge-sp-recvoerability-with-m…
id-ms 4be1ecb
changed RLock to Lock to properly protect lastFlushTime writes (#5092)
almog-t 1c25270
ledger: eternalData trackers concept
algorandskiy 0ad0943
stateproof: e2e flakey-test AttestorsChange (#5101)
algonathan 827365e
StateProof recoverability CR fixes (#5085)
id-ms 4a2f579
CR fixes
algorandskiy 295177c
Merge remote-tracking branch 'upstream/feature/stateproofs-recoverabi…
algorandskiy 493a5f8
Merge pull request #5099 from algorandskiy/pavel/sp-recov
almog-t 62b0c02
Revert "ledger: eternalData trackers concept" (#5104)
algorandskiy 618736a
Fix metrics test
algorandskiy c90be81
Merge pull request #5106 from algorandskiy/pavel/sp-recov-test
almog-t 8f8f6f1
Remove unused function to fetch message from db and small refactor
Aharonee d8ccde8
Fix errors tests log (#5117)
almog-t e331404
Inline sigExists and builderExists methods
Aharonee 760c231
Stop stateproof worker on catchpoint cathcup and restart when done. (…
Aharonee f97766a
Merge branch 'master' into spr-merge-with-master
28c1313
Merge pull request #5144 from algoidan/spr-merge-with-master
id-ms 4b8854b
StateProofRecoverability: fix comments (#5143)
id-ms f5e16e8
Merge remote-tracking branch 'upstream/master' into feature/stateproo…
algorandskiy 6e2f233
post merge fixes:
algorandskiy 9efe32c
fix lint
algorandskiy 22b3125
Fix exceed partitiontest call
algorandskiy ecbcc8e
fix flaky test
ab7a68b
code review: fix interface names as suggested
algorandskiy 8201750
Merge pull master (#5148)
algorandskiy da1d70f
fix linter in ledger_test.go
algorandskiy f19a285
StateProof Recvoerability: CR fixes (#5153)
id-ms 52c8bcb
Merge remote-tracking branch 'upstream/master' into feature/stateproo…
algorandskiy d229775
merge master into feature/stateproofs-recoverability
algorandskiy 0d0697a
Do not stop stateproof service in fast catchup mode (#5170)
algorandskiy 0bc8014
Merge branch 'master' into merge-spr-with-master
7a2714f
Merge pull request #5179 from algoidan/merge-spr-with-master
id-ms 3d303b0
fix flaky test
3c831df
CR fixes (#5176)
id-ms 8d31459
Merge remote-tracking branch 'upstream/master' into pavel/stateproofs…
algorandskiy 56cd05b
Merge remote-tracking branch 'upstream/master' into pavel/stateproofs…
algorandskiy 389b8ac
Linter fixes
algorandskiy 4f12cb6
Merge master (#5199)
algorandskiy c659a1d
Update log message
Aharonee 6e4b414
Add a second mutex to stateproof.Worker to allow more granular locking.
Aharonee 150efa2
convert to rw lock and fix linter error.
30da8b6
update comments.
8bf459b
using verification ctx from tracker when creating stats
fdbe4e5
Merge branch 'master' into merge-sp-recoverability-from-master
cd7dbcc
Merge pull request #5241 from algoidan/merge-sp-recoverability-from-m…
id-ms File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here,
latestStateProofRound
can be anywhere less thanthreshold
or much greater thanthreshold
?In case
latestStateProofRound
is much greater thanthreshold
,rnds
will have a gap?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes
The slice will have gaps and that is exactly the optimization we aimed for. If state proofs are being lagged for several intervals we don't want to keep all the potential builders in memory (since there might be a lot). so keep only the earliest
buildersCacheLength
builders + the last one.see if this comment makes sense
go-algorand/stateproof/builder.go
Line 578 in 760c231