Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - fix: recover from invalid segments #2909

Closed
wants to merge 15 commits into from

Conversation

sehz
Copy link
Contributor

@sehz sehz commented Dec 29, 2022

resolves #2509.

Recover from invalid logs due to incorrect batch encoding. It does this by setting the log size to the last correct batches.
Tested with data from corrupted segments.

  • More clean and simplification of the Log validator.
  • Migrated more API to use anyhow

@sehz
Copy link
Contributor Author

sehz commented Dec 30, 2022

bors r+

bors bot pushed a commit that referenced this pull request Dec 30, 2022
resolves #2509.

Recover from invalid logs due to incorrect batch encoding.  It does this by setting the log size to the last correct batches.
Tested with data from corrupted segments.   

- More clean and simplification of the Log validator.
- Migrated more API to use anyhow
@bors
Copy link

bors bot commented Dec 30, 2022

Canceled.

@sehz
Copy link
Contributor Author

sehz commented Dec 30, 2022

bors r-

@sehz
Copy link
Contributor Author

sehz commented Dec 30, 2022

bors r+

bors bot pushed a commit that referenced this pull request Dec 30, 2022
resolves #2509.

Recover from invalid logs due to incorrect batch encoding.  It does this by setting the log size to the last correct batches.
Tested with data from corrupted segments.   

- More clean and simplification of the Log validator.
- Migrated more API to use anyhow
@bors
Copy link

bors bot commented Dec 30, 2022

Build failed:

@sehz
Copy link
Contributor Author

sehz commented Dec 30, 2022

bors r+

bors bot pushed a commit that referenced this pull request Dec 30, 2022
resolves #2509.

Recover from invalid logs due to incorrect batch encoding.  It does this by setting the log size to the last correct batches.
Tested with data from corrupted segments.   

- More clean and simplification of the Log validator.
- Migrated more API to use anyhow
@bors
Copy link

bors bot commented Dec 30, 2022

Pull request successfully merged into master.

Build succeeded:

@bors bors bot changed the title fix: recover from invalid segments [Merged by Bors] - fix: recover from invalid segments Dec 30, 2022
@bors bors bot closed this Dec 30, 2022
@sehz sehz deleted the 2509/better_log branch March 11, 2023 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: SPU fails with "not enough for records"
1 participant