Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tool script to clean old RocksDB log files from disk #3071

Merged
merged 1 commit into from
Nov 22, 2023
Merged

Add tool script to clean old RocksDB log files from disk #3071

merged 1 commit into from
Nov 22, 2023

Conversation

timvisee
Copy link
Member

This script wipes old and obsolete RocksDB log files from disk. This is very useful to clean up disk space in case of a Qdrant crash loop, in which case tens of thousands of these may be created. We've used it in cloud before. Placing it here just makes it easily accessible.

More specifically, this keeps the 2 most recent old and obsolete log files and deletes other ones. The number of files to keep can be tweaked.

Usage, dry run for testing its effects:

$ cd ./storage
$ ../tools/clean-old-rocksdb-logs.sh

This script removes old and obsolete log files to clean up disk space.
It is potentionally dangerous. Always make sure you have a backup before running this.
Qdrant must be stopped before running this script.
Dry run? [y/n]: y
Cleaning collection: collections/benchmark
Cleaning collection: collections/test1
+ rm -f collections/test1/0/segments/a/LOG.old.170000001 collections/test1/0/segments/a/LOG.old.170000002 collections/test1/0/segments/a/LOG.old.170000003 collections/test1/0/segments/a/LOG.old.170000004 collections/test1/0/segments/a/LOG.old.170000005 collections/test1/0/segments/a/LOG.old.170000006
+ rm -f collections/test1/0/segments/b/LOG.old.170000001 collections/test1/0/segments/b/LOG.old.170000002 collections/test1/0/segments/b/LOG.old.170000003 collections/test1/0/segments/b/LOG.old.170000004 collections/test1/0/segments/b/LOG.old.170000005 collections/test1/0/segments/b/LOG.old.170000006
# -- snip --
Cleaned 180 old log files

Usage, real run for testing its effects:

$ cd ./storage
$ ../tools/clean-old-rocksdb-logs.sh

This script removes old and obsolete log files to clean up disk space.
It is potentionally dangerous. Always make sure you have a backup before running this.
Qdrant must be stopped before running this script.
Dry run? [y/n]: n
Are you sure you want to continue? [y/n]: y
Cleaning collection: collections/benchmark
Cleaning collection: collections/test1
+ rm -f collections/test1/0/segments/a/LOG.old.170000001 collections/test1/0/segments/a/LOG.old.170000002 collections/test1/0/segments/a/LOG.old.170000003 collections/test1/0/segments/a/LOG.old.170000004 collections/test1/0/segments/a/LOG.old.170000005 collections/test1/0/segments/a/LOG.old.170000006
+ rm -f collections/test1/0/segments/b/LOG.old.170000001 collections/test1/0/segments/b/LOG.old.170000002 collections/test1/0/segments/b/LOG.old.170000003 collections/test1/0/segments/b/LOG.old.170000004 collections/test1/0/segments/b/LOG.old.170000005 collections/test1/0/segments/b/LOG.old.170000006
# -- snip --
Cleaned 180 old log files

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

@timvisee timvisee merged commit d321b64 into dev Nov 22, 2023
17 checks passed
@timvisee timvisee deleted the tool-clean-old-rocksdb-logs branch November 22, 2023 08:58
generall pushed a commit that referenced this pull request Dec 6, 2023
@timvisee @generall
Add tool script to clean old RocksDB log files from disk (#3071)
0da9723
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@agourlay agourlay agourlay approved these changes

@elbart elbart Awaiting requested review from elbart

@generall generall Awaiting requested review from generall

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@timvisee @agourlay