Compatibility for mmap sparse vectors #5454
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
coszio
commented
Nov 15, 2024
| }); | ||
| } | ||
|
|
||
| #[test] |
There was a problem hiding this comment.
Also tests the simple storage for better coverage 🤓
coszio
commented
Nov 15, 2024
| pub enum SparseVectorStorageType { | ||
| /// Storage on disk | ||
| // (rocksdb storage) | ||
| #[default] |
There was a problem hiding this comment.
changing this default in the future will help us make the switch
coszio
force-pushed
the
blob_store-for-sparse-vectors
branch
from
6b4cb58 to
da71d4f
Compare
timvisee
reviewed
Nov 22, 2024
coszio
force-pushed
the
blob_store-for-sparse-vectors
branch
from
e5671d7 to
d4d9f15
Compare
timvisee
approved these changes
Nov 27, 2024
coszio
commented
Nov 27, 2024
| @@ -79,7 +79,7 @@ impl SegmentBuilder { | |||
| }; | |||
|
|
|||
| let payload_storage = | |||
| create_payload_storage(database.clone(), segment_config, segment_path)?; | |||
| create_payload_storage(database.clone(), segment_config, temp_dir.path())?; | |||
There was a problem hiding this comment.
This also fixes the mmap payload storage to be stored in the right place
generall
approved these changes
Nov 27, 2024
Plus some extra refactors
coszio
force-pushed
the
blob_store-for-sparse-vectors
branch
from
d4d9f15 to
4835944
Compare
coszio
added a commit
that referenced
this pull request
Dec 2, 2024
* implement mmap sparse vector storage * add to VectorStorageEnum * clippy * add tests, fix both simple and mmap storages * smol correction on total_vector_count * add sparse storage type to config * fix reading config without storage type * generate openapi * use blob_store by path * hidden setting to enable new storage * validate existing path in `BlobStore::open()` * use new dir for each sparse vector name * fix and rename `max_point_offset` Plus some extra refactors * add storage compat test, to always check both storages work * fix opening of storage + other misc fixes * FIX!!! `Unset` operations in the Tracker weren't updating the `next_pointer_id`. So, when reopening the storage, those points wouldn't get marked as deleted in the bitslice, thus creating the illusion that they should exist, when they did not. * refactor naming from `iter_*` to `for_each_*` * fix checking for BlobStore existance * fix typo * fix error message * better docs for open_or_create * fix after rebase
timvisee
pushed a commit
that referenced
this pull request
Dec 9, 2024
* implement mmap sparse vector storage * add to VectorStorageEnum * clippy * add tests, fix both simple and mmap storages * smol correction on total_vector_count * add sparse storage type to config * fix reading config without storage type * generate openapi * use blob_store by path * hidden setting to enable new storage * validate existing path in `BlobStore::open()` * use new dir for each sparse vector name * fix and rename `max_point_offset` Plus some extra refactors * add storage compat test, to always check both storages work * fix opening of storage + other misc fixes * FIX!!! `Unset` operations in the Tracker weren't updating the `next_pointer_id`. So, when reopening the storage, those points wouldn't get marked as deleted in the bitslice, thus creating the illusion that they should exist, when they did not. * refactor naming from `iter_*` to `for_each_*` * fix checking for BlobStore existance * fix typo * fix error message * better docs for open_or_create * fix after rebase
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds the infrastructure to integrate
blob_storeas a new mmap storage for sparse vectors.This PR does not yet create segments with this storage, but should be able to read it.
There are some additional changes to the
simplestorage, specifically:total_sparse_vector_size:Nonewhen reading the storageEdits for
blob_store:Nonevalue (deleting a point above the range of the storage) now updates thenext_pointer_offsetfor_each_unfilteredfunction which iterates through the ALL the storage with a callback, including deleted pointsopen_or_createfunction which does a better check to choose between opening or creating a storageTODO