Byte storage integration into segment #4049
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
IvanPleshkov
force-pushed
the
byte-storage-integration
branch
from
55e2deb to
da88a49
Compare
raw scorer integration config and test are you happy fmt fn renamings cow refactor use quantization branch quantization update
IvanPleshkov
force-pushed
the
byte-storage-integration
branch
from
c8a8468 to
686cf90
Compare
generall
reviewed
Apr 17, 2024
| Self { | ||
| query: TMetric::preprocess(query), | ||
| query: TElement::slice_from_float_cow(&preprocessed_vector).to_vec(), |
There was a problem hiding this comment.
It is good to know, that Cow won't actually invoke copy for owned vectors
There was a problem hiding this comment.
Here I did like that because slice_from_float_cow is only one function to convert from floats into bytes. I find Cow helpful in storages upserting
There was a problem hiding this comment.
Fixed: 0edd630
Now we don't do unnecessary copying. In this place for float and non-cosine distance there is no any reallocation anymore
generall
approved these changes
Apr 17, 2024
timvisee
pushed a commit
that referenced
this pull request
Apr 22, 2024
* byte storage with quantization raw scorer integration config and test are you happy fmt fn renamings cow refactor use quantization branch quantization update * are you happy clippy * don't use distance in quantized scorers * fix build * add fn quantization_preprocess * apply preprocessing for only cosine float metric * fix sparse vectors tests * update openapi * more complicated integration test * update openapi comment * mmap byte storages support * fix async test * move .unwrap closer to the actual check of the vector presence * fmt * remove distance similarity function * avoid copying data while working with cow --------- Co-authored-by: generall <andrey@vasnetsov.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds byte vector storage support into segment.
Main changes in this PR to do integration:
datatypefield into segment config (intoVectorDataConfig). It allows storing the type of storage and loading segment properlyVectorStorageEnum:DenseSimpleByte,DenseMemmapByteandDenseAppendableMemmapByte.Distance::preprocess_vectorandDistance::similarity.Distancedoes not describe which type of vector is processing so it cannot trigger functions from the genericMetrictrait.Distance::preprocess_vectortriggers large refactor in quantization scorers.All Submissions:
devbranch. Did you create your branch fromdev?New Feature Submissions:
cargo +nightly fmt --allcommand prior to submission?cargo clippy --all --all-featurescommand?Changes to Core Features: