Skip to content

Commit

Permalink
Add read_consistency parameter to the APIs (#1371) (#1407)
Browse files Browse the repository at this point in the history 
* WIP: Add `read_consistency` parameter to the APIs

* WIP: Add `read_consistency` parameter to the APIs

TODO:
- Add documentation

* `cargo fmt`

* Add gRPC documentation

* Add OpenAPI documentation

* Cleanup

* fixup! Add OpenAPI documentation

* fixup! Add gRPC documentation

Who would have known there's `generate_grpc_docs.sh`!? 🥲🙈🤦‍♀️

* generate openapi

* Fix `read_consistency` query parameter deserialization

* Further improve `read_consistency` query parameter deserialization

* `cargo clippy`

* Fix `Payload` comparison during read operation result resolving

* Fix grammar

* rename `read_consistency` -> `consistency` and add integration test

* use majority for test

* fix tests

* Fix tests

* fixup! Fix tests

Apply the same fix to `ScoredPoint`

* Remove an `unwrap`

* fixup! Fix tests

Gotta love those negative conditions, or how a missed `!` can ruin your day... 🤦‍♀️

* Make internal API calls strictly "local-shard only"

* Implement a few basic traits for `ResolverRecord`

* fixup! Implement a few basic traits for `ResolverRecord`

* Revert "Make internal API calls strictly "local-shard only""

This reverts commit 25378e6.

* Fix `Record::payload` and `ScoredPoint::payload` serialization

* Revert "Fix `Record::payload` and `ScoredPoint::payload` serialization"

This reverts commit b566bea.

* Fix `Record::payload` and `ScoredPoint::payload` visibility

* fixup! Fix `Record::payload` and `ScoredPoint::payload` visibility

Remove `todo!()`

* refactoring

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
  • Loading branch information
@ffuugoo @generall
ffuugoo and generall committed Feb 6, 2023
1 parent 55593f3 commit 34bdbc2
Show file tree
Hide file tree
Showing 34 changed files with 1,009 additions and 171 deletions.
1 change: 1 addition & 0 deletions Cargo.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ web = ["actix-web"]
service_debug = ["parking_lot", "parking_lot/deadlock_detection"]

[dev-dependencies]
serde_urlencoded = "0.7"

tempfile = "3.3.0"
rusty-hook = "^0.11.2"

Expand Down
37 changes: 37 additions & 0 deletions docs/grpc/docs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@
- [PointsOperationResponse](#qdrant-PointsOperationResponse)
- [PointsSelector](#qdrant-PointsSelector)
- [Range](#qdrant-Range)
- [ReadConsistency](#qdrant-ReadConsistency)
- [RecommendBatchPoints](#qdrant-RecommendBatchPoints)
- [RecommendBatchResponse](#qdrant-RecommendBatchResponse)
- [RecommendPoints](#qdrant-RecommendPoints)
Expand Down Expand Up @@ -116,6 +117,7 @@
- [WriteOrdering](#qdrant-WriteOrdering)

- [FieldType](#qdrant-FieldType)
- [ReadConsistencyType](#qdrant-ReadConsistencyType)
- [UpdateStatus](#qdrant-UpdateStatus)
- [WriteOrderingType](#qdrant-WriteOrderingType)

Expand Down Expand Up @@ -1211,6 +1213,7 @@ The JSON representation for `Value` is JSON value.
| ids | [PointId](#qdrant-PointId) | repeated | List of points to retrieve |
| with_payload | [WithPayloadSelector](#qdrant-WithPayloadSelector) | | Options for specifying which payload to include or not |
| with_vectors | [WithVectorsSelector](#qdrant-WithVectorsSelector) | optional | Options for specifying which vectors to include into response |
| read_consistency | [ReadConsistency](#qdrant-ReadConsistency) | optional | Options for specifying read consistency guarantees |



Expand Down Expand Up @@ -1472,6 +1475,22 @@ The JSON representation for `Value` is JSON value.



<a name="qdrant-ReadConsistency"></a>

### ReadConsistency



| Field | Type | Label | Description |
| ----- | ---- | ----- | ----------- |
| type | [ReadConsistencyType](#qdrant-ReadConsistencyType) | | Common read consistency configurations |
| factor | [uint64](#uint64) | | Send request to a specified number of nodes, and return points which are present on all of them |






<a name="qdrant-RecommendBatchPoints"></a>

### RecommendBatchPoints
Expand All @@ -1482,6 +1501,7 @@ The JSON representation for `Value` is JSON value.
| ----- | ---- | ----- | ----------- |
| collection_name | [string](#string) | | Name of the collection |
| recommend_points | [RecommendPoints](#qdrant-RecommendPoints) | repeated | |
| read_consistency | [ReadConsistency](#qdrant-ReadConsistency) | optional | Options for specifying read consistency guarantees |



Expand Down Expand Up @@ -1524,6 +1544,7 @@ The JSON representation for `Value` is JSON value.
| using | [string](#string) | optional | Define which vector to use for recommendation, if not specified - default vector |
| with_vectors | [WithVectorsSelector](#qdrant-WithVectorsSelector) | optional | Options for specifying which vectors to include into response |
| lookup_from | [LookupLocation](#qdrant-LookupLocation) | optional | Name of the collection to use for points lookup, if not specified - use current collection |
| read_consistency | [ReadConsistency](#qdrant-ReadConsistency) | optional | Options for specifying read consistency guarantees |



Expand Down Expand Up @@ -1628,6 +1649,7 @@ The JSON representation for `Value` is JSON value.
| limit | [uint32](#uint32) | optional | Max number of result |
| with_payload | [WithPayloadSelector](#qdrant-WithPayloadSelector) | | Options for specifying which payload to include or not |
| with_vectors | [WithVectorsSelector](#qdrant-WithVectorsSelector) | optional | Options for specifying which vectors to include into response |
| read_consistency | [ReadConsistency](#qdrant-ReadConsistency) | optional | Options for specifying read consistency guarantees |



Expand Down Expand Up @@ -1661,6 +1683,7 @@ The JSON representation for `Value` is JSON value.
| ----- | ---- | ----- | ----------- |
| collection_name | [string](#string) | | Name of the collection |
| search_points | [SearchPoints](#qdrant-SearchPoints) | repeated | |
| read_consistency | [ReadConsistency](#qdrant-ReadConsistency) | optional | Options for specifying read consistency guarantees |



Expand Down Expand Up @@ -1717,6 +1740,7 @@ The JSON representation for `Value` is JSON value.
| offset | [uint64](#uint64) | optional | Offset of the result |
| vector_name | [string](#string) | optional | Which vector to use for search, if not specified - use default vector |
| with_vectors | [WithVectorsSelector](#qdrant-WithVectorsSelector) | optional | Options for specifying which vectors to include into response |
| read_consistency | [ReadConsistency](#qdrant-ReadConsistency) | optional | Options for specifying read consistency guarantees |



Expand Down Expand Up @@ -1938,6 +1962,19 @@ The JSON representation for `Value` is JSON value.



<a name="qdrant-ReadConsistencyType"></a>

### ReadConsistencyType


| Name | Number | Description |
| ---- | ------ | ----------- |
| All | 0 | Send request to all nodes and return points which are present on all of them |
| Majority | 1 | Send requests to all nodes and return points which are present on majority of them |
| Quorum | 2 | Send requests to half &#43; 1 nodes, return points which are present on all of them |



<a name="qdrant-UpdateStatus"></a>

### UpdateStatus
Expand Down
85 changes: 85 additions & 0 deletions docs/redoc/master/openapi.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1938,6 +1938,15 @@
"schema": {
"$ref": "#/components/schemas/ExtendedPointId"
}
},
{
"name": "consistency",
"in": "query",
"description": "Define read consistency guarantees for the operation",
"required": false,
"schema": {
"$ref": "#/components/schemas/ReadConsistency"
}
}
],
"responses": {
Expand Down Expand Up @@ -2017,6 +2026,15 @@
"schema": {
"type": "string"
}
},
{
"name": "consistency",
"in": "query",
"description": "Define read consistency guarantees for the operation",
"required": false,
"schema": {
"$ref": "#/components/schemas/ReadConsistency"
}
}
],
"responses": {
Expand Down Expand Up @@ -2677,6 +2695,15 @@
"schema": {
"type": "string"
}
},
{
"name": "consistency",
"in": "query",
"description": "Define read consistency guarantees for the operation",
"required": false,
"schema": {
"$ref": "#/components/schemas/ReadConsistency"
}
}
],
"responses": {
Expand Down Expand Up @@ -2756,6 +2783,15 @@
"schema": {
"type": "string"
}
},
{
"name": "consistency",
"in": "query",
"description": "Define read consistency guarantees for the operation",
"required": false,
"schema": {
"$ref": "#/components/schemas/ReadConsistency"
}
}
],
"responses": {
Expand Down Expand Up @@ -2838,6 +2874,15 @@
"schema": {
"type": "string"
}
},
{
"name": "consistency",
"in": "query",
"description": "Define read consistency guarantees for the operation",
"required": false,
"schema": {
"$ref": "#/components/schemas/ReadConsistency"
}
}
],
"responses": {
Expand Down Expand Up @@ -2923,6 +2968,15 @@
"schema": {
"type": "string"
}
},
{
"name": "consistency",
"in": "query",
"description": "Define read consistency guarantees for the operation",
"required": false,
"schema": {
"$ref": "#/components/schemas/ReadConsistency"
}
}
],
"responses": {
Expand Down Expand Up @@ -3005,6 +3059,15 @@
"schema": {
"type": "string"
}
},
{
"name": "consistency",
"in": "query",
"description": "Define read consistency guarantees for the operation",
"required": false,
"schema": {
"$ref": "#/components/schemas/ReadConsistency"
}
}
],
"responses": {
Expand Down Expand Up @@ -6674,6 +6737,28 @@
"medium",
"strong"
]
},
"ReadConsistency": {
"description": "Read consistency parameter\n\nDefines how many replicas should be queried to get the result\n\n* `N` - send N random request and return points, which present on all of them\n\n* `majority` - send N/2+1 random request and return points, which present on all of them\n\n* `quorum` - send requests to all nodes and return points which present on majority of them\n\n* `all` - send requests to all nodes and return points which present on all of them\n\nDefault value is `Factor(1)`",
"anyOf": [
{
"type": "integer",
"format": "uint",
"minimum": 0
},
{
"$ref": "#/components/schemas/ReadConsistencyType"
}
]
},
"ReadConsistencyType": {
"description": "* `majority` - send N/2+1 random request and return points, which present on all of them\n\n* `quorum` - send requests to all nodes and return points which present on majority of nodes\n\n* `all` - send requests to all nodes and return points which present on all nodes",
"type": "string",
"enum": [
"majority",
"quorum",
"all"
]
}
}
}
Expand Down
22 changes: 0 additions & 22 deletions lib/api/src/grpc/conversions.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -427,28 +427,6 @@ impl TryFrom<Vectors> for segment::data_types::vectors::VectorStruct {
}
}

impl TryFrom<ScoredPoint> for segment::types::ScoredPoint {
type Error = Status;

fn try_from(point: ScoredPoint) -> Result<Self, Self::Error> {
let vector = point
.vectors
.map(|vectors| vectors.try_into())
.transpose()?;

Ok(Self {
id: match point.id {
None => return Err(Status::invalid_argument("Point does not have an ID")),
Some(id) => id.try_into()?,
},
payload: Some(proto_to_payloads(point.payload)?),
score: point.score,
vector,
version: point.version,
})
}
}

impl From<segment::types::WithVector> for WithVectorsSelector {
fn from(with_vectors: segment::types::WithVector) -> Self {
let selector_options = match with_vectors {
Expand Down
19 changes: 19 additions & 0 deletions lib/api/src/grpc/proto/points.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,19 @@ message WriteOrdering {
WriteOrderingType type = 1; // Write ordering guarantees
}

enum ReadConsistencyType {
All = 0; // Send request to all nodes and return points which are present on all of them
Majority = 1; // Send requests to all nodes and return points which are present on majority of them
Quorum = 2; // Send requests to half + 1 nodes, return points which are present on all of them
}

message ReadConsistency {
oneof value {
ReadConsistencyType type = 1; // Common read consistency configurations
uint64 factor = 2; // Send request to a specified number of nodes, and return points which are present on all of them
}
}

// ---------------------------------------------
// ------------- Point Id Requests -------------
// ---------------------------------------------
Expand Down Expand Up @@ -55,6 +68,7 @@ message GetPoints {
reserved 3; // deprecated "with_vector" field
WithPayloadSelector with_payload = 4; // Options for specifying which payload to include or not
optional WithVectorsSelector with_vectors = 5; // Options for specifying which vectors to include into response
optional ReadConsistency read_consistency = 6; // Options for specifying read consistency guarantees
}

message SetPayloadPoints {
Expand Down Expand Up @@ -169,11 +183,13 @@ message SearchPoints {
optional uint64 offset = 9; // Offset of the result
optional string vector_name = 10; // Which vector to use for search, if not specified - use default vector
optional WithVectorsSelector with_vectors = 11; // Options for specifying which vectors to include into response
optional ReadConsistency read_consistency = 12; // Options for specifying read consistency guarantees
}

message SearchBatchPoints {
string collection_name = 1; // Name of the collection
repeated SearchPoints search_points = 2;
optional ReadConsistency read_consistency = 3; // Options for specifying read consistency guarantees
}

message ScrollPoints {
Expand All @@ -184,6 +200,7 @@ message ScrollPoints {
reserved 5; // deprecated "with_vector" field
WithPayloadSelector with_payload = 6; // Options for specifying which payload to include or not
optional WithVectorsSelector with_vectors = 7; // Options for specifying which vectors to include into response
optional ReadConsistency read_consistency = 8; // Options for specifying read consistency guarantees
}

message LookupLocation {
Expand All @@ -205,11 +222,13 @@ message RecommendPoints {
optional string using = 11; // Define which vector to use for recommendation, if not specified - default vector
optional WithVectorsSelector with_vectors = 12; // Options for specifying which vectors to include into response
optional LookupLocation lookup_from = 13; // Name of the collection to use for points lookup, if not specified - use current collection
optional ReadConsistency read_consistency = 14; // Options for specifying read consistency guarantees
}

message RecommendBatchPoints {
string collection_name = 1; // Name of the collection
repeated RecommendPoints recommend_points = 2;
optional ReadConsistency read_consistency = 3; // Options for specifying read consistency guarantees
}

message CountPoints {
Expand Down
Loading

0 comments on commit 34bdbc2

Please sign in to comment.