Segment batch search #813

generall · 2022-07-13T21:13:44Z

Another attempt to benchmark batching query.

Main question: at what level should we split batch into individual function calls?

There are 3 main options:

On collection level - where we can simply have batch request equivalent to the list of SearchRequest
- pros: Simplest to implement - no change even in internal API
- cons: Require to perform multiple internal calls to remote shard
On local shard level
- pros:
  - No need to change code on segment level and segment holder level
  - Single request in remote shard
- cons
  - Same filter calculated multiple times
On segment level
- pros:
  - Presumably smallest overhead on function calls
  - Can reuse point selection from filters
- cons:
  - requires the most amount of changes and potential code duplication

This PR introduces benchmark to compare scenario 2 and 3.
It makes 100 single queries to 2000 stored vectors vs single batch query with 100 vectors. Dim = 100
No indexes used.
Results:

No filters:

batch-search-bench/search-0                                                                             
                        time:   [4.3911 ms 4.4788 ms 4.5933 ms]
batch-search-bench/search-batch-0                                                                             
                        time:   [3.7360 ms 3.8431 ms 3.9657 ms]

Filter which matches 1/5 of the points:

batch-search-bench/search-1                                                                            
                        time:   [13.916 ms 14.146 ms 14.397 ms]
batch-search-bench/search-batch-1                                                                             
                        time:   [1.1711 ms 1.1870 ms 1.2080 ms]

Filter which matches all data points (to estimate filter overhead):

batch-search-bench/search-2                                                                            
                        time:   [19.268 ms 19.852 ms 20.570 ms]
batch-search-bench/search-batch-2                                                                            
                        time:   [5.4323 ms 5.4829 ms 5.5404 ms]

agourlay · 2022-08-08T20:03:59Z

rebased & fixed conflicts ✔️

agourlay · 2022-08-17T14:48:33Z

We can start an official round of reviews here to get things started 🔨

agourlay · 2022-08-17T14:59:23Z

lib/collection/src/collection.rs

-            with_payload.is_required()
-        } else {
-            false
+        // search is a special case of search_batch with a single batch


@generall Can you confirm that the dedicated search path needs to be re-instated here?

agourlay · 2022-08-17T15:07:13Z

lib/api/src/grpc/proto/points.proto

@@ -117,6 +117,11 @@ message SearchPoints {
  optional uint64 offset = 9; // Offset of the result
 }

+message SearchBatchPoints {
+  string collection_name = 1; // Name of the collection


The name of the collection is passed explicitly although the search_points contain a dedicated collection_name field.

The service validates that the collection_name exists and assumes all points in the batch targets the same collection.

This is not extremely clean for the time being.

agourlay · 2022-08-17T15:08:38Z

lib/collection/benches/prof.rs

@@ -0,0 +1,89 @@
+use std::fs::File;


Is it possible to share a single prof.rs file between all the benches across crates?

agourlay · 2022-08-17T15:09:20Z

lib/collection/src/collection.rs

+    ) -> CollectionResult<Vec<Vec<ScoredPoint>>> {
+        // A factor which determines if we need to use the 2-step search or not
+        // Should be adjusted based on usage statistics.
+        const PAYLOAD_TRANSFERS_FACTOR_THRESHOLD: usize = 10;


the logic is adapted from the regular search taking in consideration the batching aspects

agourlay · 2022-08-18T10:56:36Z

lib/collection/src/collection.rs

+                };
+                // Remove offset from top result.
+                if request.offset > 0 {
+                    if top_res.len() >= request.offset {


Added a fix after testing the usage of offsets.
The bug must be present in our regular search already.

generall · 2022-08-18T12:24:16Z

LGTM

agourlay

Need to approve my own work on this one.

agourlay force-pushed the segment-batch-search branch from 4f9ee69 to 13c04ff Compare August 8, 2022 18:50

agourlay force-pushed the segment-batch-search branch 3 times, most recently from 0e9e454 to 91bdead Compare August 15, 2022 19:23

generall and others added 10 commits August 16, 2022 11:36

batch search benchmark

4bc85b2

collect filter iterator in indexed search

bcdea92

fmt

a69e2dc

fix

b6f0370

fix

6afd9eb

fmt

bb66a54

use new tempfile create

f64c89d

auto batching

6e44079

Clippy fixes

5f24ffb

REST, gRPC and internal APIs

0c8b4aa

agourlay force-pushed the segment-batch-search branch from 91bdead to 0c8b4aa Compare August 16, 2022 09:42

agourlay added 3 commits August 16, 2022 20:31

fix bugs & less duplication

3bec130

two steps payload retrieval mechanism & fix duplication

eaaf9ca

add proxy_segment implementation & tests

80ebc93

agourlay marked this pull request as ready for review August 17, 2022 14:47

add gRPC docs

b836a5d

agourlay reviewed Aug 17, 2022

View reviewed changes

generall and others added 4 commits August 17, 2022 17:54

remove unused code (#950)

6f93a39

only filter ids within a batch

4c338d4

add more equivalence tests

08766a4

add integration test search vs batch

840e212

agourlay added 3 commits August 18, 2022 11:01

assert more search options in tests

33149ec

cleanup assertions

c51c0d9

fix offset panic

bf6f4a1

agourlay reviewed Aug 18, 2022

View reviewed changes

agourlay added 2 commits August 18, 2022 13:00

rename search batch API

ae60cb8

openapi spec

04988e3

agourlay approved these changes Aug 18, 2022

View reviewed changes

agourlay merged commit a2acca0 into master Aug 18, 2022

agourlay linked an issue Aug 18, 2022 that may be closed by this pull request

batch search queries #186

Closed

agourlay deleted the segment-batch-search branch September 9, 2022 08:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segment batch search #813

Segment batch search #813

generall commented Jul 13, 2022 •

edited

Loading

agourlay commented Aug 8, 2022

agourlay commented Aug 17, 2022

agourlay Aug 17, 2022

agourlay Aug 17, 2022

agourlay Aug 17, 2022

agourlay Aug 17, 2022

agourlay Aug 18, 2022

generall commented Aug 18, 2022

agourlay left a comment

Segment batch search #813

Segment batch search #813

Conversation

generall commented Jul 13, 2022 • edited Loading

agourlay commented Aug 8, 2022

agourlay commented Aug 17, 2022

agourlay Aug 17, 2022

Choose a reason for hiding this comment

agourlay Aug 17, 2022

Choose a reason for hiding this comment

agourlay Aug 17, 2022

Choose a reason for hiding this comment

agourlay Aug 17, 2022

Choose a reason for hiding this comment

agourlay Aug 18, 2022

Choose a reason for hiding this comment

generall commented Aug 18, 2022

agourlay left a comment

Choose a reason for hiding this comment

generall commented Jul 13, 2022 •

edited

Loading