[Bug]: [benchmark][cluster] search RT almost doubled after enabling streamingNode #36804
Comments
wangting0128
added
kind/bug
|
It seems that the difference of flush policy make the final segment size different. |
|
may be related to #36761 |
|
/unassign |
yanliang567
added
triage/accepted
|
#36761 is merged, but I do not make sure that it fix these issue. |
Verification failedargo task: fouramf-kmshk server:
client log: hybrid_search request timeout |
|
After Flushing Policy Fixed, FirstThe Milvus With Streaming Service will finally generate 1.11k sealed segments while the milvus without streaming service will finally generate 2k sealed segments. Streaming: No Streaming: SecondMilvus With Streaming Service's message consumer works correctly, so it's not introduced by streaming service. 
ThirdFound that the some request still wait for tsafe for long time whether using streaming or not. and ProcessInsert Delay increase periodically: No Streaming: Streaming: FourthFound that inserting a new message cost 19min when creating a new segment. |
|
Found that the insert operation is blocked by the acquirisition of mutex Release operation of segment |
|
tsafe timeout should be fixed by pr #36997 After flushing policy fixed: Meanwhile, milvus without streaming service don't generate flushed segment fluently. So the milvus without streaming service encounter less race condition than the milvus with streaming service when handing off, and performs more better RT. |
issue: #36804 Signed-off-by: chyezh <chyezh@outlook.com>
|
@wangting0128 please retry the test at commit |
|
At latest commit, tsafe problem is fixed. 
|
issue: #36804 Signed-off-by: chyezh <chyezh@outlook.com>
|
Found that |
|
Found that scalar search latency increase:
|
this is comapred master with what? could this be impacted by null? |
This is a comparison of the deployment of instances with and without streamingNode on the same case. |
Nope, two tests both ran on a milvus with different commit without streaming enabled. |
is there a special reason why so many bin logs is actaully generated? |
There's a binlog-num-based flush policy in milvus.
It has been fixed by #37037, milvus with streaming has kept consistency with milvus without streaming. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Is there an existing issue for this?
Environment
Current Behavior
argo task: fouramf-streaming-node-corn-1728615600
test case name: test_bitmap_locust_dql_dml_partition_key_cluster
server:
enabled streamingNode👇
disabled streamingNode👇
release name: fouramf-bitmap-scenes-q27w2-7
client log:
Expected Behavior
No response
Steps To Reproduce
concurrent test and calculation of RT and QPS :purpose: `partition_key on scalar int64_1 field`, shards_num=16 verify DQL & DML scenario, which has 1 vector fields(IVF_SQ8) and building `BITMAP` index on all supported 12 scalar fields :test steps: 1. create collection with fields: 'float_vector': 128dim 'int64_1': partition_key, num_partitions=1024 all scalar fields: varchar max_length=100, array max_capacity=13 2. build indexes: IVF_SQ8: 'float_vector' BITMAP: all scalar fields 3. insert 50 million data 4. flush collection 5. build indexes again using the same params 6. load collection replica: 1 7. concurrent request: - search - query - hybrid_search - load - insert - delete: delete data 90% - flush: ignore RateLimiterMilvus Log
No response
Anything else?
test result:
The text was updated successfully, but these errors were encountered: