[Bug]: Query after Insertion timed out in v2.5.0-beta #38585
Labels
kind/bug
Issues or changes related a bug
triage/needs-information
Indicates an issue needs more information in order to work on it.
Is there an existing issue for this?
Environment
Current Behavior
Executed tasks to insert around 2k entities into a collection concurrently (concurrency is 12). Each task inserted only one entity. In the stage environment where Milvus has been upgraded to v2.5.0-beta, query got timed out after roughly 10 minutes. However, the query almost immediately returned results in Prod environment where Milvus is still at v2.4.15.
Enabled TimeTick protection in the Stage environment and noticed that the TimeTick lag went up to more than 3 minutes during the insertion and lasted about 10 minutes before subsiding gradually to the normal level. At the same time, the TimeTick lag recorded at QueryNode (for consumed insert) also went up to a couple minutes. All of the evidence seems to point to a slow-down of consumption from dml channel in QueryNode.
Appreciate anyone looking into this issue since it could be a blocker to upgrade to v2.5+ in Prod environment.
Expected Behavior
Timetick lag should not have obvious increase during insertion.
Steps To Reproduce
No response
Milvus Log
[2024/12/18 10:57:54.092 +00:00] [WARN] [querynodev2/handlers.go:227] ["failed to query on delegator"] [traceID=62bcacd674186cd5d570ed04369a103f] [msgID=454692378587327825] [collectionID=454692378588586550] [channel=by-dev-rootcoord-dml_12_454692378588586550v0] [scope=All] [error="context canceled"]
[2024/12/18 10:57:54.092 +00:00] [WARN] [delegator/delegator.go:563] ["delegator query failed to wait tsafe"] [traceID=62bcacd674186cd5d570ed04369a103f] [collectionID=454692378588586550] [channel=by-dev-rootcoord-dml_12_454692378588586550v0] [replicaID=454692378786922498] [error="context canceled"]
Anything else?
No response
The text was updated successfully, but these errors were encountered: