-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: In stability testing, the latency of query/search is unstable and has increased by many times at certain moments. #37527
Comments
/assign @liliu-z |
/assign @zhuwenxing |
/unassign /assign @liliu-z There's a strange point here: the time point at which the search/query delay increases coincides with the emergence of the rate limit on insert/delete operations. It's necessary to confirm whether these two events are related or independent. detailed pod info including node name
|
Looks like the increased deletion volume caused the rate limit. But it is weird to make search that slow. |
maybe related with #37413 |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Is there an existing issue for this?
Environment
Current Behavior
When the delay in query/search increased, a lot of rate limits on insert/delete operations appeared.
At the point of increased latency, the CPU and memory usage of the query node are relatively stable.
Expected Behavior
The latency of search/query can be relatively stable.
Steps To Reproduce
No response
Milvus Log
cluster: 4am
ns: chaos-testing
pod info
grafana link
Anything else?
Will the rate limit of insert/delete operations affect the latency of search/query? If so, it would contradict the design principle of Milvus's read-write separation.
The text was updated successfully, but these errors were encountered: