Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Very high p99 with low requests and sufficient resources #34476

Open
1 task done
syang1997 opened this issue Jul 8, 2024 · 59 comments
Open
1 task done

[Bug]: Very high p99 with low requests and sufficient resources #34476

syang1997 opened this issue Jul 8, 2024 · 59 comments
Assignees
@yanliang567 @congqixia
Labels
help wanted Extra attention is needed
Milestone
2.4.19

Comments

@syang1997
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: v2.3.15
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):  Java sdk v2.3.4
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: qurey node 8c 16g
- GPU: 
- Others:

Current Behavior

During the smooth request, p99 suddenly increased to 15kms,
but the resources were sufficient and the CPU and memory were low. What was the reason?

The following is the monitoring

img
img_1
img_2
img_3
img_4

The following is the qureynode log

[newVersion=1720315568890138213] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/07/07 01:26:18.893 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1720315558893267636] [newVersion=1720315568890093833] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/07/07 01:26:28.887 +00:00] [INFO] [querynodev2/services.go:1385] ["sync action"] [traceID=5f972dd954fb612893c39b167c22d19c] [collectionID=449980176319159710] [channel=milvus-c-customer-service-rootcoord-dml_4_449980176319159710v0] [currentNodeID=39] [Action=UpdateVersion] [TargetVersion=1720315578895106277]
[2024/07/07 01:26:28.887 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1720315568889907859] [newVersion=1720315578895106277] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/07/07 01:26:28.888 +00:00] [INFO] [querynodev2/services.go:1385] ["sync action"] [traceID=f99fdc60cdb2dfa46725f861cad74639] [collectionID=449980176295686266] [channel=milvus-c-customer-service-rootcoord-dml_10_449980176295686266v0] [currentNodeID=39] [Action=UpdateVersion] [TargetVersion=1720315578895383796]
[2024/07/07 01:26:28.888 +00:00] [INFO] [querynodev2/services.go:1385] ["sync action"] [traceID=37a70ecd2a87c776cf6b344bec75bfc4] [collectionID=449980176309032169] [channel=milvus-c-customer-service-rootcoord-dml_15_449980176309032169v0] [currentNodeID=39] [Action=UpdateVersion] [TargetVersion=1720315578894055581]
[2024/07/07 01:26:28.889 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1720315568890093833] [newVersion=1720315578894055581] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/07/07 01:26:28.888 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1720315568890025022] [newVersion=1720315578895383796] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/07/07 01:26:28.889 +00:00] [INFO] [querynodev2/services.go:1385] ["sync action"] [traceID=70147845a02f2277c5442f83e17b61f5] [collectionID=449980176317610505] [channel=milvus-c-customer-service-rootcoord-dml_3_449980176317610505v0] [currentNodeID=39] [Action=UpdateVersion] [TargetVersion=1720315578894246706]
[2024/07/07 01:26:28.889 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1720315568889805018] [newVersion=1720315578894246706] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/07/07 01:26:28.888 +00:00] [INFO] [querynodev2/services.go:1385] ["sync action"] [traceID=a588afc25adcb0b5d29fb43e1eaeb6a3] [collectionID=449980176308582545] [channel=milvus-c-customer-service-rootcoord-dml_13_449980176308582545v0] [currentNodeID=39] [Action=UpdateVersion] [TargetVersion=1720315578895006088]
[2024/07/07 01:26:28.889 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1720315568890138213] [newVersion=1720315578895006088] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/07/07 01:26:28.888 +00:00] [INFO] [querynodev2/services.go:1385] ["sync action"] [traceID=dd7f1cc82b6dcd01add1589bb330dec3] [collectionID=449980176316006753] [channel=milvus-c-customer-service-rootcoord-dml_1_449980176316006753v0] [currentNodeID=39] [Action=UpdateVersion] [TargetVersion=1720315578895190401]
[2024/07/07 01:26:28.889 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1720315568889576391] [newVersion=1720315578895190401] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/07/07 01:26:28.888 +00:00] [INFO] [querynodev2/services.go:1385] ["sync action"] [traceID=6a896409a6ffd785eac14e3e97cdd30b] [collectionID=450884966347940353] [channel=milvus-c-customer-service-rootcoord-dml_2_450884966347940353v0] [currentNodeID=39] [Action=UpdateVersion] [TargetVersion=1720315578895063008]
[2024/07/07 01:26:28.889 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1720315568889949859] [newVersion=1720315578895063008] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/07/07 01:26:28.888 +00:00] [INFO] [querynodev2/services.go:1385] ["sync action"] [traceID=798958dbfb9c98e936505940cacb1e67] [collectionID=449980176314540038] [channel=milvus-c-customer-service-rootcoord-dml_0_449980176314540038v0] [currentNodeID=39]

Expected Behavior

No response

Steps To Reproduce

Problems occur occasionally but cannot be repeated

Milvus Log

No response

Anything else?

No response
@syang1997 syang1997 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 8, 2024
@yanliang567
Copy link
Contributor

quick questions:

  1. are you running all the query nodes with exclusive hardwares underlying?
  2. do you happen to have any metrics about network and disk performance at that moment?
  3. please share the completed milvus logs

/assign @congqixia @syang1997

@yanliang567 yanliang567 added triage/needs-information Indicates an issue needs more information in order to work on it. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 8, 2024
@yanliang567 yanliang567 added this to the 2.4.6 milestone Jul 8, 2024
@syang1997
Copy link
Author

quick questions:

  1. are you running all the query nodes with exclusive hardwares underlying?
  2. do you happen to have any metrics about network and disk performance at that moment?
  3. please share the completed milvus logs

/assign @congqixia @syang1997

  1. The k8s cluster is dedicated to milvus, and the underlying node of this milvus qureynode has no fluctuation in cpu usage and resources are available.
  2. The memory index used, and no writes and other operations were performed during the abnormal time period. No obvious abnormalities were found on the network
  3. There are too many logs for all components. I checked qureynode's logs and found no valuable information. Which component logs you need to exclude here, I can intercept the logs for the time period and upload them.

@syang1997
Copy link
Author

Node partial monitoring indicators @yanliang567
img_5
img_6

@yanliang567
Copy link
Contributor

/assign @congqixia
please help to take a look

@yanliang567 yanliang567 added help wanted Extra attention is needed and removed kind/bug Issues or changes related a bug triage/needs-information Indicates an issue needs more information in order to work on it. labels Jul 8, 2024
@syang1997
Copy link
Author

@yanliang567 Who can help with the investigation? Recently it appeared again.

@yanliang567
Copy link
Contributor

not have any clues yet. We need more info in logs to know what was happening at that moment. Please offer the full milvus pod logs during the period.

@yanliang567 yanliang567 modified the milestones: 2.4.6, 2.4.7 Jul 19, 2024
@syang1997
Copy link
Author

milvus-log (3).tar.gz
@yanliang567 I got the latest debug log of p99 timeout.
I filtered the log information and found that there is a log that is waiting for search to return to the long log in the proxy log, as follows:

Line 13498: [2024/07/23 15:05:56.323 +00:00] [DEBUG] [proxy/task_search.go:436] ["tr/proxy execute search 451345685543387137"] [traceID=02c29e210100a6076c41537ce36985cb] [msg=done] [duration=3.780114072s]
Line 13511: [2024/07/23 15:05:56.323 +00:00] [DEBUG] [proxy/impl.go:2634] [tr/Search] [traceID=02c29e210100a6076c41537ce36985cb] [msg="wait search result"] [duration=3.780748241s]

But I did not filter a similar query time in the log of QueryNode. It was initially suspected that it was a MQ problem, but the resource usage rate of monitoring and viewing Pulsar was very low, and the nodes at the time were not abnormal.

@syang1997
Copy link
Author

@yanliang567 Can you help me analyze the cause of timeout?

@yanliang567
Copy link
Contributor

okay, let me check the logs

@bigsheeper
Copy link
Contributor

bigsheeper commented Jul 24, 2024 •
edited
Loading

Hello @syang1997 , could you please provide us the monitoring for wait tsafe latency?

@syang1997
Copy link
Author

Hello @syang1997 , could you please provide us the monitoring for wait tsafe latency?

image image

@yanliang567
Copy link
Contributor

Additionally, please attach the metric screenshots around 2024/07/23 15:05(+1h、-1h), it would be helpful for us to address the issue. @syang1997

@syang1997
Copy link
Author

@bigsheeper The delay waiting for search is long, but the delay of QueryNode is not long.

@syang1997
Copy link
Author

Additionally, please attach the metric screenshots around 2024/07/23 15:05(+1h、-1h), it would be helpful for us to address the issue. @syang1997

Is there any way to export the specified time period? It has been a long time to introduce the log script to choose the 24 -hour log is too large

@bigsheeper
Copy link
Contributor

bigsheeper commented Jul 24, 2024 •
edited
Loading

@bigsheeper The delay waiting for search is long, but the delay of QueryNode is not long.

@syang1997 The wait tsafe latency monitoring is like this:

image

@syang1997
Copy link
Author

@bigsheeper The delay waiting for search is long, but the delay of QueryNode is not long.
@syang1997 The wait tsafe latency monitoring is like this:

image

I didn't find this panel, which version of Grafana Dashboard you used

@bigsheeper
Copy link
Contributor

bigsheeper commented Jul 24, 2024 •
edited
Loading

I didn't find this panel, which version of Grafana Dashboard you used

It's ok, if the querynode search request latency is low, then the wait tsafe latency is likely to be low as well.

@syang1997
Copy link
Author

@bigsheeper @yanliang567 What more information do you need from me?

@syang1997
Copy link
Author

proxy-57.log
querynode-70.log
querynode-71.log

traceID=e8d19598e7a4b42dfaddd2ea28565acd

@syang1997
Copy link
Author

image image

@syang1997
Copy link
Author

image image

@syang1997
Copy link
Author

image image image image image image image image

@xiaofan-luan
Copy link
Collaborator

even proxy mq latency increase
Uploading image.png…

this means this is most likely a proxy issue

@xiaofan-luan
Copy link
Collaborator

do we have host machine monitoring metrics?

@syang1997
Copy link
Author

syang1997 commented Aug 8, 2024 •
edited
Loading

@xiaofan-luan proxy node monitoring
image
image
image

@syang1997
Copy link
Author

syang1997 commented Aug 8, 2024 •
edited
Loading

image There is also a problem that PROXY includes cache memory high levels and is not GC.

@syang1997
Copy link
Author

querynode-71

log[2024/08/06 20:12:05.108 +00:00] [DEBUG] [querynodev2/services.go:69] ["QueryNode current state"] [NodeID=71] [StateCode=Healthy]
[2024/08/06 20:12:05.413 +00:00] [DEBUG] [config/etcd_source.go:142] ["etcd refreshConfigurations"] [prefix=milvus-c-customer-service/config] [endpoints="[milvus-c-customer-service-etcd.shein-paas-component:2379]"]
[2024/08/06 20:12:06.143 +00:00] [DEBUG] [querynodev2/services.go:69] ["QueryNode current state"] [NodeID=71] [StateCode=Healthy]
[2024/08/06 20:12:06.728 +00:00] [DEBUG] [pipeline/stream_pipeline.go:59] ["stream pipeline fetch msg"] [sum=0]
[2024/08/06 20:12:06.729 +00:00] [DEBUG] [querynodev2/services.go:696] ["start to search segments on worker"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msgID=451667590983188481] [collectionID=451243423475883453] [channel=milvus-c-customer-service-rootcoord-dml_11_451243423475883453v0] [scope=Historical] [segmentIDs="[451243423471022002]"]
[2024/08/06 20:12:06.729 +00:00] [DEBUG] [querynodev2/services.go:703] ["search segments..."] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msgID=451667590983188481] [collectionID=451243423475883453] [channel=milvus-c-customer-service-rootcoord-dml_11_451243423475883453v0] [scope=Historical]
[2024/08/06 20:12:06.729 +00:00] [DEBUG] [segments/validate.go:50] ["read target partitions"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [collectionID=451243423475883453] [partitionIDs="[451243423475883454]"]
[2024/08/06 20:12:06.729 +00:00] [DEBUG] [segments/segment.go:364] ["search segment..."] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [collectionID=451243423475883453] [segmentID=451243423471022002] [segmentType=Sealed] [withIndex=true]
[2024/08/06 20:12:06.751 +00:00] [DEBUG] [segments/segment.go:386] ["search segment done"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [collectionID=451243423475883453] [segmentID=451243423471022002] [segmentType=Sealed] [withIndex=true]
[2024/08/06 20:12:06.751 +00:00] [DEBUG] [querynodev2/services.go:727] [tr/searchSegments] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg="search segments done, channel = milvus-c-customer-service-rootcoord-dml_11_451243423475883453v0, segmentIDs = [451243423471022002]"] [duration=22.349629ms]
[2024/08/06 20:12:07.108 +00:00] [DEBUG] [querynodev2/services.go:69] ["QueryNode current state"] [NodeID=71] [StateCode=Healthy]
[2024/08/06 20:12:07.530 +00:00] [DEBUG] [querynodev2/services.go:696] ["start to search segments on worker"] [traceID=521a93e3eee743d974984af280207faf] [msgID=451667591821787137] [collectionID=451243423475883453] [channel=milvus-c-customer-service-rootcoord-dml_11_451243423475883453v0] [scope=Historical] [segmentIDs="[451243423471022002]"]
[2024/08/06 20:12:07.530 +00:00] [DEBUG] [querynodev2/services.go:703] ["search segments..."] [traceID=521a93e3eee743d974984af280207faf] [msgID=451667591821787137] [collectionID=451243423475883453] [channel=milvus-c-customer-service-rootcoord-dml_11_451243423475883453v0] [scope=Historical]
[2024/08/06 20:12:07.530 +00:00] [DEBUG] [segments/validate.go:50] ["read target partitions"] [traceID=521a93e3eee743d974984af280207faf] [collectionID=451243423475883453] [partitionIDs="[451243423475883454]"]
[2024/08/06 20:12:07.530 +00:00] [DEBUG] [segments/segment.go:364] ["search segment..."] [traceID=521a93e3eee743d974984af280207faf] [collectionID=451243423475883453] [segmentID=451243423471022002] [segmentType=Sealed] [withIndex=true]
[2024/08/06 20:12:07.550 +00:00] [DEBUG] [segments/segment.go:386] ["search segment done"] [traceID=521a93e3eee743d974984af280207faf] [collectionID=451243423475883453] [segmentID=451243423471022002] [segmentType=Sealed] [withIndex=true]
[2024/08/06 20:12:07.550 +00:00] [DEBUG] [querynodev2/services.go:727] [tr/searchSegments] [traceID=521a93e3eee743d974984af280207faf] [msg="search segments done, channel = milvus-c-customer-service-rootcoord-dml_11_451243423475883453v0, segmentIDs = [451243423471022002]"] [duration=20.633616ms]
[2024/08/06 20:12:07.857 +00:00] [DEBUG] [querynodev2/services.go:69] ["QueryNode current state"] [NodeID=71] [StateCode=Healthy]

proxy-66

[2024/08/06 20:12:04.300 +00:00] [DEBUG] [proxy/impl.go:2591] ["Search received"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [role=proxy] [db=ai_customer_service] [collection=tk_en_240731] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=3084] [OutputFields="[standard_question_id]"] [search_params="[{\"key\":\"anns_field\",\"value\":\"embedding\"},{\"key\":\"topk\",\"value\":\"1000\"},{\"key\":\"metric_type\",\"value\":\"COSINE\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"ignore_growing\",\"value\":\"false\"},{\"key\":\"offset\",\"value\":\"0\"},{\"key\":\"params\",\"value\":\"{\\\"nprobe\\\":64}\"}]"] [guarantee_timestamp=1]
[2024/08/06 20:12:04.301 +00:00] [DEBUG] [proxy/impl.go:2609] [tr/Search] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg="search request enqueue"] [duration=272.433µs]
[2024/08/06 20:12:04.301 +00:00] [DEBUG] [proxy/impl.go:2611] ["Search enqueued"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [role=proxy] [db=ai_customer_service] [collection=tk_en_240731] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=3084] [OutputFields="[standard_question_id]"] [search_params="[{\"key\":\"anns_field\",\"value\":\"embedding\"},{\"key\":\"topk\",\"value\":\"1000\"},{\"key\":\"metric_type\",\"value\":\"COSINE\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"ignore_growing\",\"value\":\"false\"},{\"key\":\"offset\",\"value\":\"0\"},{\"key\":\"params\",\"value\":\"{\\\"nprobe\\\":64}\"}]"] [guarantee_timestamp=1] [timestamp=451667590983188481]
[2024/08/06 20:12:04.301 +00:00] [DEBUG] [proxy/task_search.go:250] ["translate output fields"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [collID=451243423475883453] [collName=tk_en_240731] ["output fields"="[standard_question_id]"]
[2024/08/06 20:12:04.301 +00:00] [DEBUG] [proxy/task_search.go:314] ["create query plan"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [collID=451243423475883453] [collName=tk_en_240731] [nq=1] [dsl=] ["anns field"=embedding] ["query info"="topk:1000 metric_type:\"COSINE\" search_params:\"{\\\"nprobe\\\":64}\" round_decimal:-1 "]
[2024/08/06 20:12:04.301 +00:00] [DEBUG] [proxy/task_search.go:355] [Proxy::searchTask::PreExecute] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [collID=451243423475883453] [collName=tk_en_240731] [nq=1] [plan.OutputFieldIds="[107]"] [plan="vector_anns:<vector_type:FloatVector field_id:101 query_info:<topk:1000 metric_type:\"COSINE\" search_params:\"{\\\"nprobe\\\":64}\" round_decimal:-1 > placeholder_tag:\"$0\" > output_field_ids:107 "]
[2024/08/06 20:12:04.301 +00:00] [DEBUG] [proxy/task_search.go:402] ["search PreExecute done."] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [collID=451243423475883453] [collName=tk_en_240731] [nq=1] [guarantee_ts=451667589672468481] [use_default_consistency=true] ["consistency level"=Bounded] [timeout_ts=0]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:433] ["Search Execute done."] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [nq=1] [collection=451243423475883453] [partitionIDs="[]"]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:436] ["tr/proxy execute search 451667590983188481"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg=done] [duration=2.451684558s]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:712] ["all searches are finished or canceled"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:715] ["proxy receives one search result"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [sourceID=0]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:472] ["tr/searchTask PostExecute"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg=decodeResultStart] [duration=14.826µs]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:739] [tr/decodeSearchResults] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg="decodeSearchResults done"] [duration=119.946µs]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:487] ["proxy search post execute reduce"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [nq=1] [collection=451243423475883453] [partitionIDs="[]"] ["number of valid search results"=1]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:491] ["tr/searchTask PostExecute"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg=reduceResultStart] [duration=7.98µs]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:801] [reduceSearchResultData] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [len(subSearchResultData)=1] [nq=1] [offset=0] [limit=1000] [metricType=COSINE]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:839] [subSearchResultData] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] ["result No."=0] [nq=1] [topk=1000] ["length of pks"=1000] ["length of FieldsData"=1]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:931] ["skip duplicated search result"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [count=0]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:797] [tr/reduceSearchResultData] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg=done] [duration=249.528µs]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:518] ["Search post execute done"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [nq=1] [collection=451243423475883453] [partitionIDs="[]"]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/task_search.go:445] ["tr/searchTask PostExecute"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg=done] [duration=439.406µs]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/impl.go:2634] [tr/Search] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg="wait search result"] [duration=2.452387093s]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/impl.go:2640] [tr/Search] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [msg="wait search result"] [duration=17.01µs]
[2024/08/06 20:12:06.753 +00:00] [DEBUG] [proxy/impl.go:2641] ["Search done"] [traceID=d4e71f132bc05b66cd3f63063b7e04ed] [role=proxy] [db=ai_customer_service] [collection=tk_en_240731] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=3084] [OutputFields="[standard_question_id]"] [search_params="[{\"key\":\"anns_field\",\"value\":\"embedding\"},{\"key\":\"topk\",\"value\":\"1000\"},{\"key\":\"metric_type\",\"value\":\"COSINE\"},{\"key\":\"round_decimal\",\"value\":\"-1\"},{\"key\":\"ignore_growing\",\"value\":\"false\"},{\"key\":\"offset\",\"value\":\"0\"},{\"key\":\"params\",\"value\":\"{\\\"nprobe\\\":64}\"}]"] [guarantee_timestamp=1]

@xiaofan-luan
Copy link
Collaborator

it seems that when querynode receive this request, it already takes 2 seconds

@yanliang567 yanliang567 modified the milestones: 2.4.7, 2.4.8 Aug 12, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.8, 2.4.10 Aug 19, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.10, 2.4.11 Sep 5, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.11, 2.4.12 Sep 18, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.12, 2.4.13 Sep 27, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.13, 2.4.14 Oct 15, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.14, 2.4.16 Nov 14, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.16, 2.4.17, 2.4.18 Nov 21, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.18, 2.4.19 Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yanliang567 yanliang567

@congqixia congqixia

Labels
help wanted Extra attention is needed
Projects
None yet
Milestone
2.4.19
Development

No branches or pull requests
5 participants
@bigsheeper @syang1997 @yanliang567 @xiaofan-luan @congqixia