Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster] proxy mem usage increased abnormally and did not decrease after the test, client load raises error collection not loaded #31942

Open
1 task done
wangting0128 opened this issue Apr 7, 2024 · 8 comments
Open
1 task done

[Bug]: [benchmark][cluster] proxy mem usage increased abnormally and did not decrease after the test, client load raises error collection not loaded #31942

wangting0128 opened this issue Apr 7, 2024 · 8 comments
Assignees
@zhagnlu @longjiquan
Labels
2.4-features kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.21

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented Apr 7, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.4-20240403-39047192-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-multi-vector-dn4zz
test case name: test_concurrent_locust_25m_multi_hnsw_ddl_dql_dml_cluster

server:

NAME                                                              READY   STATUS                            RESTARTS           AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-multi-vr-dn4zz-1-8798-etcd-0                              1/1     Running                           0                  3d17h   10.104.33.173   4am-node36   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-etcd-1                              1/1     Running                           0                  3d17h   10.104.30.241   4am-node38   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-etcd-2                              1/1     Running                           0                  3d17h   10.104.31.41    4am-node34   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-datacoord-66f547c7fc-9sc46   1/1     Running                           0                  3d17h   10.104.14.7     4am-node18   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-datanode-7d6bc5f49b-4z9dg    1/1     Running                           1 (3d17h ago)      3d17h   10.104.19.34    4am-node28   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-datanode-7d6bc5f49b-59z49    1/1     Running                           1 (3d17h ago)      3d17h   10.104.9.170    4am-node14   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-indexcoord-76988cf79dg2b96   1/1     Running                           0                  3d17h   10.104.14.9     4am-node18   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-indexnode-7495c85474-6lqlq   1/1     Running                           0                  3d17h   10.104.33.171   4am-node36   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-indexnode-7495c85474-6xlsg   1/1     Running                           0                  3d17h   10.104.4.238    4am-node11   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-indexnode-7495c85474-h2vl8   1/1     Running                           0                  3d17h   10.104.14.6     4am-node18   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-indexnode-7495c85474-nbvmw   1/1     Running                           0                  3d17h   10.104.30.237   4am-node38   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-proxy-6d785946f4-9r8k5       1/1     Running                           1 (3d17h ago)      3d17h   10.104.4.237    4am-node11   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-querycoord-6d89989fccb9wkh   1/1     Running                           1 (3d17h ago)      3d17h   10.104.9.168    4am-node14   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-querynode-7587b58f79-6pm9d   1/1     Running                           0                  3d17h   10.104.24.83    4am-node29   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-querynode-7587b58f79-fwxn4   1/1     Running                           0                  3d17h   10.104.32.121   4am-node39   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-querynode-7587b58f79-hk6nv   1/1     Running                           0                  3d17h   10.104.13.116   4am-node16   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-querynode-7587b58f79-j5vfg   1/1     Running                           0                  3d17h   10.104.21.142   4am-node24   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-querynode-7587b58f79-nkxtj   1/1     Running                           0                  3d17h   10.104.1.43     4am-node10   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-querynode-7587b58f79-wkffq   1/1     Running                           0                  3d17h   10.104.6.46     4am-node13   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-milvus-rootcoord-64985b9789-tk9cj   1/1     Running                           1 (3d17h ago)      3d17h   10.104.9.167    4am-node14   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-minio-0                             1/1     Running                           0                  3d17h   10.104.33.174   4am-node36   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-minio-1                             1/1     Running                           0                  3d17h   10.104.26.172   4am-node32   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-minio-2                             1/1     Running                           0                  3d17h   10.104.31.43    4am-node34   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-minio-3                             1/1     Running                           0                  3d17h   10.104.16.236   4am-node21   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-bookie-0                     1/1     Running                           0                  3d17h   10.104.31.39    4am-node34   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-bookie-1                     1/1     Running                           0                  3d17h   10.104.30.242   4am-node38   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-bookie-2                     1/1     Running                           0                  3d17h   10.104.26.173   4am-node32   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-bookie-init-wpxbm            0/1     Completed                         0                  3d17h   10.104.9.169    4am-node14   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-broker-0                     1/1     Running                           0                  3d17h   10.104.27.85    4am-node31   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-proxy-0                      1/1     Running                           0                  3d17h   10.104.14.8     4am-node18   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-pulsar-init-wjc6q            0/1     Completed                         0                  3d17h   10.104.9.165    4am-node14   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-recovery-0                   1/1     Running                           0                  3d17h   10.104.9.166    4am-node14   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-zookeeper-0                  1/1     Running                           0                  3d17h   10.104.27.88    4am-node31   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-zookeeper-1                  1/1     Running                           0                  3d17h   10.104.16.242   4am-node21   <none>           <none>
fouramf-multi-vr-dn4zz-1-8798-pulsar-zookeeper-2                  1/1     Running                           0                  3d17h   10.104.34.164   4am-node37   <none>           <none>

queryNode CPU usage
截屏2024-04-07 11 34 46
queryNode memory usage
截屏2024-04-07 11 36 02

proxy CPU usage
截屏2024-04-07 11 35 24
proxy memory usage
image

client pod name: fouramf-multi-vector-dn4zz-3641650604
client log:
test_concurrent_locust_25m_multi_hnsw_ddl_dql_dml_cluster.zip
截屏2024-04-07 11 23 37

Expected Behavior

No response

Steps To Reproduce

1. create a collection with 8 fields: ["id", "float_vector", "float_vector_1", "float_vector_2", "float_vector_3", "int64_1", "bool_1", "varchar_1"]
2. build indexes:
   HNSW: "float_vector", "float_vector_1", "float_vector_2", "float_vector_3"
   STL_SORT: "id"
3. insert 25 million data
4. flush collection
5. build indexes with the same params again
6. load collection
7. concurrent requests:
   - delete
   - hybrid_search
   - insert
   - load <- raise error
   - query
   - scene_hybrid_search_test <- raise error
     4 vector fields, 4 scalar fields
     (collection: create->insert->flush->index->load->hybrid_search->drop)
   - scene_test <- raise error
     1 vector fields, 1 primaryKey field
     (collection: create->insert->flush->index->drop)
   - search

Milvus Log

No response

Anything else?

test result:

[2024-04-06 11:52:50,894 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-04-06 11:52:50,896 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-04-06 11:52:50,896 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-06 11:52:50,896 -  INFO - fouram]: grpc     delete                                                                         36565     0(0.00%) |    161       2   37023      8 |    0.14        0.00 (stats.py:789)
[2024-04-06 11:52:50,896 -  INFO - fouram]: grpc     hybrid_search                                                                 726200     0(0.00%) |     78       8   12920     22 |    2.80        0.00 (stats.py:789)
[2024-04-06 11:52:50,896 -  INFO - fouram]: grpc     insert                                                                         36596     0(0.00%) |    213      12   42277     33 |    0.14        0.00 (stats.py:789)
[2024-04-06 11:52:50,897 -  INFO - fouram]: grpc     load                                                                           36754    13(0.04%) |   1465       5   61959    120 |    0.14        0.00 (stats.py:789)
[2024-04-06 11:52:50,897 -  INFO - fouram]: grpc     query                                                                         362979     0(0.00%) |    165       3   42896      9 |    1.40        0.00 (stats.py:789)
[2024-04-06 11:52:50,897 -  INFO - fouram]: grpc     scene_hybrid_search_test                                                       72597  4667(6.43%) | 264363   14051 1842128 186000 |    0.28        0.02 (stats.py:789)
[2024-04-06 11:52:50,897 -  INFO - fouram]: grpc     scene_test                                                                     72881     1(0.00%) |  88203   57623  651388  77000 |    0.28        0.00 (stats.py:789)
[2024-04-06 11:52:50,897 -  INFO - fouram]: grpc     search                                                                        726604     0(0.00%) |     93      11   27306     26 |    2.80        0.00 (stats.py:789)
[2024-04-06 11:52:50,897 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-06 11:52:50,897 -  INFO - fouram]:          Aggregated                                                                   2071176  4681(0.23%) |  12491       2 1842128     24 |    7.99        0.02 (stats.py:789)
[2024-04-06 11:52:50,897 -  INFO - fouram]:  (stats.py:790)
[2024-04-06 11:52:50,904 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c2m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '16',
                                                              'memory': '32Gi'},
                                                   'requests': {'cpu': '8',
                                                                'memory': '16Gi'}},
                                     'replicas': 6},
                       'indexNode': {'resources': {'limits': {'cpu': '6.0',
                                                              'memory': '4Gi'},
                                                   'requests': {'cpu': '4.0',
                                                                'memory': '3Gi'}},
                                     'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '2Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '2Gi'}},
                                    'replicas': 2},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.4-20240403-39047192-amd64'}}},
            'host': 'fouramf-multi-vr-dn4zz-1-8798-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_25m_multi_hnsw_ddl_dql_dml_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_3': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img',
                                                                                                           'dim': 200}},
                                                                       'float_vector_2': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_3': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img',
                                                                                                           'dim': 200}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 25000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'float_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 100,
                                                       'during_time': '72h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 1,
                                                                  'timeout': 600,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 25000000}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'delete_length': 1,
                                                                  'timeout': 30}},
                                                      {'type': 'search',
                                                       'weight': 20,
                                                       'params': {'nq': 10,
                                                                  'top_k': 10,
                                                                  'search_param': {'ef': 32},
                                                                  'expr': {'float_1': {'GT': -1.0,
                                                                                       'LT': 12500000.0}},
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['float_1',
                                                                                    'float_vector_1'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 10,
                                                       'params': {'ids': None,
                                                                  'expr': {'float_1': {'GT': 0,
                                                                                       'LT': 100}},
                                                                  'output_fields': None,
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': False,
                                                                  'random_count': 0,
                                                                  'random_range': [0,
                                                                                   1],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 30}},
                                                      {'type': 'scene_test',
                                                       'weight': 2,
                                                       'params': {'dim': 128,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2',
                                                                  'other_fields': [],
                                                                  'scalars_params': {},
                                                                  'scalars_index': {},
                                                                  'vectors_index': {}}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 20,
                                                       'params': {'nq': 1,
                                                                  'top_k': 10,
                                                                  'reqs': [{'search_param': {'ef': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'top_k': 100},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'ef': 256},
                                                                            'anns_field': 'float_vector_2',
                                                                            'top_k': 200},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_3',
                                                                            'top_k': 30}],
                                                                  'rerank': {'WeightedRanker': [0.85,
                                                                                                0.95,
                                                                                                0.5,
                                                                                                0.5]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'scene_hybrid_search_test',
                                                       'weight': 2,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'top_k': 100},
                                                                           {'search_param': {'nprobe': 32},
                                                                            'anns_field': 'float_vector_1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'ef': 32},
                                                                            'anns_field': 'float_vector_2',
                                                                            'top_k': 5},
                                                                           {'search_param': {'search_list': 20},
                                                                            'anns_field': 'float_vector_3',
                                                                            'top_k': 10}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'dataset': 'local',
                                                                  'dim': 128,
                                                                  'shards_num': 2,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2',
                                                                  'other_fields': ['float_vector_1',
                                                                                   'float_vector_2',
                                                                                   'float_vector_3',
                                                                                   'int64_1',
                                                                                   'bool_1',
                                                                                   'varchar_1'],
                                                                  'replica_number': 1,
                                                                  'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift',
                                                                                                                         'dim': 128}},
                                                                                     'float_vector_2': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift',
                                                                                                                         'dim': 128}},
                                                                                     'float_vector_3': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift',
                                                                                                                         'dim': 128}}},
                                                                  'scalars_index': {'int64_1': {},
                                                                                    'bool_1': {'index_type': 'INVERTED'},
                                                                                    'varchar_1': {'index_type': 'INVERTED'}},
                                                                  'vectors_index': {'float_vector_1': {'index_type': 'IVF_FLAT',
                                                                                                       'index_param': {'nlist': 1024},
                                                                                                       'metric_type': 'L2'},
                                                                                    'float_vector_2': {'index_type': 'HNSW',
                                                                                                       'index_param': {'M': 8,
                                                                                                                       'efConstruction': 200},
                                                                                                       'metric_type': 'L2'},
                                                                                    'float_vector_3': {'index_type': 'DISKANN',
                                                                                                       'index_param': {},
                                                                                                       'metric_type': 'IP'}},
                                                                  'prepare_before_insert': False,
                                                                  'hybrid_search_counts': 10,
                                                                  'new_connect': False,
                                                                  'new_user': False}}]},
            'run_id': 2024040367696685,
            'datetime': '2024-04-03 09:32:49.836280',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 571.6071,
                                      'float_vector_1': {'RT': 314.0117},
                                      'float_vector_2': {'RT': 73.1191},
                                      'float_vector_3': {'RT': 38.3375},
                                      'id': {'RT': 0.5477}},
                            'insert': {'total_time': 4945.8941,
                                       'VPS': 5054.6978,
                                       'batch_time': 1.9784,
                                       'batch': 10000},
                            'flush': {'RT': 2.5256},
                            'load': {'RT': 25.7495},
                            'Locust': {'Aggregated': {'Requests': 2071176,
                                                      'Fails': 4681,
                                                      'RPS': 7.99,
                                                      'fail_s': 0.0,
                                                      'RT_max': 1842128.29,
                                                      'RT_avg': 12491.78,
                                                      'TP50': 24,
                                                      'TP99': 333000.0},
                                       'delete': {'Requests': 36565,
                                                  'Fails': 0,
                                                  'RPS': 0.14,
                                                  'fail_s': 0.0,
                                                  'RT_max': 37023.39,
                                                  'RT_avg': 161.73,
                                                  'TP50': 8,
                                                  'TP99': 3700.0},
                                       'hybrid_search': {'Requests': 726200,
                                                         'Fails': 0,
                                                         'RPS': 2.8,
                                                         'fail_s': 0.0,
                                                         'RT_max': 12920.28,
                                                         'RT_avg': 78.17,
                                                         'TP50': 22,
                                                         'TP99': 1400.0},
                                       'insert': {'Requests': 36596,
                                                  'Fails': 0,
                                                  'RPS': 0.14,
                                                  'fail_s': 0.0,
                                                  'RT_max': 42277.27,
                                                  'RT_avg': 213.43,
                                                  'TP50': 33,
                                                  'TP99': 3900.0},
                                       'load': {'Requests': 36754,
                                                'Fails': 13,
                                                'RPS': 0.14,
                                                'fail_s': 0.0,
                                                'RT_max': 61959.75,
                                                'RT_avg': 1465.9,
                                                'TP50': 120.0,
                                                'TP99': 16000.0},
                                       'query': {'Requests': 362979,
                                                 'Fails': 0,
                                                 'RPS': 1.4,
                                                 'fail_s': 0.0,
                                                 'RT_max': 42896.39,
                                                 'RT_avg': 165.21,
                                                 'TP50': 9,
                                                 'TP99': 3800.0},
                                       'scene_hybrid_search_test': {'Requests': 72597,
                                                                    'Fails': 4667,
                                                                    'RPS': 0.28,
                                                                    'fail_s': 0.06,
                                                                    'RT_max': 1842128.29,
                                                                    'RT_avg': 264363.27,
                                                                    'TP50': 186000.0,
                                                                    'TP99': 1032000.0},
                                       'scene_test': {'Requests': 72881,
                                                      'Fails': 1,
                                                      'RPS': 0.28,
                                                      'fail_s': 0.0,
                                                      'RT_max': 651388.04,
                                                      'RT_avg': 88203.99,
                                                      'TP50': 77000.0,
                                                      'TP99': 266000.0},
                                       'search': {'Requests': 726604,
                                                  'Fails': 0,
                                                  'RPS': 2.8,
                                                  'fail_s': 0.0,
                                                  'RT_max': 27306.25,
                                                  'RT_avg': 93.54,
                                                  'TP50': 26,
                                                  'TP99': 1500.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test 2.4-features labels Apr 7, 2024
@wangting0128 wangting0128 added this to the 2.4.0 milestone Apr 7, 2024
@wangting0128 wangting0128 mentioned this issue Apr 7, 2024
Closed
1 task
@yanliang567
Copy link
Contributor

/assign @zhagnlu
/unassign

@sre-ci-robot sre-ci-robot assigned zhagnlu and unassigned yanliang567 Apr 8, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 8, 2024
@wangting0128
Copy link
Contributor Author

Recurrent

test case name：test_concurrent_locust_25m_multi_hnsw_ddl_dql_dml_cluster
image：2.4-20240412-9613d368-amd64

server：

NAME                                                              READY   STATUS                            RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-multi-vr-4n9z2-52-7648-etcd-0                             1/1     Running                           0                2d14h   10.104.17.59    4am-node23   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-etcd-1                             1/1     Running                           0                2d14h   10.104.30.146   4am-node38   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-etcd-2                             1/1     Running                           0                2d14h   10.104.34.195   4am-node37   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-datacoord-868d6c4b754c9vw   1/1     Running                           0                2d14h   10.104.6.166    4am-node13   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-datanode-789d5cc66-4v2tw    1/1     Running                           1 (2d14h ago)    2d14h   10.104.5.230    4am-node12   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-datanode-789d5cc66-g6m8f    1/1     Running                           0                2d14h   10.104.6.167    4am-node13   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-indexcoord-9b95b694-bdbwc   1/1     Running                           0                2d14h   10.104.17.53    4am-node23   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-indexnode-785d96769d46dsp   1/1     Running                           0                2d14h   10.104.1.225    4am-node10   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-indexnode-785d96769d9wz4t   1/1     Running                           0                2d14h   10.104.24.118   4am-node29   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-indexnode-785d96769dfsqdt   1/1     Running                           0                2d14h   10.104.5.231    4am-node12   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-indexnode-785d96769dn6dzs   1/1     Running                           0                2d14h   10.104.15.115   4am-node20   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-proxy-9f7654d86-p9fff       1/1     Running                           1 (2d14h ago)    2d14h   10.104.4.69     4am-node11   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-querycoord-64cf66cc4tdwjv   1/1     Running                           1 (2d14h ago)    2d14h   10.104.17.54    4am-node23   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-querynode-6799986d875h9p4   1/1     Running                           0                2d14h   10.104.23.249   4am-node27   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-querynode-6799986d876cbvm   1/1     Running                           0                2d14h   10.104.20.250   4am-node22   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-querynode-6799986d8776cbq   1/1     Running                           0                2d14h   10.104.13.228   4am-node16   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-querynode-6799986d87krkwh   1/1     Running                           0                2d14h   10.104.29.110   4am-node35   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-querynode-6799986d87l72cq   1/1     Running                           0                2d14h   10.104.31.174   4am-node34   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-querynode-6799986d87tq256   1/1     Running                           0                2d14h   10.104.4.70     4am-node11   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-milvus-rootcoord-7f58c6856788586   1/1     Running                           1 (2d14h ago)    2d14h   10.104.34.190   4am-node37   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-minio-0                            1/1     Running                           0                2d14h   10.104.34.193   4am-node37   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-minio-1                            1/1     Running                           0                2d14h   10.104.33.119   4am-node36   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-minio-2                            1/1     Running                           0                2d14h   10.104.27.30    4am-node31   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-minio-3                            1/1     Running                           0                2d14h   10.104.19.134   4am-node28   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-bookie-0                    1/1     Running                           0                2d14h   10.104.17.60    4am-node23   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-bookie-1                    1/1     Running                           0                2d14h   10.104.30.147   4am-node38   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-bookie-2                    1/1     Running                           0                2d14h   10.104.27.33    4am-node31   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-bookie-init-6b7px           0/1     Completed                         0                2d14h   10.104.21.103   4am-node24   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-broker-0                    1/1     Running                           0                2d14h   10.104.26.27    4am-node32   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-proxy-0                     1/1     Running                           0                2d14h   10.104.19.132   4am-node28   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-pulsar-init-p9n4x           0/1     Completed                         0                2d14h   10.104.26.28    4am-node32   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-recovery-0                  1/1     Running                           0                2d14h   10.104.21.104   4am-node24   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-zookeeper-0                 1/1     Running                           0                2d14h   10.104.33.118   4am-node36   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-zookeeper-1                 1/1     Running                           0                2d14h   10.104.20.254   4am-node22   <none>           <none>
fouramf-multi-vr-4n9z2-52-7648-pulsar-zookeeper-2                 1/1     Running                           0                2d14h   10.104.34.201   4am-node37   <none>           <none>

memory of dataCoord, rootCoord, proxy increased during test
截屏2024-04-15 17 16 17
截屏2024-04-15 17 17 48

CPU of dataCoord, rootCoord, proxy increased during test
截屏2024-04-15 17 21 25
截屏2024-04-15 17 22 03

client pod name: fouramf-multi-vector-4n9z2-121786545
client log:
client.log.zip
截屏2024-04-15 17 15 03

test result:

[2024-04-14 23:44:23,363 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-04-14 23:44:23,364 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-04-14 23:44:23,364 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-14 23:44:23,364 -  INFO - fouram]: grpc     delete                                                                         24100     0(0.00%) |     34       2    8026      7 |    0.11        0.00 (stats.py:789)
[2024-04-14 23:44:23,364 -  INFO - fouram]: grpc     hybrid_search                                                                 478007     0(0.00%) |     36       8  149367     17 |    2.21        0.00 (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]: grpc     insert                                                                         23881     0(0.00%) |    607      12  106914     28 |    0.11        0.00 (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]: grpc     load                                                                           24052    40(0.17%) |    941       5   30003     56 |    0.11        0.00 (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]: grpc     query                                                                         238556     0(0.00%) |     33       3   40121      7 |    1.10        0.00 (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]: grpc     scene_hybrid_search_test                                                       48175    18(0.04%) | 330567    8253 2342025 289000 |    0.22        0.00 (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]: grpc     scene_test                                                                     47785    17(0.04%) | 115598    7590 1519290 108000 |    0.22        0.00 (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]: grpc     search                                                                        476990     0(0.00%) |     38      10  135415     20 |    2.21        0.00 (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]:          Aggregated                                                                   1361546    75(0.01%) |  15813       2 2342025     18 |    6.31        0.00 (stats.py:789)
[2024-04-14 23:44:23,365 -  INFO - fouram]:  (stats.py:790)
[2024-04-14 23:44:23,370 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c2m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '16',
                                                              'memory': '32Gi'},
                                                   'requests': {'cpu': '8',
                                                                'memory': '16Gi'}},
                                     'replicas': 6},
                       'indexNode': {'resources': {'limits': {'cpu': '6.0',
                                                              'memory': '4Gi'},
                                                   'requests': {'cpu': '4.0',
                                                                'memory': '3Gi'}},
                                     'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '2Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '2Gi'}},
                                    'replicas': 2},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.4-20240412-9613d368-amd64'}}},
            'host': 'fouramf-multi-vr-4n9z2-52-7648-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_25m_multi_hnsw_ddl_dql_dml_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_3': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img',
                                                                                                           'dim': 200}},
                                                                       'float_vector_2': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_3': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img',
                                                                                                           'dim': 200}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 25000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'float_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 100,
                                                       'during_time': '60h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 1,
                                                                  'timeout': 600,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 25000000}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'delete_length': 1,
                                                                  'timeout': 30}},
                                                      {'type': 'search',
                                                       'weight': 20,
                                                       'params': {'nq': 10,
                                                                  'top_k': 10,
                                                                  'search_param': {'ef': 32},
                                                                  'expr': {'float_1': {'GT': -1.0,
                                                                                       'LT': 12500000.0}},
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['float_1',
                                                                                    'float_vector_1'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 10,
                                                       'params': {'ids': None,
                                                                  'expr': {'float_1': {'GT': 0,
                                                                                       'LT': 100}},
                                                                  'output_fields': None,
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': False,
                                                                  'random_count': 0,
                                                                  'random_range': [0,
                                                                                   1],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 30}},
                                                      {'type': 'scene_test',
                                                       'weight': 2,
                                                       'params': {'dim': 128,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2',
                                                                  'other_fields': [],
                                                                  'scalars_params': {},
                                                                  'scalars_index': {},
                                                                  'vectors_index': {}}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 20,
                                                       'params': {'nq': 1,
                                                                  'top_k': 10,
                                                                  'reqs': [{'search_param': {'ef': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'top_k': 100},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'ef': 256},
                                                                            'anns_field': 'float_vector_2',
                                                                            'top_k': 200},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_3',
                                                                            'top_k': 30}],
                                                                  'rerank': {'WeightedRanker': [0.85,
                                                                                                0.95,
                                                                                                0.5,
                                                                                                0.5]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'scene_hybrid_search_test',
                                                       'weight': 2,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'top_k': 100},
                                                                           {'search_param': {'nprobe': 32},
                                                                            'anns_field': 'float_vector_1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'ef': 32},
                                                                            'anns_field': 'float_vector_2',
                                                                            'top_k': 5},
                                                                           {'search_param': {'search_list': 20},
                                                                            'anns_field': 'float_vector_3',
                                                                            'top_k': 10}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'dataset': 'local',
                                                                  'dim': 128,
                                                                  'shards_num': 2,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2',
                                                                  'other_fields': ['float_vector_1',
                                                                                   'float_vector_2',
                                                                                   'float_vector_3',
                                                                                   'int64_1',
                                                                                   'bool_1',
                                                                                   'varchar_1'],
                                                                  'replica_number': 1,
                                                                  'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift',
                                                                                                                         'dim': 128}},
                                                                                     'float_vector_2': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift',
                                                                                                                         'dim': 128}},
                                                                                     'float_vector_3': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift',
                                                                                                                         'dim': 128}}},
                                                                  'scalars_index': {'int64_1': {},
                                                                                    'bool_1': {'index_type': 'INVERTED'},
                                                                                    'varchar_1': {'index_type': 'INVERTED'}},
                                                                  'vectors_index': {'float_vector_1': {'index_type': 'IVF_FLAT',
                                                                                                       'index_param': {'nlist': 1024},
                                                                                                       'metric_type': 'L2'},
                                                                                    'float_vector_2': {'index_type': 'HNSW',
                                                                                                       'index_param': {'M': 8,
                                                                                                                       'efConstruction': 200},
                                                                                                       'metric_type': 'L2'},
                                                                                    'float_vector_3': {'index_type': 'DISKANN',
                                                                                                       'index_param': {},
                                                                                                       'metric_type': 'IP'}},
                                                                  'prepare_before_insert': False,
                                                                  'hybrid_search_counts': 10,
                                                                  'new_connect': False,
                                                                  'new_user': False}}]},
            'run_id': 2024041248952987,
            'datetime': '2024-04-12 09:41:35.152868',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 983.6586,
                                      'float_vector_1': {'RT': 452.5967},
                                      'float_vector_2': {'RT': 391.3642},
                                      'float_vector_3': {'RT': 1.0187},
                                      'id': {'RT': 0.5197}},
                            'insert': {'total_time': 3630.6284,
                                       'VPS': 6885.8603,
                                       'batch_time': 1.4523,
                                       'batch': 10000},
                            'flush': {'RT': 2.526},
                            'load': {'RT': 22.1774},
                            'Locust': {'Aggregated': {'Requests': 1361546,
                                                      'Fails': 75,
                                                      'RPS': 6.31,
                                                      'fail_s': 0.0,
                                                      'RT_max': 2342025.36,
                                                      'RT_avg': 15813.42,
                                                      'TP50': 18,
                                                      'TP99': 379000.0},
                                       'delete': {'Requests': 24100,
                                                  'Fails': 0,
                                                  'RPS': 0.11,
                                                  'fail_s': 0.0,
                                                  'RT_max': 8026.62,
                                                  'RT_avg': 34.12,
                                                  'TP50': 7,
                                                  'TP99': 540.0},
                                       'hybrid_search': {'Requests': 478007,
                                                         'Fails': 0,
                                                         'RPS': 2.21,
                                                         'fail_s': 0.0,
                                                         'RT_max': 149367.0,
                                                         'RT_avg': 36.11,
                                                         'TP50': 17,
                                                         'TP99': 510.0},
                                       'insert': {'Requests': 23881,
                                                  'Fails': 0,
                                                  'RPS': 0.11,
                                                  'fail_s': 0.0,
                                                  'RT_max': 106914.11,
                                                  'RT_avg': 607.16,
                                                  'TP50': 28,
                                                  'TP99': 8400.0},
                                       'load': {'Requests': 24052,
                                                'Fails': 40,
                                                'RPS': 0.11,
                                                'fail_s': 0.0,
                                                'RT_max': 30003.68,
                                                'RT_avg': 941.78,
                                                'TP50': 56,
                                                'TP99': 9800.0},
                                       'query': {'Requests': 238556,
                                                 'Fails': 0,
                                                 'RPS': 1.1,
                                                 'fail_s': 0.0,
                                                 'RT_max': 40121.62,
                                                 'RT_avg': 34.0,
                                                 'TP50': 7,
                                                 'TP99': 520.0},
                                       'scene_hybrid_search_test': {'Requests': 48175,
                                                                    'Fails': 18,
                                                                    'RPS': 0.22,
                                                                    'fail_s': 0.0,
                                                                    'RT_max': 2342025.36,
                                                                    'RT_avg': 330567.44,
                                                                    'TP50': 289000.0,
                                                                    'TP99': 916000.0},
                                       'scene_test': {'Requests': 47785,
                                                      'Fails': 17,
                                                      'RPS': 0.22,
                                                      'fail_s': 0.0,
                                                      'RT_max': 1519290.3,
                                                      'RT_avg': 115598.71,
                                                      'TP50': 108000.0,
                                                      'TP99': 240000.0},
                                       'search': {'Requests': 476990,
                                                  'Fails': 0,
                                                  'RPS': 2.21,
                                                  'fail_s': 0.0,
                                                  'RT_max': 135415.52,
                                                  'RT_avg': 38.53,
                                                  'TP50': 20,
                                                  'TP99': 520.0}}}}}

@xiaofan-luan
Copy link
Collaborator

/assign @longjiquan

@xiaofan-luan
Copy link
Collaborator

please help on it

@zhagnlu
Copy link
Contributor

zhagnlu commented Apr 16, 2024

this issue include several problems:
first:
insert occassionly failed, such as
image
two bugs:

  1. not show errmsg correctly, the reason, datacoord AssignSegment not return err correctly，
    image

  2. datacoord load collection from rootcoord maybe timeout, for this, need add retry mechanism

@zhagnlu zhagnlu mentioned this issue Apr 16, 2024
Merged
xiaofan-luan pushed a commit that referenced this issue Apr 16, 2024
@zhagnlu
fix: correct AssignSegmentID return and add retry for loadCollectionF… (
4586bce 
#32335)

#32322
#31942

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
@yanliang567 yanliang567 modified the milestones: 2.4.0, 2.4.1 Apr 18, 2024
@wangting0128 wangting0128 added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Apr 23, 2024
@congqixia
Copy link
Contributor

@wangting0128 do we have any run with pyroscope enabled?

@wangting0128
Copy link
Contributor Author

@wangting0128 do we have any run with pyroscope enabled?

Yes, let us communicate offline

@yanliang567 yanliang567 modified the milestones: 2.4.1, 2.4.2 May 7, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.2, 2.4.3 May 24, 2024
@wangting0128
Copy link
Contributor Author

Recurrent

argo task: fouramf-wxjlk-3985588210
test case name: test_concurrent_locust_25m_multi_hnsw_ddl_dql_dml_cluster
image: 2.4-20240614-fd1c7b1a-amd64

server:

NAME                                                              READY   STATUS      RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-25m-etcd-0                                           1/1     Running     0                2d22h   10.104.34.185   4am-node37   <none>           <none>
multi-vector-25m-etcd-1                                           1/1     Running     0                2d22h   10.104.18.122   4am-node25   <none>           <none>
multi-vector-25m-etcd-2                                           1/1     Running     0                2d22h   10.104.26.6     4am-node32   <none>           <none>
multi-vector-25m-milvus-datacoord-5c8484f95c-5bzpc                1/1     Running     3 (2d22h ago)    2d22h   10.104.26.2     4am-node32   <none>           <none>
multi-vector-25m-milvus-datanode-5b6c98ddfb-njttz                 1/1     Running     3 (2d22h ago)    2d22h   10.104.20.87    4am-node22   <none>           <none>
multi-vector-25m-milvus-datanode-5b6c98ddfb-qvmsx                 1/1     Running     3 (2d22h ago)    2d22h   10.104.13.14    4am-node16   <none>           <none>
multi-vector-25m-milvus-indexcoord-bc59d4984-6wwss                1/1     Running     0                2d22h   10.104.13.15    4am-node16   <none>           <none>
multi-vector-25m-milvus-indexnode-7cdf4458dd-jhh6x                1/1     Running     3 (2d22h ago)    2d22h   10.104.30.35    4am-node38   <none>           <none>
multi-vector-25m-milvus-indexnode-7cdf4458dd-prwdl                1/1     Running     3 (2d22h ago)    2d22h   10.104.6.178    4am-node13   <none>           <none>
multi-vector-25m-milvus-indexnode-7cdf4458dd-x5dx8                1/1     Running     3 (2d22h ago)    2d22h   10.104.26.254   4am-node32   <none>           <none>
multi-vector-25m-milvus-indexnode-7cdf4458dd-znghg                1/1     Running     3 (2d22h ago)    2d22h   10.104.17.168   4am-node23   <none>           <none>
multi-vector-25m-milvus-proxy-6d49558fdd-fk8t6                    1/1     Running     3 (2d22h ago)    2d22h   10.104.13.17    4am-node16   <none>           <none>
multi-vector-25m-milvus-querycoord-64f44b5fc9-blzl7               1/1     Running     3 (2d22h ago)    2d22h   10.104.20.88    4am-node22   <none>           <none>
multi-vector-25m-milvus-querynode-57cbcbc985-4zwvh                1/1     Running     3 (2d22h ago)    2d22h   10.104.20.89    4am-node22   <none>           <none>
multi-vector-25m-milvus-querynode-57cbcbc985-cd59f                1/1     Running     3 (2d22h ago)    2d22h   10.104.34.182   4am-node37   <none>           <none>
multi-vector-25m-milvus-querynode-57cbcbc985-jfqfc                1/1     Running     3 (2d22h ago)    2d22h   10.104.18.115   4am-node25   <none>           <none>
multi-vector-25m-milvus-querynode-57cbcbc985-kr27h                1/1     Running     3 (2d22h ago)    2d22h   10.104.5.186    4am-node12   <none>           <none>
multi-vector-25m-milvus-querynode-57cbcbc985-t2h2k                1/1     Running     3 (2d22h ago)    2d22h   10.104.13.18    4am-node16   <none>           <none>
multi-vector-25m-milvus-querynode-57cbcbc985-t84cd                1/1     Running     2 (2d22h ago)    2d22h   10.104.4.84     4am-node11   <none>           <none>
multi-vector-25m-milvus-rootcoord-6fc6c69b9c-hsct6                1/1     Running     3 (2d22h ago)    2d22h   10.104.26.253   4am-node32   <none>           <none>
multi-vector-25m-minio-0                                          1/1     Running     0                2d22h   10.104.25.181   4am-node30   <none>           <none>
multi-vector-25m-minio-1                                          1/1     Running     0                2d22h   10.104.26.7     4am-node32   <none>           <none>
multi-vector-25m-minio-2                                          1/1     Running     0                2d22h   10.104.16.160   4am-node21   <none>           <none>
multi-vector-25m-minio-3                                          1/1     Running     0                2d22h   10.104.30.37    4am-node38   <none>           <none>
multi-vector-25m-pulsar-bookie-0                                  1/1     Running     0                2d22h   10.104.25.180   4am-node30   <none>           <none>
multi-vector-25m-pulsar-bookie-1                                  1/1     Running     0                2d22h   10.104.18.121   4am-node25   <none>           <none>
multi-vector-25m-pulsar-bookie-2                                  1/1     Running     0                2d22h   10.104.16.161   4am-node21   <none>           <none>
multi-vector-25m-pulsar-bookie-init-dc2cm                         0/1     Completed   0                2d22h   10.104.13.19    4am-node16   <none>           <none>
multi-vector-25m-pulsar-broker-0                                  1/1     Running     0                2d22h   10.104.17.167   4am-node23   <none>           <none>
multi-vector-25m-pulsar-proxy-0                                   1/1     Running     0                2d22h   10.104.14.20    4am-node18   <none>           <none>
multi-vector-25m-pulsar-pulsar-init-gfh6v                         0/1     Completed   0                2d22h   10.104.25.175   4am-node30   <none>           <none>
multi-vector-25m-pulsar-recovery-0                                1/1     Running     0                2d22h   10.104.34.181   4am-node37   <none>           <none>
multi-vector-25m-pulsar-zookeeper-0                               1/1     Running     0                2d22h   10.104.18.120   4am-node25   <none>           <none>
multi-vector-25m-pulsar-zookeeper-1                               1/1     Running     0                2d22h   10.104.25.183   4am-node30   <none>           <none>
multi-vector-25m-pulsar-zookeeper-2                               1/1     Running     0                2d22h   10.104.16.165   4am-node21   <none>           <none>

image

client pod name: fouramf-wxjlk-3985588210
client logs:
load timeout 30s

[2024-06-14 21:37:15,491 - ERROR - fouram]: grpc RpcError: [load_collection], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-06-14 21:36:45.481683', 'gRPC error': '2024-06-14 21:37:15.491306'}> (decorators.py:150)
[2024-06-14 21:37:15,498 - ERROR - fouram]: (api_response) : [Collection.load] <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-06-14T21:37:15.483385029+00:00"}"
>, [requestId: 32a3896c-2a96-11ef-9c53-3ec6fc29743c] (api_request.py:57)
[2024-06-14 21:37:15,499 - ERROR - fouram]: [CheckFunc] load request check failed, response:<_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-06-14T21:37:15.483385029+00:00"}"
> (func_check.py:48)
[2024-06-14 21:37:36,127 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-06-14 21:37:36,127 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-14 21:37:36,127 -  INFO - fouram]: grpc     delete                                                                         10250     0(0.00%) |     67       2    7682     10 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:37:36,127 -  INFO - fouram]: grpc     hybrid_search                                                                 206240     0(0.00%) |     53       8    7374     21 |    1.10        0.00 (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]: grpc     insert                                                                         10385     0(0.00%) |     96       4    7615     19 |    0.10        0.00 (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]: grpc     load                                                                           10549     1(0.01%) |    986       6   30011    410 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]: grpc     query                                                                         103092     0(0.00%) |     58       3    8325     11 |    0.30        0.00 (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]: grpc     scene_hybrid_search_test                                                       20574     0(0.00%) | 108112   10817  561526  99000 |    0.20        0.00 (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]: grpc     scene_test                                                                     20344     0(0.00%) |  80166   63290  243972  74000 |    0.30        0.00 (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]: grpc     search                                                                        206008     0(0.00%) |     60      12    7305     27 |    0.80        0.00 (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]:          Aggregated                                                                    587442     1(0.00%) |   6633       2  561526     24 |    2.80        0.00 (stats.py:789)
[2024-06-14 21:37:36,128 -  INFO - fouram]:  (stats.py:790)

[2024-06-14 21:39:53,940 - ERROR - fouram]: grpc RpcError: [load_collection], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-06-14 21:39:23.938380', 'gRPC error': '2024-06-14 21:39:53.940185'}> (decorators.py:150)
[2024-06-14 21:39:53,942 - ERROR - fouram]: (api_response) : [Collection.load] <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-06-14T21:39:53.939026209+00:00"}"
>, [requestId: 91161ad2-2a96-11ef-9c53-3ec6fc29743c] (api_request.py:57)
[2024-06-14 21:39:53,942 - ERROR - fouram]: [CheckFunc] load request check failed, response:<_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-06-14T21:39:53.939026209+00:00"}"
> (func_check.py:48)
[2024-06-14 21:39:56,407 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-06-14 21:39:56,407 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-14 21:39:56,408 -  INFO - fouram]: grpc     delete                                                                         10254     0(0.00%) |     67       2    7682     10 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,408 -  INFO - fouram]: grpc     hybrid_search                                                                 206313     0(0.00%) |     53       8    7374     21 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,408 -  INFO - fouram]: grpc     insert                                                                         10389     0(0.00%) |     96       4    7615     19 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,408 -  INFO - fouram]: grpc     load                                                                           10553     2(0.02%) |    992       6   30011    410 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,412 -  INFO - fouram]: grpc     query                                                                         103133     0(0.00%) |     58       3    8325     11 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,412 -  INFO - fouram]: grpc     scene_hybrid_search_test                                                       20583     0(0.00%) | 108280   10817  567065  99000 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,412 -  INFO - fouram]: grpc     scene_test                                                                     20357     0(0.00%) |  80250   63290  248739  74000 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,412 -  INFO - fouram]: grpc     search                                                                        206075     0(0.00%) |     60      12    7305     27 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,412 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-14 21:39:56,412 -  INFO - fouram]:          Aggregated                                                                    587657     2(0.00%) |   6643       2  567065     24 |    0.00        0.00 (stats.py:789)
[2024-06-14 21:39:56,412 -  INFO - fouram]:  (stats.py:790)

test result:

[2024-06-14 22:43:42,594 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-06-14 22:43:42,594 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: grpc     delete                                                                         11091     0(0.00%) |     67       2    7682     10 |    0.26        0.00 (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: grpc     hybrid_search                                                                 224244     0(0.00%) |     53       8    7374     22 |    5.19        0.00 (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: grpc     insert                                                                         11329     0(0.00%) |     97       4    7615     20 |    0.26        0.00 (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: grpc     load                                                                           11454     2(0.02%) |   1008       6   30011    420 |    0.27        0.00 (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: grpc     query                                                                         112006     0(0.00%) |     57       3    8325     11 |    2.59        0.00 (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: grpc     scene_hybrid_search_test                                                       22384     0(0.00%) | 110576   10817  695817  99000 |    0.52        0.00 (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: grpc     scene_test                                                                     22206     0(0.00%) |  80556   63290  292848  74000 |    0.51        0.00 (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: grpc     search                                                                        224084     0(0.00%) |     60      12    7305     28 |    5.19        0.00 (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-14 22:43:42,595 -  INFO - fouram]:          Aggregated                                                                    638798     2(0.00%) |   6746       2  695817     25 |   14.79        0.00 (stats.py:789)
[2024-06-14 22:43:42,596 -  INFO - fouram]:  (stats.py:790)
[2024-06-14 22:43:42,603 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c2m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '16',
                                                              'memory': '32Gi'},
                                                   'requests': {'cpu': '8',
                                                                'memory': '16Gi'}},
                                     'replicas': 6},
                       'indexNode': {'resources': {'limits': {'cpu': '6.0',
                                                              'memory': '4Gi'},
                                                   'requests': {'cpu': '4.0',
                                                                'memory': '3Gi'}},
                                     'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '2Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '2Gi'}},
                                    'replicas': 2},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.4-20240614-fd1c7b1a-amd64'}}},
            'host': 'multi-vector-25m-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_25m_multi_hnsw_ddl_dql_dml_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_3': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img'}},
                                                                       'float_vector_2': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_3': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img'}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 25000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'float_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 100,
                                                       'during_time': '12h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 1,
                                                                  'timeout': 600,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 25000000}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'delete_length': 1,
                                                                  'timeout': 30}},
                                                      {'type': 'search',
                                                       'weight': 20,
                                                       'params': {'nq': 10,
                                                                  'top_k': 10,
                                                                  'search_param': {'ef': 32},
                                                                  'expr': {'float_1': {'GT': -1.0,
                                                                                       'LT': 12500000.0}},
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['float_1',
                                                                                    'float_vector_1'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 10,
                                                       'params': {'ids': None,
                                                                  'expr': {'float_1': {'GT': 0,
                                                                                       'LT': 100}},
                                                                  'output_fields': None,
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': False,
                                                                  'random_count': 0,
                                                                  'random_range': [0,
                                                                                   1],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 30}},
                                                      {'type': 'scene_test',
                                                       'weight': 2,
                                                       'params': {'dim': 128,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2',
                                                                  'other_fields': [],
                                                                  'scalars_params': {},
                                                                  'scalars_index': {},
                                                                  'vectors_index': {}}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 20,
                                                       'params': {'nq': 1,
                                                                  'top_k': 10,
                                                                  'reqs': [{'search_param': {'ef': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'top_k': 100},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'ef': 256},
                                                                            'anns_field': 'float_vector_2',
                                                                            'top_k': 200},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_3',
                                                                            'top_k': 30}],
                                                                  'rerank': {'WeightedRanker': [0.85,
                                                                                                0.95,
                                                                                                0.5,
                                                                                                0.5]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'scene_hybrid_search_test',
                                                       'weight': 2,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'top_k': 100},
                                                                           {'search_param': {'nprobe': 32},
                                                                            'anns_field': 'float_vector_1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'ef': 32},
                                                                            'anns_field': 'float_vector_2',
                                                                            'top_k': 5},
                                                                           {'search_param': {'search_list': 20},
                                                                            'anns_field': 'float_vector_3',
                                                                            'top_k': 10}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'dataset': 'local',
                                                                  'dim': 128,
                                                                  'shards_num': 2,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2',
                                                                  'other_fields': ['float_vector_1',
                                                                                   'float_vector_2',
                                                                                   'float_vector_3',
                                                                                   'int64_1',
                                                                                   'bool_1',
                                                                                   'varchar_1'],
                                                                  'replica_number': 1,
                                                                  'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift'}},
                                                                                     'float_vector_2': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift'}},
                                                                                     'float_vector_3': {'params': {'dim': 128},
                                                                                                        'other_params': {'dataset': 'sift'}}},
                                                                  'scalars_index': {'int64_1': {},
                                                                                    'bool_1': {'index_type': 'INVERTED'},
                                                                                    'varchar_1': {'index_type': 'INVERTED'}},
                                                                  'vectors_index': {'float_vector_1': {'index_type': 'IVF_FLAT',
                                                                                                       'index_param': {'nlist': 1024},
                                                                                                       'metric_type': 'L2'},
                                                                                    'float_vector_2': {'index_type': 'HNSW',
                                                                                                       'index_param': {'M': 8,
                                                                                                                       'efConstruction': 200},
                                                                                                       'metric_type': 'L2'},
                                                                                    'float_vector_3': {'index_type': 'DISKANN',
                                                                                                       'index_param': {},
                                                                                                       'metric_type': 'IP'}},
                                                                  'prepare_before_insert': False,
                                                                  'hybrid_search_counts': 10,
                                                                  'new_connect': False,
                                                                  'new_user': False}}]},
            'run_id': 2024061455048265,
            'datetime': '2024-06-14 08:58:24.563525',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 797.7971,
                                      'float_vector_1': {'RT': 314.0947},
                                      'float_vector_2': {'RT': 160.9543},
                                      'float_vector_3': {'RT': 56.7346},
                                      'id': {'RT': 0.5175}},
                            'insert': {'total_time': 4500.9088,
                                       'VPS': 5554.4338,
                                       'batch_time': 1.8004,
                                       'batch': 10000},
                            'flush': {'RT': 2.5759},
                            'load': {'RT': 26.6893},
                            'Locust': {'Aggregated': {'Requests': 638798,
                                                      'Fails': 2,
                                                      'RPS': 14.79,
                                                      'fail_s': 0.0,
                                                      'RT_max': 695817.7,
                                                      'RT_avg': 6746.23,
                                                      'TP50': 25,
                                                      'TP99': 142000.0},
                                       'delete': {'Requests': 11091,
                                                  'Fails': 0,
                                                  'RPS': 0.26,
                                                  'fail_s': 0.0,
                                                  'RT_max': 7682.49,
                                                  'RT_avg': 67.31,
                                                  'TP50': 10,
                                                  'TP99': 1300.0},
                                       'hybrid_search': {'Requests': 224244,
                                                         'Fails': 0,
                                                         'RPS': 5.19,
                                                         'fail_s': 0.0,
                                                         'RT_max': 7374.99,
                                                         'RT_avg': 53.59,
                                                         'TP50': 22,
                                                         'TP99': 780.0},
                                       'insert': {'Requests': 11329,
                                                  'Fails': 0,
                                                  'RPS': 0.26,
                                                  'fail_s': 0.0,
                                                  'RT_max': 7615.48,
                                                  'RT_avg': 97.93,
                                                  'TP50': 20,
                                                  'TP99': 1600.0},
                                       'load': {'Requests': 11454,
                                                'Fails': 2,
                                                'RPS': 0.27,
                                                'fail_s': 0.0,
                                                'RT_max': 30011.03,
                                                'RT_avg': 1008.53,
                                                'TP50': 420.0,
                                                'TP99': 8400.0},
                                       'query': {'Requests': 112006,
                                                 'Fails': 0,
                                                 'RPS': 2.59,
                                                 'fail_s': 0.0,
                                                 'RT_max': 8325.88,
                                                 'RT_avg': 57.44,
                                                 'TP50': 11,
                                                 'TP99': 1000.0},
                                       'scene_hybrid_search_test': {'Requests': 22384,
                                                                    'Fails': 0,
                                                                    'RPS': 0.52,
                                                                    'fail_s': 0.0,
                                                                    'RT_max': 695817.7,
                                                                    'RT_avg': 110576.9,
                                                                    'TP50': 99000.0,
                                                                    'TP99': 356000.0},
                                       'scene_test': {'Requests': 22206,
                                                      'Fails': 0,
                                                      'RPS': 0.51,
                                                      'fail_s': 0.0,
                                                      'RT_max': 292848.61,
                                                      'RT_avg': 80556.57,
                                                      'TP50': 74000.0,
                                                      'TP99': 153000.0},
                                       'search': {'Requests': 224084,
                                                  'Fails': 0,
                                                  'RPS': 5.19,
                                                  'fail_s': 0.0,
                                                  'RT_max': 7305.05,
                                                  'RT_avg': 60.82,
                                                  'TP50': 28,
                                                  'TP99': 850.0}}}}}

@yanliang567 yanliang567 modified the milestones: 2.4.5, 2.4.6 Jun 26, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.6, 2.4.7 Jul 19, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.7, 2.4.8 Aug 12, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.8, 2.4.10 Aug 19, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.10, 2.4.11 Sep 5, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.11, 2.4.12 Sep 18, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.12, 2.4.13 Sep 27, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.13, 2.4.14 Oct 15, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.14, 2.4.16 Nov 14, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.16, 2.4.17, 2.4.18 Nov 21, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.18, 2.4.19, 2.4.20 Dec 24, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.20, 2.4.21 Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhagnlu zhagnlu

@longjiquan longjiquan

Labels
2.4-features kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.21
Development

No branches or pull requests
6 participants
@zhagnlu @wangting0128 @longjiquan @yanliang567 @xiaofan-luan @congqixia