[Bug]: [benchmark][cluster] queryNode restarts twice during concurrent dql & dml scene #35547

wangting0128 · 2024-08-19T03:44:29Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: 2.4-20240818-248a6ea4-amd64
- Deployment mode(standalone or cluster):cluster    
- MQ type(rocksmq, pulsar or kafka):pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.5rc7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: inverted-corn-1724014800
test case name: test_inverted_locust_hnsw_diskann_dml_dql_cluster

server:
inverted-corn-114800-8-20-2243-milvus-querynode-7c7b866c77pfpfp restarts twice

[2024-08-19 01:01:13,421 -  INFO - fouram]: [Base] Deploy initial state: 
I0818 21:09:26.653821     395 request.go:665] Waited for 1.180104035s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s
I0818 21:09:36.853163     395 request.go:665] Waited for 4.198498634s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s
NAME                                                              READY   STATUS      RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
inverted-corn-114800-8-20-2243-etcd-0                             1/1     Running     0                 5m32s   10.104.33.77    4am-node36   <none>           <none>
inverted-corn-114800-8-20-2243-etcd-1                             1/1     Running     0                 5m32s   10.104.34.231   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-etcd-2                             1/1     Running     0                 5m32s   10.104.26.151   4am-node32   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-datanode-9f5495564-r7cwj    1/1     Running     3 (4m38s ago)     5m32s   10.104.14.60    4am-node18   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-indexnode-54c98bd6ff5h6hg   1/1     Running     3 (4m38s ago)     5m32s   10.104.20.209   4am-node22   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-indexnode-54c98bd6ffb9q7c   1/1     Running     3 (4m35s ago)     5m32s   10.104.27.154   4am-node31   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-indexnode-54c98bd6ffdsx6d   1/1     Running     3 (4m39s ago)     5m32s   10.104.5.119    4am-node12   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-indexnode-54c98bd6ffzhtsf   1/1     Running     3 (4m38s ago)     5m32s   10.104.32.246   4am-node39   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-mixcoord-6cf5cb964-qb7fm    1/1     Running     3 (4m38s ago)     5m32s   10.104.5.117    4am-node12   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-proxy-b54ccbd6d-qwp5v       1/1     Running     3 (4m38s ago)     5m32s   10.104.34.219   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-querynode-7c7b866c77pfpfp   1/1     Running     3 (4m39s ago)     5m32s   10.104.24.28    4am-node29   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-querynode-7c7b866c77t8zmr   1/1     Running     3 (4m39s ago)     5m32s   10.104.23.96    4am-node27   <none>           <none>
inverted-corn-114800-8-20-2243-minio-0                            1/1     Running     0                 5m32s   10.104.18.11    4am-node25   <none>           <none>
inverted-corn-114800-8-20-2243-minio-1                            1/1     Running     0                 5m32s   10.104.33.76    4am-node36   <none>           <none>
inverted-corn-114800-8-20-2243-minio-2                            1/1     Running     0                 5m32s   10.104.34.235   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-minio-3                            1/1     Running     0                 5m32s   10.104.26.152   4am-node32   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-bookie-0                    1/1     Running     0                 5m32s   10.104.33.78    4am-node36   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-bookie-1                    1/1     Running     0                 5m32s   10.104.34.236   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-bookie-2                    1/1     Running     0                 5m31s   10.104.18.14    4am-node25   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-bookie-init-kclkn           0/1     Completed   0                 5m32s   10.104.34.218   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-broker-0                    1/1     Running     0                 5m32s   10.104.18.2     4am-node25   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-proxy-0                     1/1     Running     0                 5m32s   10.104.34.220   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-pulsar-init-9rjbc           0/1     Completed   0                 5m32s   10.104.5.118    4am-node12   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-recovery-0                  1/1     Running     0                 5m32s   10.104.26.139   4am-node32   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-zookeeper-0                 1/1     Running     0                 5m32s   10.104.34.234   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-zookeeper-1                 1/1     Running     0                 4m3s    10.104.20.229   4am-node22   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-zookeeper-2                 1/1     Running     0                 3m28s   10.104.26.159   4am-node32   <none>           <none> (base.py:261)
[2024-08-19 01:01:13,421 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'NAME|inverted-corn-114800-8-20-2243-milvus|inverted-corn-114800-8-20-2243-minio|inverted-corn-114800-8-20-2243-etcd|inverted-corn-114800-8-20-2243-pulsar|inverted-corn-114800-8-20-2243-zookeeper|inverted-corn-114800-8-20-2243-kafka|inverted-corn-114800-8-20-2243-log|inverted-corn-114800-8-20-2243-tikv'  (util_cmd.py:14)
[2024-08-19 01:01:33,871 -  INFO - fouram]: [CliClient] pod details of release(inverted-corn-114800-8-20-2243): 
 I0819 01:01:14.681733     529 request.go:665] Waited for 1.17717521s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/node.k8s.io/v1?timeout=32s
I0819 01:01:24.880975     529 request.go:665] Waited for 4.197595635s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/coordination.k8s.io/v1?timeout=32s
NAME                                                              READY   STATUS      RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
inverted-corn-114800-8-20-2243-etcd-0                             1/1     Running     0                 3h57m   10.104.33.77    4am-node36   <none>           <none>
inverted-corn-114800-8-20-2243-etcd-1                             1/1     Running     0                 3h57m   10.104.34.231   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-etcd-2                             1/1     Running     0                 3h57m   10.104.26.151   4am-node32   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-datanode-9f5495564-r7cwj    1/1     Running     3 (3h56m ago)     3h57m   10.104.14.60    4am-node18   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-indexnode-54c98bd6ff5h6hg   1/1     Running     3 (3h56m ago)     3h57m   10.104.20.209   4am-node22   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-indexnode-54c98bd6ffb9q7c   1/1     Running     3 (3h56m ago)     3h57m   10.104.27.154   4am-node31   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-indexnode-54c98bd6ffdsx6d   1/1     Running     3 (3h56m ago)     3h57m   10.104.5.119    4am-node12   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-indexnode-54c98bd6ffzhtsf   1/1     Running     3 (3h56m ago)     3h57m   10.104.32.246   4am-node39   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-mixcoord-6cf5cb964-qb7fm    1/1     Running     3 (3h56m ago)     3h57m   10.104.5.117    4am-node12   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-proxy-b54ccbd6d-qwp5v       1/1     Running     3 (3h56m ago)     3h57m   10.104.34.219   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-querynode-7c7b866c77pfpfp   1/1     Running     5 (66m ago)       3h57m   10.104.24.28    4am-node29   <none>           <none>
inverted-corn-114800-8-20-2243-milvus-querynode-7c7b866c77t8zmr   1/1     Running     3 (3h56m ago)     3h57m   10.104.23.96    4am-node27   <none>           <none>
inverted-corn-114800-8-20-2243-minio-0                            1/1     Running     0                 3h57m   10.104.18.11    4am-node25   <none>           <none>
inverted-corn-114800-8-20-2243-minio-1                            1/1     Running     0                 3h57m   10.104.33.76    4am-node36   <none>           <none>
inverted-corn-114800-8-20-2243-minio-2                            1/1     Running     0                 3h57m   10.104.34.235   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-minio-3                            1/1     Running     0                 3h57m   10.104.26.152   4am-node32   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-bookie-0                    1/1     Running     0                 3h57m   10.104.33.78    4am-node36   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-bookie-1                    1/1     Running     0                 3h57m   10.104.34.236   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-bookie-2                    1/1     Running     0                 3h57m   10.104.18.14    4am-node25   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-bookie-init-kclkn           0/1     Completed   0                 3h57m   10.104.34.218   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-broker-0                    1/1     Running     0                 3h57m   10.104.18.2     4am-node25   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-proxy-0                     1/1     Running     0                 3h57m   10.104.34.220   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-pulsar-init-9rjbc           0/1     Completed   0                 3h57m   10.104.5.118    4am-node12   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-recovery-0                  1/1     Running     0                 3h57m   10.104.26.139   4am-node32   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-zookeeper-0                 1/1     Running     0                 3h57m   10.104.34.234   4am-node37   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-zookeeper-1                 1/1     Running     0                 3h55m   10.104.20.229   4am-node22   <none>           <none>
inverted-corn-114800-8-20-2243-pulsar-zookeeper-2                 1/1     Running     0                 3h55m   10.104.26.159   4am-node32   <none>           <none>

queryNode restart

2024-08-18 22:24:00.528 | stderr | Welcome to use Milvus! |  
-- | -- | -- | --
 2024-08-18 23:55:35.610 | stderr | Welcome to use Milvus!

inverted-corn-114800-8-20-2243-milvus-querynode-7c7b866c77pfpfp_restart.log

client pod name: inverted-corn-1724014800-823447725
fouram_log_err.log

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `vector: memory and disk index`
            verify concurrent DML & DQL scenario which has 4 float_vector fields & 16 scalar fields

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 200dim,
                'float_vector_3': 200dim,
                'int8_1', 'int16_1', 'int32_1', 'int64_1', 'double_1', 'float_1', 'varchar_1', 'bool_1',
                'int8_2', 'int16_2', 'int32_2', 'int64_2', 'double_2', 'float_2', 'varchar_2', 'bool_2'
            2. build indexes:
                HNSW: 'float_vector'
                DIAKANN_IP: 'float_vector_1'
                HNSW: 'float_vector_2'
                DIAKANN_L2: 'float_vector_3'
                scalar_default_index: 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'double_1', 'float_1', 'varchar_1'
                scalar_INVERTED_index: 'int8_2', 'int16_2', 'int32_2', 'int64_2', 'double_2', 'float_2', 'varchar_2', 'bool_2'
            3. insert 5 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - insert
                - delete
                - flush
                - load
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

test result:

[2024-08-19 01:00:43,553 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-08-19 01:00:43,554 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: grpc     delete                                                                          3779     0(0.00%) |    103       3   28006      7 |    0.35        0.00 (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: grpc     flush                                                                           3924     0(0.00%) |  46748     433  199848   4800 |    0.36        0.00 (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: grpc     hybrid_search                                                                   3831    42(1.10%) |   2023       2   50420    220 |    0.35        0.00 (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: grpc     insert                                                                          3873     0(0.00%) |    115       8   28651     14 |    0.36        0.00 (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: grpc     load                                                                            3786     0(0.00%) |    238       7   34464     60 |    0.35        0.00 (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: grpc     query                                                                           3807    32(0.84%) |   1856       1   45207    270 |    0.35        0.00 (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: grpc     search                                                                          3765    32(0.85%) |   3668      78   53320   1100 |    0.35        0.00 (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]:          Aggregated                                                                     26765   106(0.40%) |   7988       1  199848    230 |    2.48        0.01 (stats.py:789)
[2024-08-19 01:00:43,554 -  INFO - fouram]:  (stats.py:790)
[2024-08-19 01:00:43,558 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_8c16m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '16.0',
                                                              'memory': '64Gi'},
                                                   'requests': {'cpu': '9.0',
                                                                'memory': '33Gi'}},
                                     'replicas': 2},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '16Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '9Gi'}},
                                     'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '8.0',
                                                             'memory': '16Gi'},
                                                  'requests': {'cpu': '5.0',
                                                               'memory': '9Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'extraConfigFiles': {'user.yaml': 'queryNode:\n'
                                                         '  mmap:\n'
                                                         '    mmapEnabled: '
                                                         'true\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.4-20240818-248a6ea4-amd64'}}},
            'host': 'inverted-corn-114800-8-20-2243-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_inverted_locust_hnsw_diskann_dml_dql_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'int8_1': {},
                                                                      'int16_1': {},
                                                                      'int32_1': {},
                                                                      'int64_1': {},
                                                                      'double_1': {},
                                                                      'float_1': {},
                                                                      'varchar_1': {},
                                                                      'int8_2': {'index_type': 'INVERTED'},
                                                                      'int16_2': {'index_type': 'INVERTED'},
                                                                      'int32_2': {'index_type': 'INVERTED'},
                                                                      'int64_2': {'index_type': 'INVERTED'},
                                                                      'double_2': {'index_type': 'INVERTED'},
                                                                      'float_2': {'index_type': 'INVERTED'},
                                                                      'varchar_2': {'index_type': 'INVERTED'},
                                                                      'bool_2': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'IP'},
                                                                      'float_vector_2': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_3': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_2': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img'}},
                                                                       'float_vector_3': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img'}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 5000000,
                                                    'ni_per': 5000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'int8_1',
                                                                        'int16_1',
                                                                        'int32_1',
                                                                        'int64_1',
                                                                        'double_1',
                                                                        'float_1',
                                                                        'varchar_1',
                                                                        'bool_1',
                                                                        'int8_2',
                                                                        'int16_2',
                                                                        'int32_2',
                                                                        'int64_2',
                                                                        'double_2',
                                                                        'float_2',
                                                                        'varchar_2',
                                                                        'bool_2'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '3h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 10,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 5000000}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'delete_length': 9,
                                                                  'timeout': 30}},
                                                      {'type': 'flush',
                                                       'weight': 1,
                                                       'params': {'timeout': 180,
                                                                  'check_task': 'check_ignore_rate_limit'}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 180}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'ef': 64},
                                                                  'expr': 'int64_1 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&& '
                                                                          'id '
                                                                          '> '
                                                                          '-1',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 180,
                                                                  'random_data': True}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 10,
                                                                  'reqs': [{'search_param': {'ef': 1280},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'int64_1 '
                                                                                    '< '
                                                                                    '100000 '
                                                                                    '&& '
                                                                                    'float_2 '
                                                                                    '> '
                                                                                    '10.0',
                                                                            'top_k': 1000},
                                                                           {'search_param': {'search_list': 30},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'varchar_1 '
                                                                                    'like '
                                                                                    '"0%" '
                                                                                    '&& '
                                                                                    'bool_2 '
                                                                                    '== '
                                                                                    'True'},
                                                                           {'search_param': {'ef': 1024},
                                                                            'anns_field': 'float_vector_2',
                                                                            'expr': 'int8_1 '
                                                                                    '< '
                                                                                    '64 '
                                                                                    '&& '
                                                                                    'bool_1 '
                                                                                    '== '
                                                                                    'False',
                                                                            'top_k': 1009},
                                                                           {'search_param': {'search_list': 40},
                                                                            'anns_field': 'float_vector_3',
                                                                            'expr': 'int8_2 '
                                                                                    '> '
                                                                                    '64 '
                                                                                    '|| '
                                                                                    'double_2 '
                                                                                    '> '
                                                                                    '1000000.0'}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 60,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&&  '
                                                                          'int64_2 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&& ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 180,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [2500000.0,
                                                                                   5000000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}}]},
            'run_id': 2024081850509939,
            'datetime': '2024-08-18 21:04:10.308699',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 847.5802,
                                      'float_vector_1': {'RT': 703.0809},
                                      'float_vector_2': {'RT': 331.2981},
                                      'float_vector_3': {'RT': 143.9072},
                                      'int8_1': {'RT': 0.5159},
                                      'int16_1': {'RT': 0.5161},
                                      'int32_1': {'RT': 0.5162},
                                      'int64_1': {'RT': 0.5171},
                                      'double_1': {'RT': 0.5167},
                                      'float_1': {'RT': 0.5158},
                                      'varchar_1': {'RT': 0.5157},
                                      'int8_2': {'RT': 0.5149},
                                      'int16_2': {'RT': 0.5155},
                                      'int32_2': {'RT': 0.5136},
                                      'int64_2': {'RT': 0.5166},
                                      'double_2': {'RT': 0.5178},
                                      'float_2': {'RT': 0.5157},
                                      'varchar_2': {'RT': 0.5155},
                                      'bool_2': {'RT': 0.5166}},
                            'insert': {'total_time': 910.8198,
                                       'VPS': 5489.5601,
                                       'batch_time': 0.9108,
                                       'batch': 5000},
                            'flush': {'RT': 3.0248},
                            'load': {'RT': 14.9941},
                            'Locust': {'Aggregated': {'Requests': 26765,
                                                      'Fails': 106,
                                                      'RPS': 2.48,
                                                      'fail_s': 0.0,
                                                      'RT_max': 199848.69,
                                                      'RT_avg': 7988.65,
                                                      'TP50': 230.0,
                                                      'TP99': 181000.0},
                                       'delete': {'Requests': 3779,
                                                  'Fails': 0,
                                                  'RPS': 0.35,
                                                  'fail_s': 0.0,
                                                  'RT_max': 28006.28,
                                                  'RT_avg': 103.09,
                                                  'TP50': 7,
                                                  'TP99': 890.0},
                                       'flush': {'Requests': 3924,
                                                 'Fails': 0,
                                                 'RPS': 0.36,
                                                 'fail_s': 0.0,
                                                 'RT_max': 199848.69,
                                                 'RT_avg': 46748.88,
                                                 'TP50': 4900.0,
                                                 'TP99': 182000.0},
                                       'hybrid_search': {'Requests': 3831,
                                                         'Fails': 42,
                                                         'RPS': 0.35,
                                                         'fail_s': 0.01,
                                                         'RT_max': 50420.15,
                                                         'RT_avg': 2023.06,
                                                         'TP50': 220.0,
                                                         'TP99': 26000.0},
                                       'insert': {'Requests': 3873,
                                                  'Fails': 0,
                                                  'RPS': 0.36,
                                                  'fail_s': 0.0,
                                                  'RT_max': 28651.35,
                                                  'RT_avg': 115.87,
                                                  'TP50': 14,
                                                  'TP99': 1100.0},
                                       'load': {'Requests': 3786,
                                                'Fails': 0,
                                                'RPS': 0.35,
                                                'fail_s': 0.0,
                                                'RT_max': 34464.73,
                                                'RT_avg': 238.49,
                                                'TP50': 60,
                                                'TP99': 4200.0},
                                       'query': {'Requests': 3807,
                                                 'Fails': 32,
                                                 'RPS': 0.35,
                                                 'fail_s': 0.01,
                                                 'RT_max': 45207.93,
                                                 'RT_avg': 1856.67,
                                                 'TP50': 270.0,
                                                 'TP99': 27000.0},
                                       'search': {'Requests': 3765,
                                                  'Fails': 32,
                                                  'RPS': 0.35,
                                                  'fail_s': 0.01,
                                                  'RT_max': 53320.5,
                                                  'RT_avg': 3668.97,
                                                  'TP50': 1100.0,
                                                  'TP99': 37000.0}}}}}

The text was updated successfully, but these errors were encountered:

yanliang567 · 2024-08-19T06:03:25Z

/assign @zhagnlu
/unassign

wangting0128 · 2024-08-20T09:08:22Z

/assign @MrPresent-Han
/unassign @zhagnlu

xiaofan-luan · 2024-12-14T10:36:45Z

is this still an issue?

wangting0128 · 2024-12-16T03:23:02Z

is this still an issue?

not reproduce recently，maybe close it first

wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Aug 19, 2024

wangting0128 added this to the 2.4.8 milestone Aug 19, 2024

wangting0128 assigned yanliang567 Aug 19, 2024

sre-ci-robot assigned zhagnlu and unassigned yanliang567 Aug 19, 2024

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 19, 2024

yanliang567 modified the milestones: 2.4.8, 2.4.10 Aug 19, 2024

sre-ci-robot assigned MrPresent-Han and unassigned zhagnlu Aug 20, 2024

yanliang567 modified the milestones: 2.4.10, 2.4.11 Sep 5, 2024

yanliang567 modified the milestones: 2.4.11, 2.4.12 Sep 18, 2024

yanliang567 modified the milestones: 2.4.12, 2.4.13 Sep 27, 2024

yanliang567 modified the milestones: 2.4.13, 2.4.14 Oct 15, 2024

yanliang567 modified the milestones: 2.4.14, 2.4.16 Nov 14, 2024

yanliang567 modified the milestones: 2.4.16, 2.4.17, 2.4.18 Nov 21, 2024

yanliang567 modified the milestones: 2.4.18, 2.4.19, 2.4.20 Dec 24, 2024

yanliang567 modified the milestones: 2.4.20, 2.4.21 Jan 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: [benchmark][cluster] queryNode restarts twice during concurrent dql & dml scene #35547

[Bug]: [benchmark][cluster] queryNode restarts twice during concurrent dql & dml scene #35547

wangting0128 commented Aug 19, 2024

yanliang567 commented Aug 19, 2024

wangting0128 commented Aug 20, 2024

xiaofan-luan commented Dec 14, 2024

wangting0128 commented Dec 16, 2024

[Bug]: [benchmark][cluster] queryNode restarts twice during concurrent dql & dml scene #35547

[Bug]: [benchmark][cluster] queryNode restarts twice during concurrent dql & dml scene #35547

Comments

wangting0128 commented Aug 19, 2024

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

yanliang567 commented Aug 19, 2024

wangting0128 commented Aug 20, 2024

xiaofan-luan commented Dec 14, 2024

wangting0128 commented Dec 16, 2024