Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: search failed with error segment lacks[segment=451679606836035900]: channel not available after standalone pod kill chaos test #35361

Open
1 task done
zhuwenxing opened this issue Aug 8, 2024 · 4 comments
Open
1 task done

[Bug]: search failed with error segment lacks[segment=451679606836035900]: channel not available after standalone pod kill chaos test #35361

zhuwenxing opened this issue Aug 8, 2024 · 4 comments
Assignees
@zhuwenxing
Labels
kind/bug Issues or changes related a bug test/chaos chaos test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.19

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.4-20240807-d14d00b0-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:37 - DEBUG - ci_test]: (api_request)  : [Connections.connect] args: ['default', '', '', 'default', ''], kwargs: {'host': '10.255.181.57', 'port': 19530} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:37 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:37 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:37 - DEBUG - ci_test]: (api_response) : True  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:37 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['Checker__P5GMBjpl', {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType......, kwargs: {'consistency_level': 'Strong'} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:37 - DEBUG - ci_test]: (api_response) : <Collection>:

[2024-08-07T09:13:11.193Z] -------------

[2024-08-07T09:13:11.193Z] <name>: Checker__P5GMBjpl

[2024-08-07T09:13:11.193Z] <description>: 

[2024-08-07T09:13:11.193Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}......  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:37 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:38 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:38 - DEBUG - ci_test]: (api_request)  : [Collection.compact] args: [180], kwargs: {} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:38 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:39 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:49 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:49 - INFO - ci_test]: assert create collection: 0.34676313400268555, init_entities: 136995 (test_all_collections_after_chaos.py:49)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:50 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [[[-3000, -2999, -2998, -2997, -2996, -2995, -2994, -2993, -2992, -2991, -2990, -2989, -2988, -2987, -2986, -2985, -2984, -2983, -2982, -2981, -2980, -2979, -2978, -2977, -2976, -2975, -2974, -2973, -2972, -2971, -2970, -2969, -2968, -2967, -2966, -2965, -2964, -2963, -2962, -2961, -2960, -2959, -29......, kwargs: {'timeout': 180} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:51 - DEBUG - ci_test]: (api_response) : (insert count: 2000, delete count: 0, upsert count: 0, timestamp: 451679840116670466, success count: 2000, err count: 0  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:51 - INFO - ci_test]: assert insert: 1.272249460220337 (test_all_collections_after_chaos.py:57)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:51 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:59 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:10:59 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - INFO - ci_test]: assert flush: 7.671832799911499, entities: 138995 (test_all_collections_after_chaos.py:67)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - INFO - ci_test]: index info: [{'collection': 'Checker__P5GMBjpl', 'field': 'float_vector', 'index_name': 'float_vector', 'index_param': {'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}, 'index_type': 'HNSW'}}, {'collection': 'Checker__P5GMBjpl', 'field': 'image_emb', 'index_name': 'image_emb', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}, {'collection': 'Checker__P5GMBjpl', 'field': 'text_emb', 'index_name': 'text_emb', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}, {'collection': 'Checker__P5GMBjpl', 'field': 'voice_emb', 'index_name': 'voice_emb', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}, {'collection': 'Checker__P5GMBjpl', 'field': 'int64', 'index_name': 'int64', 'index_param': {'index_type': 'INVERTED'}}, {'collection': 'Checker__P5GMBjpl', 'field': 'float', 'index_name': 'float', 'index_param': {'index_type': 'INVERTED'}}, {'collection': 'Checker__P5GMBjpl', 'field': 'varchar', 'index_name': 'varchar', 'index_param': {'index_type': 'INVERTED'}}] (test_all_collections_after_chaos.py:71)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - INFO - ci_test]: index info: [{'collection': 'Checker__P5GMBjpl', 'field': 'voice_emb', 'index_name': 'voice_emb', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}, {'collection': 'Checker__P5GMBjpl', 'field': 'int64', 'index_name': 'int64', 'index_param': {'index_type': 'INVERTED'}}, {'collection': 'Checker__P5GMBjpl', 'field': 'float', 'index_name': 'float', 'index_param': {'index_type': 'INVERTED'}}, {'collection': 'Checker__P5GMBjpl', 'field': 'varchar', 'index_name': 'varchar', 'index_param': {'index_type': 'INVERTED'}}, {'collection': 'Checker__P5GMBjpl', 'field': 'float_vector', 'index_name': 'float_vector', 'index_param': {'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}, 'index_type': 'HNSW'}}, {'collection': 'Checker__P5GMBjpl', 'field': 'image_emb', 'index_name': 'image_emb', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}, {'collection': 'Checker__P5GMBjpl', 'field': 'text_emb', 'index_name': 'text_emb', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}] (test_all_collections_after_chaos.py:86)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.10321139337677954, 0.12237845132389466, 0.0008106671732834907, 0.07056239880516353, 0.08767849430454548, 0.0672583600964388, 0.023293687165044725, 0.0807304363631584, 0.008693150625500795, 0.03487908435206497, 0.11515404612929885, 0.14392505232247643, 0.06795949935007947, 0.024356334911214687, ......, kwargs: {} (api_request.py:62)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=503, message=failed to search: segment lacks[segment=451679606836035900]: channel not available[channel=by-dev-rootcoord-dml_7_451679330252759529v0])>, <Time:{'RPC start': '2024-08-07 09:11:09.288978', 'RPC error': '2024-08-07 09:11:09.291185'}> (decorators.py:146)

[2024-08-07T09:13:11.193Z] [2024-08-07 09:11:09 - ERROR - ci_test]: Traceback (most recent call last):

[2024-08-07T09:13:11.193Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2024-08-07T09:13:11.193Z]     res = func(*args, **_kwargs)

[2024-08-07T09:13:11.193Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2024-08-07T09:13:11.193Z]     return func(*arg, **kwargs)

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 801, in search

[2024-08-07T09:13:11.193Z]     resp = conn.search(

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 147, in handler

[2024-08-07T09:13:11.193Z]     raise e from e

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 143, in handler

[2024-08-07T09:13:11.193Z]     return func(*args, **kwargs)

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 182, in handler

[2024-08-07T09:13:11.193Z]     return func(self, *args, **kwargs)

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 122, in handler

[2024-08-07T09:13:11.193Z]     raise e from e

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 87, in handler

[2024-08-07T09:13:11.193Z]     return func(*args, **kwargs)

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 801, in search

[2024-08-07T09:13:11.193Z]     return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 742, in _execute_search

[2024-08-07T09:13:11.193Z]     raise e from e

[2024-08-07T09:13:11.193Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 735, in _execute_search

[2024-08-07T09:13:11.193Z]     check_status(response.status)

[2024-08-07T09:13:11.194Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/utils.py", line 63, in check_status

[2024-08-07T09:13:11.194Z]     raise MilvusException(status.code, status.reason, status.error_code)

[2024-08-07T09:13:11.194Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=failed to search: segment lacks[segment=451679606836035900]: channel not available[channel=by-dev-rootcoord-dml_7_451679330252759529v0])>

[2024-08-07T09:13:11.194Z]  (api_request.py:45)

[2024-08-07T09:13:11.194Z] [2024-08-07 09:11:09 - ERROR - ci_test]: (api_response) : <MilvusException: (code=503, message=failed to search: segment lacks[segment=451679606836035900]: channel not available[channel=by-dev-rootcoord-dml_7_451679330252759529v0])> (api_request.py:46)

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-for-release-cron/detail/chaos-test-kafka-for-release-cron/15665/pipeline
log:
artifacts-standalone-pod-kill-15665-server-logs.tar.gz

cluster: 4am
ns:chaos-testing
pod

[2024-08-07T09:09:15.217Z] + grep standalone-pod-kill-15665

[2024-08-07T09:09:15.472Z] standalone-pod-kill-15665-etcd-0                                  1/1     Running       0               32m     10.104.34.193   4am-node37   <none>           <none>

[2024-08-07T09:09:15.472Z] standalone-pod-kill-15665-etcd-1                                  1/1     Running       0               32m     10.104.24.148   4am-node29   <none>           <none>

[2024-08-07T09:09:15.472Z] standalone-pod-kill-15665-etcd-2                                  1/1     Running       0               32m     10.104.26.188   4am-node32   <none>           <none>

[2024-08-07T09:09:15.472Z] standalone-pod-kill-15665-kafka-0                                 2/2     Running       2 (31m ago)     32m     10.104.24.147   4am-node29   <none>           <none>

[2024-08-07T09:09:15.472Z] standalone-pod-kill-15665-kafka-1                                 2/2     Running       1 (31m ago)     32m     10.104.23.72    4am-node27   <none>           <none>

[2024-08-07T09:09:15.472Z] standalone-pod-kill-15665-kafka-2                                 2/2     Running       1 (31m ago)     32m     10.104.26.190   4am-node32   <none>           <none>

[2024-08-07T09:09:15.472Z] standalone-pod-kill-15665-kafka-exporter-5788cd868d-wqbks         1/1     Running       4 (31m ago)     32m     10.104.33.65    4am-node36   <none>           <none>

[2024-08-07T09:09:15.472Z] standalone-pod-kill-15665-milvus-standalone-655bdcc4d7-gdsxg      1/1     Running       0               9m14s   10.104.33.78    4am-node36   <none>           <none>

[2024-08-07T09:09:15.473Z] standalone-pod-kill-15665-minio-747dd9479d-d7r6l                  1/1     Running       0               32m     10.104.33.69    4am-node36   <none>           <none>

[2024-08-07T09:09:15.473Z] standalone-pod-kill-15665-zookeeper-0                             1/1     Running       0               32m     10.104.32.92    4am-node39   <none>           <none>

[2024-08-07T09:09:15.473Z] standalone-pod-kill-15665-zookeeper-1                             1/1     Running       0               32m     10.104.23.73    4am-node27   <none>           <none>

[2024-08-07T09:09:15.473Z] standalone-pod-kill-15665-zookeeper-2                             1/1     Running       0               32m     10.104.26.191   4am-node32   <none>           <none>

Anything else?

No response
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 8, 2024
@zhuwenxing zhuwenxing added this to the 2.4.7 milestone Aug 8, 2024
@zhuwenxing zhuwenxing added the test/chaos chaos test label Aug 8, 2024
@zhuwenxing
Copy link
Contributor Author

/assign @weiliu1031
PTAL

@zhuwenxing
Copy link
Contributor Author

not a stable reproduced issue
success retry: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-for-release-cron/detail/chaos-test-kafka-for-release-cron/15693/pipeline

This was referenced Aug 8, 2024
Merged
Merged
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 8, 2024
@yanliang567 yanliang567 removed their assignment Aug 8, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.7, 2.4.8 Aug 12, 2024
sre-ci-robot pushed a commit that referenced this issue Aug 13, 2024
@weiliu1031
fix: force update next target if target can't be loaded (#35366)
b316040 
issue: #35361
pr: #35365

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Aug 15, 2024
@weiliu1031
fix: force update next target if target can't be loaded (#35365)
f6aaf3f 
issue: #35361

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
@yanliang567 yanliang567 modified the milestones: 2.4.8, 2.4.10 Aug 19, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.10, 2.4.11 Sep 5, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.11, 2.4.12 Sep 18, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.12, 2.4.13 Sep 27, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.13, 2.4.14 Oct 15, 2024
@weiliu1031
Copy link
Contributor

please verify this with latest image

@weiliu1031
Copy link
Contributor

/assign @zhuwenxing

@yanliang567 yanliang567 modified the milestones: 2.4.14, 2.4.16 Nov 14, 2024
@zhuwenxing zhuwenxing reopened this Nov 16, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.16, 2.4.17, 2.4.18 Nov 21, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.18, 2.4.19 Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhuwenxing zhuwenxing

Labels
kind/bug Issues or changes related a bug test/chaos chaos test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.19
Development

No branches or pull requests
3 participants
@zhuwenxing @yanliang567 @weiliu1031