[Bug]: milvus-standalone always restart #38513

ahuai · 2024-12-17T04:12:35Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version:milvusdb/milvus:v2.5.0-beta
- Deployment mode(standalone or cluster):standalone 
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 2核(vCPU) 8 GiB
- GPU: 
- Others:

Current Behavior

I used docker-compose to start Milvus.
Currently, the running logs keep reporting errors. How can I fix it?
It has been running normally since I started it 2 months ago, but recently it has been frequently restarting.
I upgraded the Milvus version to image: milvusdb/milvus:v2.5.0-beta.
This did not fix the problem, and it is still frequently restarting.

docker-compose.yml
`
version: '3.5'

services:
etcd:
container_name: milvus-etcd-prod
image: quay.io/coreos/etcd:v3.5.5
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health"]
interval: 30s
timeout: 20s
retries: 3
restart: always
deploy:
resources:
limits:
cpus: '0.5'
memory: '512M'

minio:
container_name: milvus-minio-prod
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
# ports:
# - "9001:9001"
# - "9000:9000"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
restart: always
deploy:
resources:
limits:
cpus: '0.5'
memory: '512M'

standalone:
container_name: milvus-standalone-prod
image: milvusdb/milvus:v2.5.0-beta
logging:
driver: "json-file"
options:
max-size: "100m"
max-file: "3"
command: ["milvus", "run", "standalone"]
security_opt:
- seccomp:unconfined
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
interval: 30s
start_period: 300s
timeout: 20s
retries: 3
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"
restart: always
deploy:
resources:
limits:
cpus: '2.0'
memory: '6G'

networks:
default:
name: milvus_net
`

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

[2024/12/17 04:07:00.776 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:00.776 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:00.776 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:00.818 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=13]
[2024/12/17 04:07:00.818 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=454664283063162562] ["task type"=JobTypeStatsJob] [state=JobStateInProgress]
[2024/12/17 04:07:00.819 +00:00] [INFO] [datacoord/task_stats.go:255] ["query stats task result success"] [taskID=454664283063162562] [segmentID=452745552034354339] ["result state"=JobStateInProgress] [failReason=]
[2024/12/17 04:07:00.819 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=454664283063166178] ["task type"=JobTypeStatsJob] [state=JobStateInit]
[2024/12/17 04:07:00.819 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=f92810679b9ff7645b5a56c1ae818d04] [unissued=0] [active=1] [slot=0]
[2024/12/17 04:07:00.820 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, wait a minute..."]
[2024/12/17 04:07:00.975 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361283] [subChannelCount=1] [loadSegmentCount=0]
[2024/12/17 04:07:00.975 +00:00] [WARN] [observers/collection_observer.go:350] ["failed to manual check current target, skip update load status"] [collectionID=449976277547361282] [partitionID=449976277547361283]
[2024/12/17 04:07:00.975 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361294] [subChannelCount=1] [loadSegmentCount=3]
[2024/12/17 04:07:00.975 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277507568832] [replicaID=449976289146175490] [nodes="["{NodeID: 5864, AssignedScore: 66085.000000, CurrentScore: 72693.500000, Priority: 6608}"]"]
[2024/12/17 04:07:00.975 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277526694406] [replicaID=449976289146175499] [nodes="["{NodeID: 5864, AssignedScore: 66314.000000, CurrentScore: 72945.400000, Priority: 6631}"]"]
[2024/12/17 04:07:00.976 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361282] [replicaID=449976289146175500] [nodes="["{NodeID: 5864, AssignedScore: 491811.000000, CurrentScore: 540992.100000, Priority: 49181}"]"]
[2024/12/17 04:07:00.976 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361318] [replicaID=449976289146175501] [nodes="["{NodeID: 5864, AssignedScore: 491280.000000, CurrentScore: 540408.000000, Priority: 49128}"]"]
[2024/12/17 04:07:00.976 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=452745552034264114] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:00.976 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454245220196488370] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:00.976 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:00.976 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:00.976 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:01.175 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361294] [subChannelCount=1] [loadSegmentCount=3]
[2024/12/17 04:07:01.175 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361283] [subChannelCount=1] [loadSegmentCount=0]
[2024/12/17 04:07:01.175 +00:00] [WARN] [observers/collection_observer.go:350] ["failed to manual check current target, skip update load status"] [collectionID=449976277547361282] [partitionID=449976277547361283]
[2024/12/17 04:07:01.175 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277507568832] [replicaID=449976289146175490] [nodes="["{NodeID: 5864, AssignedScore: 66085.000000, CurrentScore: 72693.500000, Priority: 6608}"]"]
[2024/12/17 04:07:01.175 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277526694406] [replicaID=449976289146175499] [nodes="["{NodeID: 5864, AssignedScore: 66314.000000, CurrentScore: 72945.400000, Priority: 6631}"]"]
[2024/12/17 04:07:01.176 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361282] [replicaID=449976289146175500] [nodes="["{NodeID: 5864, AssignedScore: 491811.000000, CurrentScore: 540992.100000, Priority: 49181}"]"]
[2024/12/17 04:07:01.176 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361318] [replicaID=449976289146175501] [nodes="["{NodeID: 5864, AssignedScore: 491280.000000, CurrentScore: 540408.000000, Priority: 49128}"]"]
[2024/12/17 04:07:01.176 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=452745552034264114] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:01.176 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454245220196488370] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:01.176 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:01.176 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:01.176 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
I20241217 04:07:01.215241 169 ChunkedSegmentSealedImpl.cpp:287] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 loads field 1 mmap false done
[2024/12/17 04:07:01.215 +00:00] [INFO] [segments/segment.go:977] ["load field done"] [traceID=752dd42748c2d90d0f502bcc702384b0] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=452745552034264114] [fieldID=1] [rowCount=89288]
[2024/12/17 04:07:01.215 +00:00] [INFO] [segments/segment.go:965] ["submitted loadFieldData task to load pool"] [traceID=752dd42748c2d90d0f502bcc702384b0] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=452745552034264114] [fieldID=108] [rowCount=89288]
I20241217 04:07:01.216002 169 ChunkedSegmentSealedImpl.cpp:262] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 loads field 108 with num_rows 89288
I20241217 04:07:01.216042 169 ChunkedSegmentSealedImpl.cpp:276] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 submits load field 108 task to thread pool
[2024/12/17 04:07:02.065 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=13]
[2024/12/17 04:07:02.065 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=454664283063162562] ["task type"=JobTypeStatsJob] [state=JobStateInProgress]
[2024/12/17 04:07:01.904 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361283] [subChannelCount=1] [loadSegmentCount=0]
[2024/12/17 04:07:02.102 +00:00] [WARN] [observers/collection_observer.go:350] ["failed to manual check current target, skip update load status"] [collectionID=449976277547361282] [partitionID=449976277547361283]
[2024/12/17 04:07:02.102 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361294] [subChannelCount=1] [loadSegmentCount=3]
[2024/12/17 04:07:02.102 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361283] [subChannelCount=1] [loadSegmentCount=0]
[2024/12/17 04:07:02.102 +00:00] [WARN] [observers/collection_observer.go:350] ["failed to manual check current target, skip update load status"] [collectionID=449976277547361282] [partitionID=449976277547361283]
[2024/12/17 04:07:02.102 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361294] [subChannelCount=1] [loadSegmentCount=3]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=452745552034264114] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454245220196488370] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=452745552034264114] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454245220196488370] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.104 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277507568832] [replicaID=449976289146175490] [nodes="["{NodeID: 5864, AssignedScore: 66085.000000, CurrentScore: 72693.500000, Priority: 6608}"]"]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277526694406] [replicaID=449976289146175499] [nodes="["{NodeID: 5864, AssignedScore: 66314.000000, CurrentScore: 72945.400000, Priority: 6631}"]"]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361282] [replicaID=449976289146175500] [nodes="["{NodeID: 5864, AssignedScore: 491811.000000, CurrentScore: 540992.100000, Priority: 49181}"]"]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361318] [replicaID=449976289146175501] [nodes="["{NodeID: 5864, AssignedScore: 491280.000000, CurrentScore: 540408.000000, Priority: 49128}"]"]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277507568832] [replicaID=449976289146175490] [nodes="["{NodeID: 5864, AssignedScore: 66085.000000, CurrentScore: 72693.500000, Priority: 6608}"]"]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277526694406] [replicaID=449976289146175499] [nodes="["{NodeID: 5864, AssignedScore: 66314.000000, CurrentScore: 72945.400000, Priority: 6631}"]"]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361282] [replicaID=449976289146175500] [nodes="["{NodeID: 5864, AssignedScore: 491811.000000, CurrentScore: 540992.100000, Priority: 49181}"]"]
[2024/12/17 04:07:02.105 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361318] [replicaID=449976289146175501] [nodes="["{NodeID: 5864, AssignedScore: 491280.000000, CurrentScore: 540408.000000, Priority: 49128}"]"]
[2024/12/17 04:07:02.146 +00:00] [INFO] [dist/dist_handler.go:260] ["leader is not available due to target version not ready"] [collectionID=449976277547361282] [nodeID=5864] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [error="service internal error: target version mismatch, collection: 449976277547361282, channel: by-dev-rootcoord-dml_11_449976277547361282v0, current target version: 0, leader version: 0"]
I20241217 04:07:02.152447 164 ChunkedSegmentSealedImpl.cpp:287] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 loads field 101 mmap false done
[2024/12/17 04:07:02.163 +00:00] [INFO] [task/scheduler.go:688] ["processed tasks"] [nodeID=5864] [toProcessNum=5] [committedNum=0] [toRemoveNum=0]
[2024/12/17 04:07:02.163 +00:00] [INFO] [task/scheduler.go:694] ["process tasks related to node done"] [nodeID=5864] [processingTaskNum=5] [waitingTaskNum=0] [segmentTaskNum=5] [channelTaskNum=0]
[2024/12/17 04:07:02.165 +00:00] [INFO] [segments/segment.go:965] ["submitted loadFieldData task to load pool"] [traceID=752dd42748c2d90d0f502bcc702384b0] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=452745552034264114] [fieldID=104] [rowCount=89288]
I20241217 04:07:02.165231 164 ChunkedSegmentSealedImpl.cpp:262] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 loads field 104 with num_rows 89288
I20241217 04:07:02.165284 164 ChunkedSegmentSealedImpl.cpp:276] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 submits load field 104 task to thread pool
[2024/12/17 04:07:02.177 +00:00] [INFO] [segments/segment.go:977] ["load field done"] [traceID=752dd42748c2d90d0f502bcc702384b0] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=452745552034264114] [fieldID=101] [rowCount=89288]
I20241217 04:07:02.206741 170 ChunkedSegmentSealedImpl.cpp:287] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 loads field 100 mmap false done
[2024/12/17 04:07:02.207 +00:00] [INFO] [segments/segment.go:1028] ["add field data info done"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] ["row count"=1491]
[2024/12/17 04:07:02.207 +00:00] [INFO] [segments/segment_loader.go:760] ["Start loading fields..."] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [segmentID=454575894751547022] [indexedFields="[]"] ["indexed text fields"="[]"] ["unindexed text fields"="[]"]
[2024/12/17 04:07:02.207 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=109] [rowCount=1491]
[2024/12/17 04:07:02.208 +00:00] [INFO] [segments/segment.go:965] ["submitted loadFieldData task to load pool"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=109] [rowCount=1491]
I20241217 04:07:02.208029 170 ChunkedSegmentSealedImpl.cpp:262] [SERVER][LoadFieldData][milvus][]segment 454575894751547022 loads field 109 with num_rows 1491
I20241217 04:07:02.208061 170 ChunkedSegmentSealedImpl.cpp:276] [SERVER][LoadFieldData][milvus][]segment 454575894751547022 submits load field 109 task to thread pool
[2024/12/17 04:07:02.214 +00:00] [INFO] [syncmgr/meta_writer.go:75] [SaveBinlogPath] [SegmentID=454664283061949128] [CollectionID=449976277547361282] [ParitionID=449976277547361294] [startPos="[{"start_position":{"channel_name":"by-dev-rootcoord-dml_11_449976277547361282v0","msgID":"5gP8x/42TwY=","msgGroup":"datanode-5826-by-dev-rootcoord-dml_11_449976277547361282v0-true","timestamp":454663099801665537},"segmentID":454642555292159502}]"] [checkPoints="[{"segmentID":454664283061949128,"position":{"channel_name":"by-dev-rootcoord-dml_11_449976277547361282v0","msgID":"vW/9x/42TwY=","msgGroup":"datanode-5864-by-dev-rootcoord-dml_11_449976277547361282v0-true","timestamp":454664757073674241},"num_of_rows":180}]"] [binlogNum=12] [statslogNum=1] [deltalogNum=0] [bm25logNum=0] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0]
[2024/12/17 04:07:02.218 +00:00] [INFO] [segments/segment.go:977] ["load field done"] [traceID=752dd42748c2d90d0f502bcc702384b0] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=452745552034264114] [fieldID=100] [rowCount=89288]
[2024/12/17 04:07:02.220 +00:00] [INFO] [segments/segment.go:1028] ["add field data info done"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] ["row count"=4104]
[2024/12/17 04:07:02.220 +00:00] [INFO] [segments/segment_loader.go:760] ["Start loading fields..."] [traceID=36a310902545ded443f154948d745524] [segmentID=454664283062150492] [indexedFields="[]"] ["indexed text fields"="[]"] ["unindexed text fields"="[]"]
[2024/12/17 04:07:02.220 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=109] [rowCount=4104]
[2024/12/17 04:07:02.220 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=0] [rowCount=1491]
[2024/12/17 04:07:02.221 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=1] [rowCount=1491]
[2024/12/17 04:07:02.221 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=100] [rowCount=1491]
[2024/12/17 04:07:02.221 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=101] [rowCount=1491]
[2024/12/17 04:07:02.221 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=102] [rowCount=1491]
[2024/12/17 04:07:02.221 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=103] [rowCount=1491]
[2024/12/17 04:07:02.221 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=104] [rowCount=1491]
[2024/12/17 04:07:02.221 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=105] [rowCount=1491]
[2024/12/17 04:07:02.222 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=106] [rowCount=1491]
[2024/12/17 04:07:02.222 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=107] [rowCount=1491]
[2024/12/17 04:07:02.222 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=108] [rowCount=1491]
[2024/12/17 04:07:02.222 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=0] [rowCount=4104]
[2024/12/17 04:07:02.223 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=1] [rowCount=4104]
[2024/12/17 04:07:02.224 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=100] [rowCount=4104]
[2024/12/17 04:07:02.224 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=101] [rowCount=4104]
[2024/12/17 04:07:02.224 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=102] [rowCount=4104]
[2024/12/17 04:07:02.224 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=103] [rowCount=4104]
[2024/12/17 04:07:02.224 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=104] [rowCount=4104]
[2024/12/17 04:07:02.224 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=105] [rowCount=4104]
[2024/12/17 04:07:02.224 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=106] [rowCount=4104]
[2024/12/17 04:07:02.225 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=107] [rowCount=4104]
[2024/12/17 04:07:02.225 +00:00] [INFO] [segments/segment.go:923] ["start loading field data for field"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=108] [rowCount=4104]
[2024/12/17 04:07:02.261 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361294] [subChannelCount=1] [loadSegmentCount=3]
[2024/12/17 04:07:02.261 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361283] [subChannelCount=1] [loadSegmentCount=0]
[2024/12/17 04:07:02.261 +00:00] [WARN] [observers/collection_observer.go:350] ["failed to manual check current target, skip update load status"] [collectionID=449976277547361282] [partitionID=449976277547361283]
[2024/12/17 04:07:02.261 +00:00] [INFO] [datacoord/services.go:494] ["receive SaveBinlogPaths request"] [traceID=daadcaf4ab53e57a352f13024a2e1085] [nodeID=5864] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [collectionID=449976277547361282] [segmentID=454664283061949128] [level=L1] [isFlush=false] [isDropped=false] [checkpoints="[{"segmentID":454664283061949128,"position":{"channel_name":"by-dev-rootcoord-dml_11_449976277547361282v0","msgID":"vW/9x/42TwY=","msgGroup":"datanode-5864-by-dev-rootcoord-dml_11_449976277547361282v0-true","timestamp":454664757073674241},"num_of_rows":180}]"]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277507568832] [replicaID=449976289146175490] [nodes="["{NodeID: 5864, AssignedScore: 66085.000000, CurrentScore: 72693.500000, Priority: 6608}"]"]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277526694406] [replicaID=449976289146175499] [nodes="["{NodeID: 5864, AssignedScore: 66314.000000, CurrentScore: 72945.400000, Priority: 6631}"]"]
[2024/12/17 04:07:02.261 +00:00] [INFO] [datacoord/meta.go:976] ["check point reported inconsistent with bin log row count"] ["current rows (wrong)"=190] ["segment bin log row count (correct)"=180]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361282] [replicaID=449976289146175500] [nodes="["{NodeID: 5864, AssignedScore: 491811.000000, CurrentScore: 540992.100000, Priority: 49181}"]"]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361318] [replicaID=449976289146175501] [nodes="["{NodeID: 5864, AssignedScore: 491280.000000, CurrentScore: 540408.000000, Priority: 49128}"]"]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=452745552034264114] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454245220196488370] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.261 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.269 +00:00] [INFO] [datacoord/task_stats.go:255] ["query stats task result success"] [taskID=454664283063162562] [segmentID=452745552034354339] ["result state"=JobStateInProgress] [failReason=]
[2024/12/17 04:07:02.270 +00:00] [INFO] [datacoord/compaction.go:634] ["compactionHandler cannot find datanode for compaction task"] [planID=454664254582251058] [type=MixCompaction] [vchannel=by-dev-rootcoord-dml_11_449976277547361282v0]
[2024/12/17 04:07:02.272 +00:00] [INFO] [datacoord/services.go:205] ["handle assign segment request"] [traceID=2988fe72c19592f7f9b591c32aa59e80] [collectionID=449976277547361282] [partitionID=449976277547361294] [channelName=by-dev-rootcoord-dml_11_449976277547361282v0] [count=1] ["segment level"=Legacy]
[2024/12/17 04:07:02.276 +00:00] [INFO] [syncmgr/meta_writer.go:75] [SaveBinlogPath] [SegmentID=454642555292159502] [CollectionID=449976277547361282] [ParitionID=449976277547361294] [startPos="[{"start_position":{"channel_name":"by-dev-rootcoord-dml_11_449976277547361282v0","msgID":"5gP8x/42TwY=","msgGroup":"datanode-5826-by-dev-rootcoord-dml_11_449976277547361282v0-true","timestamp":454663099801665537},"segmentID":454642555292159502}]"] [checkPoints="[{"segmentID":454642555292159502,"position":{"channel_name":"by-dev-rootcoord-dml_11_449976277547361282v0","msgID":"vW/9x/42TwY=","msgGroup":"datanode-5864-by-dev-rootcoord-dml_11_449976277547361282v0-true","timestamp":454664757073674241},"num_of_rows":550}]"] [binlogNum=12] [statslogNum=1] [deltalogNum=0] [bm25logNum=0] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0]
[2024/12/17 04:07:02.277 +00:00] [INFO] [datacoord/services.go:494] ["receive SaveBinlogPaths request"] [traceID=515394ece2d9ff7a06aadae297be7d13] [nodeID=5864] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [collectionID=449976277547361282] [segmentID=454642555292159502] [level=L1] [isFlush=false] [isDropped=false] [checkpoints="[{"segmentID":454642555292159502,"position":{"channel_name":"by-dev-rootcoord-dml_11_449976277547361282v0","msgID":"vW/9x/42TwY=","msgGroup":"datanode-5864-by-dev-rootcoord-dml_11_449976277547361282v0-true","timestamp":454664757073674241},"num_of_rows":550}]"]
[2024/12/17 04:07:02.278 +00:00] [INFO] [dist/dist_handler.go:260] ["leader is not available due to target version not ready"] [collectionID=449976277547361282] [nodeID=5864] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [error="service internal error: target version mismatch, collection: 449976277547361282, channel: by-dev-rootcoord-dml_11_449976277547361282v0, current target version: 0, leader version: 0"]
[2024/12/17 04:07:02.279 +00:00] [INFO] [task/scheduler.go:688] ["processed tasks"] [nodeID=5864] [toProcessNum=5] [committedNum=0] [toRemoveNum=0]
[2024/12/17 04:07:02.279 +00:00] [INFO] [task/scheduler.go:694] ["process tasks related to node done"] [nodeID=5864] [processingTaskNum=5] [waitingTaskNum=0] [segmentTaskNum=5] [channelTaskNum=0]
[2024/12/17 04:07:02.282 +00:00] [INFO] [dist/dist_handler.go:260] ["leader is not available due to target version not ready"] [collectionID=449976277547361282] [nodeID=5864] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [error="service internal error: target version mismatch, collection: 449976277547361282, channel: by-dev-rootcoord-dml_11_449976277547361282v0, current target version: 0, leader version: 0"]
[2024/12/17 04:07:02.282 +00:00] [INFO] [task/scheduler.go:688] ["processed tasks"] [nodeID=5864] [toProcessNum=5] [committedNum=0] [toRemoveNum=0]
[2024/12/17 04:07:02.282 +00:00] [INFO] [task/scheduler.go:694] ["process tasks related to node done"] [nodeID=5864] [processingTaskNum=5] [waitingTaskNum=0] [segmentTaskNum=5] [channelTaskNum=0]
[2024/12/17 04:07:02.283 +00:00] [WARN] [rootcoord/quota_center.go:977] ["QuotaCenter: QueryNode memory to low water level, limit writing rate"] [Node=querynode-5864] [collections="[449976277507568832,449976277526694406,449976277547361282,449976277547361318]"] [UsedMem=5953187840] [TotalMem=6442450944] [curWatermark=0.9240563710530599] [lowWatermark=0.85] [highWatermark=0.95]
[2024/12/17 04:07:02.283 +00:00] [WARN] [rootcoord/quota_center.go:1004] ["QuotaCenter: DataNode memory to low water level, limit writing rate"] [Node=datanode-5864] [collections="[449976277526694406,449976277507568832,449976277547361318,449976277547361282]"] [UsedMem=5947514880] [TotalMem=6442450944] [curWatermark=0.9231758117675781] [lowWatermark=0.85] [highWatermark=0.95]
I20241217 04:07:02.312698 162 ChunkedSegmentSealedImpl.cpp:287] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 loads field 102 mmap false done
[2024/12/17 04:07:02.323 +00:00] [INFO] [segments/segment.go:965] ["submitted loadFieldData task to load pool"] [traceID=36a310902545ded443f154948d745524] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664283062150492] [fieldID=109] [rowCount=4104]
I20241217 04:07:02.323127 162 ChunkedSegmentSealedImpl.cpp:262] [SERVER][LoadFieldData][milvus][]segment 454664283062150492 loads field 109 with num_rows 4104
I20241217 04:07:02.323163 162 ChunkedSegmentSealedImpl.cpp:276] [SERVER][LoadFieldData][milvus][]segment 454664283062150492 submits load field 109 task to thread pool
[2024/12/17 04:07:02.344 +00:00] [INFO] [segments/segment.go:977] ["load field done"] [traceID=752dd42748c2d90d0f502bcc702384b0] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=452745552034264114] [fieldID=102] [rowCount=89288]
I20241217 04:07:02.364588 166 ChunkedSegmentSealedImpl.cpp:287] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 loads field 103 mmap false done
[2024/12/17 04:07:02.369 +00:00] [INFO] [segments/segment.go:965] ["submitted loadFieldData task to load pool"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=0] [rowCount=1491]
I20241217 04:07:02.369778 166 ChunkedSegmentSealedImpl.cpp:262] [SERVER][LoadFieldData][milvus][]segment 454575894751547022 loads field 0 with num_rows 1491
I20241217 04:07:02.369829 166 ChunkedSegmentSealedImpl.cpp:276] [SERVER][LoadFieldData][milvus][]segment 454575894751547022 submits load field 0 task to thread pool
[2024/12/17 04:07:02.372 +00:00] [INFO] [segments/segment.go:977] ["load field done"] [traceID=752dd42748c2d90d0f502bcc702384b0] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=452745552034264114] [fieldID=103] [rowCount=89288]
[2024/12/17 04:07:02.375 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361283] [subChannelCount=1] [loadSegmentCount=0]
[2024/12/17 04:07:02.375 +00:00] [WARN] [observers/collection_observer.go:350] ["failed to manual check current target, skip update load status"] [collectionID=449976277547361282] [partitionID=449976277547361283]
[2024/12/17 04:07:02.376 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361294] [subChannelCount=1] [loadSegmentCount=3]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277507568832] [replicaID=449976289146175490] [nodes="["{NodeID: 5864, AssignedScore: 66085.000000, CurrentScore: 72693.500000, Priority: 6608}"]"]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277526694406] [replicaID=449976289146175499] [nodes="["{NodeID: 5864, AssignedScore: 66314.000000, CurrentScore: 72945.400000, Priority: 6631}"]"]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361282] [replicaID=449976289146175500] [nodes="["{NodeID: 5864, AssignedScore: 491811.000000, CurrentScore: 540992.100000, Priority: 49181}"]"]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361318] [replicaID=449976289146175501] [nodes="["{NodeID: 5864, AssignedScore: 491280.000000, CurrentScore: 540408.000000, Priority: 49128}"]"]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=452745552034264114] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454245220196488370] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.376 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
I20241217 04:07:02.566706 163 ChunkedSegmentSealedImpl.cpp:287] [SERVER][LoadFieldData][milvus][]segment 454664254582251057 loads field 106 mmap false done
[2024/12/17 04:07:02.568 +00:00] [INFO] [segments/segment.go:965] ["submitted loadFieldData task to load pool"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=1] [rowCount=1491]
I20241217 04:07:02.568136 163 ChunkedSegmentSealedImpl.cpp:262] [SERVER][LoadFieldData][milvus][]segment 454575894751547022 loads field 1 with num_rows 1491
I20241217 04:07:02.568176 163 ChunkedSegmentSealedImpl.cpp:276] [SERVER][LoadFieldData][milvus][]segment 454575894751547022 submits load field 1 task to thread pool
[2024/12/17 04:07:02.584 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361283] [subChannelCount=1] [loadSegmentCount=0]
[2024/12/17 04:07:02.584 +00:00] [WARN] [observers/collection_observer.go:350] ["failed to manual check current target, skip update load status"] [collectionID=449976277547361282] [partitionID=449976277547361283]
[2024/12/17 04:07:02.584 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361294] [subChannelCount=1] [loadSegmentCount=3]
[2024/12/17 04:07:02.584 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277507568832] [replicaID=449976289146175490] [nodes="["{NodeID: 5864, AssignedScore: 66085.000000, CurrentScore: 72693.500000, Priority: 6608}"]"]
[2024/12/17 04:07:02.584 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277526694406] [replicaID=449976289146175499] [nodes="["{NodeID: 5864, AssignedScore: 66314.000000, CurrentScore: 72945.400000, Priority: 6631}"]"]
[2024/12/17 04:07:02.584 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361282] [replicaID=449976289146175500] [nodes="["{NodeID: 5864, AssignedScore: 491811.000000, CurrentScore: 540992.100000, Priority: 49181}"]"]
[2024/12/17 04:07:02.585 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361318] [replicaID=449976289146175501] [nodes="["{NodeID: 5864, AssignedScore: 491280.000000, CurrentScore: 540408.000000, Priority: 49128}"]"]
[2024/12/17 04:07:02.585 +00:00] [INFO] [segments/segment.go:977] ["load field done"] [traceID=0572292963dd41eb4a3c7817a0d2909d] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454664254582251057] [fieldID=106] [rowCount=8812]
[2024/12/17 04:07:02.585 +00:00] [INFO] [segments/segment_loader.go:991] ["load field binlogs done for sealed segment"] [traceID=0572292963dd41eb4a3c7817a0d2909d] [collection=449976277547361282] [segment=454664254582251057] [len(field)=12] [segmentType=Sealed]
[2024/12/17 04:07:02.585 +00:00] [INFO] [segments/segment_loader.go:817] ["Finish loading segment"] [traceID=0572292963dd41eb4a3c7817a0d2909d] [segmentID=454664254582251057] [loadFieldsIndexSpan=26.13µs] [complementScalarDataSpan=7.428µs] [loadRawDataSpan=16.62358853s] [patchEntryNumberSpan=865ns] [loadTextIndexesSpan=368ns]
I20241217 04:07:02.711825 165 ChunkedSegmentSealedImpl.cpp:287] [SERVER][LoadFieldData][milvus][]segment 452745552034264114 loads field 105 mmap false done
[2024/12/17 04:07:02.765 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=452745552034264114] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.765 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454245220196488370] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.766 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.766 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.766 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.770 +00:00] [INFO] [segments/segment.go:965] ["submitted loadFieldData task to load pool"] [traceID=4e8a1b93f4ef3027aca28234eb2ac01b] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=454575894751547022] [fieldID=100] [rowCount=1491]
I20241217 04:07:02.770570 165 ChunkedSegmentSealedImpl.cpp:262] [SERVER][LoadFieldData][milvus][]segment 454575894751547022 loads field 100 with num_rows 1491
I20241217 04:07:02.770635 165 ChunkedSegmentSealedImpl.cpp:276] [SERVER][LoadFieldData][milvus][]segment 454575894751547022 submits load field 100 task to thread pool
[2024/12/17 04:07:02.783 +00:00] [INFO] [datacoord/meta.go:1068] ["meta update: update flush segments info - update flush segments info successfully"]
[2024/12/17 04:07:02.793 +00:00] [INFO] [segments/segment.go:977] ["load field done"] [traceID=752dd42748c2d90d0f502bcc702384b0] [collectionID=449976277547361282] [partitionID=449976277547361294] [segmentID=452745552034264114] [fieldID=105] [rowCount=89288]
[2024/12/17 04:07:02.796 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361283] [subChannelCount=1] [loadSegmentCount=0]
[2024/12/17 04:07:02.796 +00:00] [WARN] [observers/collection_observer.go:350] ["failed to manual check current target, skip update load status"] [collectionID=449976277547361282] [partitionID=449976277547361283]
[2024/12/17 04:07:02.796 +00:00] [INFO] [observers/collection_observer.go:336] ["partition load progress"] [collectionID=449976277547361282] [partitionID=449976277547361294] [subChannelCount=1] [loadSegmentCount=3]
[2024/12/17 04:07:02.796 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277507568832] [replicaID=449976289146175490] [nodes="["{NodeID: 5864, AssignedScore: 66085.000000, CurrentScore: 72693.500000, Priority: 6608}"]"]
[2024/12/17 04:07:02.796 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277526694406] [replicaID=449976289146175499] [nodes="["{NodeID: 5864, AssignedScore: 66314.000000, CurrentScore: 72945.400000, Priority: 6631}"]"]
[2024/12/17 04:07:02.796 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361282] [replicaID=449976289146175500] [nodes="["{NodeID: 5864, AssignedScore: 491811.000000, CurrentScore: 540992.100000, Priority: 49181}"]"]
[2024/12/17 04:07:02.797 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=449976277547361318] [replicaID=449976289146175501] [nodes="["{NodeID: 5864, AssignedScore: 491280.000000, CurrentScore: 540408.000000, Priority: 49128}"]"]
[2024/12/17 04:07:02.798 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=452745552034264114] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.798 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454245220196488370] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.798 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664254582251057] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.798 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454664283062150492] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]
[2024/12/17 04:07:02.798 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=449976277547361282] [segmentID=454575894751547022] [replica=449976289146175500] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [level=Legacy] [from=-1] [to=5864]

Anything else?

I can't find similar questions in issues

The text was updated successfully, but these errors were encountered:

ahuai · 2024-12-17T05:43:03Z

My previous version number was mage: milvusdb/milvus:v2.3.4
If I roll back to 2.3.4 now, the Milvus logs will keep scrolling.
`
[2024/12/17 05:41:50.607 +00:00] [INFO] [datacoord/channel_manager.go:675] ["datanode release channel successfully, will reassign"] [nodeID=5979] [channel=by-dev-rootcoord-dml_11_449976277547361282v0]
[2024/12/17 05:41:50.607 +00:00] [INFO] [datacoord/channel_checker.go:134] ["stop timer before timeout"] ["watch state"=ToRelease] [nodeID=5979] [channelName=by-dev-rootcoord-dml_11_449976277547361282v0] ["timeout interval"=5m0s] [runningTimerCount=3]
[2024/12/17 05:41:50.607 +00:00] [INFO] [datanode/event_manager.go:210] ["handle put event success"] [key=channelwatch/5979/by-dev-rootcoord-dml_1_449976277507568832v0] [state=ReleaseSuccess] [vChanName=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.607 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_10_449976277526694406v0] ["watch state"=ToRelease]
[2024/12/17 05:41:50.607 +00:00] [INFO] [datanode/event_manager.go:112] ["DataNode received a PUT event with an end State"] [state=ReleaseSuccess]
[2024/12/17 05:41:50.607 +00:00] [INFO] [datanode/data_node.go:293] ["try to release flowgraph"] [vChanName=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.608 +00:00] [INFO] [datacoord/policy.go:459] ["AverageReassignPolicy working"] [avaNodesCount=0] [toAssignChannelNum=1] [avaNodesChannelSum=0]
[2024/12/17 05:41:50.608 +00:00] [WARN] [datacoord/policy.go:464] ["there is no available nodes when reassigning, return"]
[2024/12/17 05:41:50.608 +00:00] [WARN] [datacoord/channel_manager.go:821] ["failed to reassign channel to other nodes, assigning to the original DataNode"] [nodeID=5979] [channelName=by-dev-rootcoord-dml_11_449976277547361282v0]
[2024/12/17 05:41:50.608 +00:00] [INFO] [datacoord/channel_manager.go:827] ["channel manager reassigning channels"] ["old node ID"=5979] [updates="["{type=Add,nodeID=5979,channels=\"[by-dev-rootcoord-dml_11_449976277547361282v0]\"}"]"]
[2024/12/17 05:41:50.608 +00:00] [INFO] [datacoord/handler.go:62] [GetDataVChanPositions] [collectionID=449976277547361282] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [numOfSegments=14]
[2024/12/17 05:41:50.608 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [posTs=454663099801665537] [posTime=2024/12/17 02:21:23.594 +00:00]
[2024/12/17 05:41:50.608 +00:00] [INFO] [datacoord/channel_checker.go:113] ["timer started"] ["watch state"=ToWatch] [nodeID=5979] [channelName=by-dev-rootcoord-dml_11_449976277547361282v0] ["check interval"=5m0s]
[2024/12/17 05:41:50.608 +00:00] [INFO] [datanode/event_manager.go:112] ["DataNode received a PUT event with an end State"] [state=ReleaseSuccess]
[2024/12/17 05:41:50.608 +00:00] [INFO] [datanode/event_manager.go:210] ["handle put event success"] [key=channelwatch/5979/by-dev-rootcoord-dml_10_449976277526694406v0] [state=ReleaseSuccess] [vChanName=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.609 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_11_449976277547361282v0] ["watch state"=ToWatch]
[2024/12/17 05:41:50.609 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [progress=0]
[2024/12/17 05:41:50.609 +00:00] [INFO] [datacoord/channel_checker.go:167] ["stop timer for channel"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [timerCount=3]
[2024/12/17 05:41:50.609 +00:00] [INFO] [datacoord/channel_manager.go:675] ["datanode release channel successfully, will reassign"] [nodeID=5979] [channel=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.609 +00:00] [INFO] [datacoord/channel_checker.go:134] ["stop timer before timeout"] ["watch state"=ToRelease] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0] ["timeout interval"=5m0s] [runningTimerCount=3]
[2024/12/17 05:41:50.610 +00:00] [INFO] [datacoord/policy.go:459] ["AverageReassignPolicy working"] [avaNodesCount=0] [toAssignChannelNum=1] [avaNodesChannelSum=0]
[2024/12/17 05:41:50.610 +00:00] [WARN] [datacoord/policy.go:464] ["there is no available nodes when reassigning, return"]
[2024/12/17 05:41:50.610 +00:00] [WARN] [datacoord/channel_manager.go:821] ["failed to reassign channel to other nodes, assigning to the original DataNode"] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.610 +00:00] [INFO] [datacoord/channel_manager.go:827] ["channel manager reassigning channels"] ["old node ID"=5979] [updates="["{type=Add,nodeID=5979,channels=\"[by-dev-rootcoord-dml_1_449976277507568832v0]\"}"]"]
[2024/12/17 05:41:50.610 +00:00] [INFO] [datacoord/handler.go:62] [GetDataVChanPositions] [collectionID=449976277507568832] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [numOfSegments=3]
[2024/12/17 05:41:50.610 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [posTs=454666197896790019] [posTime=2024/12/17 05:38:21.889 +00:00]
[2024/12/17 05:41:50.610 +00:00] [INFO] [datacoord/channel_checker.go:113] ["timer started"] ["watch state"=ToWatch] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0] ["check interval"=5m0s]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:231] ["recover growing segments from checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=454665522779200920] [numRows=141]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:231] ["recover growing segments from checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=449976277547361282] [numRows=454665470268391503]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=452745552034263119] [numRows=89271]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=454245220196488370] [numRows=89094]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=454664283061949128] [numRows=15100]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=454664254582251057] [numRows=8812]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=454664283062150492] [numRows=4104]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=454575894751547022] [numRows=1491]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=454558948053483876] [numRows=53835]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=452745552034264114] [numRows=89288]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_11_449976277547361282v0] [segmentID=454664283062152925] [numRows=89269]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=452745552034263119] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454664283062152925] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=452745552034263119] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454664283062152925] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Normal] [segmentID=454665522779200920] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition="channel_name:"by-dev-rootcoord-dml_11_449976277547361282v0" msgID:"\035\243\375\307\3766O\006" msgGroup:"datanode-5976-by-dev-rootcoord-dml_11_449976277547361282v0-true" timestamp:454666183874183169 "] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454665522779200920] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454245220196488370] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454245220196488370] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [WARN] [datanode/channel_meta.go:215] ["failed to addSegment, collection mismatch"] ["current collection ID"=449976277547361294] ["expected collection ID"=449976277547361282]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454664283061949128] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454664283061949128] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454558948053483876] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454558948053483876] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=452745552034264114] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=452745552034264114] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454664254582251057] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454664254582251057] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454575894751547022] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454664283062150492] [collectionID=449976277547361282] [partitionID=449976277547361294] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [startPosition=] [endPosition=] [recoverTs=454663099801665537] [importing=false]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454664283062150492] [statsBinLogsLen=1]
[2024/12/17 05:41:50.615 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454575894751547022] [statsBinLogsLen=1]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_1_449976277507568832v0] ["watch state"=ToWatch]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datacoord/channel_checker.go:167] ["stop timer for channel"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [timerCount=3]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datacoord/channel_manager.go:675] ["datanode release channel successfully, will reassign"] [nodeID=5979] [channel=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datacoord/channel_checker.go:134] ["stop timer before timeout"] ["watch state"=ToRelease] [nodeID=5979] [channelName=by-dev-rootcoord-dml_10_449976277526694406v0] ["timeout interval"=5m0s] [runningTimerCount=3]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datacoord/policy.go:459] ["AverageReassignPolicy working"] [avaNodesCount=0] [toAssignChannelNum=1] [avaNodesChannelSum=0]
[2024/12/17 05:41:50.616 +00:00] [WARN] [datacoord/policy.go:464] ["there is no available nodes when reassigning, return"]
[2024/12/17 05:41:50.616 +00:00] [WARN] [datacoord/channel_manager.go:821] ["failed to reassign channel to other nodes, assigning to the original DataNode"] [nodeID=5979] [channelName=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datacoord/channel_manager.go:827] ["channel manager reassigning channels"] ["old node ID"=5979] [updates="["{type=Add,nodeID=5979,channels=\"[by-dev-rootcoord-dml_10_449976277526694406v0]\"}"]"]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datacoord/handler.go:62] [GetDataVChanPositions] [collectionID=449976277526694406] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [numOfSegments=3]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [posTs=454666197896790019] [posTime=2024/12/17 05:38:21.889 +00:00]
[2024/12/17 05:41:50.616 +00:00] [INFO] [datacoord/channel_checker.go:113] ["timer started"] ["watch state"=ToWatch] [nodeID=5979] [channelName=by-dev-rootcoord-dml_10_449976277526694406v0] ["check interval"=5m0s]
[2024/12/17 05:41:50.617 +00:00] [INFO] [datanode/data_sync_service.go:231] ["recover growing segments from checkpoints"] [vChannelName=by-dev-rootcoord-dml_1_449976277507568832v0] [segmentID=449976277507568832] [numRows=454664283063168584]
[2024/12/17 05:41:50.617 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_1_449976277507568832v0] [segmentID=454664283063168583] [numRows=178]
[2024/12/17 05:41:50.617 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454664283063168583] [collectionID=449976277507568832] [partitionID=449976277507568845] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [startPosition=] [endPosition=] [recoverTs=454666197896790019] [importing=false]
[2024/12/17 05:41:50.617 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454664283063168583] [statsBinLogsLen=1]
[2024/12/17 05:41:50.617 +00:00] [WARN] [datanode/channel_meta.go:215] ["failed to addSegment, collection mismatch"] ["current collection ID"=449976277507568845] ["expected collection ID"=449976277507568832]
[2024/12/17 05:41:50.617 +00:00] [WARN] [datanode/flow_graph_manager.go:136] ["fail to create new DataSyncService"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [error="collection not match: invalid parameter[expected=449976277507568832][actual=449976277507568845]"] [errorVerbose="collection not match: invalid parameter[expected=449976277507568832][actual=449976277507568845]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:218\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection not match\nWraps: (3) invalid parameter[expected=449976277507568832][actual=449976277507568845]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.617 +00:00] [WARN] [datanode/event_manager.go:175] ["handle put event: new data sync service failed"] [vChanName=by-dev-rootcoord-dml_1_449976277507568832v0] [error="collection not match: invalid parameter[expected=449976277507568832][actual=449976277507568845]"] [errorVerbose="collection not match: invalid parameter[expected=449976277507568832][actual=449976277507568845]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:218\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection not match\nWraps: (3) invalid parameter[expected=449976277507568832][actual=449976277507568845]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.617 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_10_449976277526694406v0] ["watch state"=ToWatch]
[2024/12/17 05:41:50.617 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_11_449976277547361282v0] [progress=0]
[2024/12/17 05:41:50.617 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [progress=0]
[2024/12/17 05:41:50.617 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [progress=0]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datanode/data_sync_service.go:231] ["recover growing segments from checkpoints"] [vChannelName=by-dev-rootcoord-dml_10_449976277526694406v0] [segmentID=449976277526694406] [numRows=454664283062151717]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_10_449976277526694406v0] [segmentID=454664283062151705] [numRows=407]
[2024/12/17 05:41:50.618 +00:00] [WARN] [datanode/channel_meta.go:215] ["failed to addSegment, collection mismatch"] ["current collection ID"=449976277526694420] ["expected collection ID"=449976277526694406]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454664283062151705] [collectionID=449976277526694406] [partitionID=449976277526694420] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [startPosition=] [endPosition=] [recoverTs=454666197896790019] [importing=false]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454664283062151705] [statsBinLogsLen=1]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datacoord/channel_checker.go:167] ["stop timer for channel"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [timerCount=3]
[2024/12/17 05:41:50.618 +00:00] [WARN] [datacoord/channel_manager.go:656] ["datanode watch channel failed or timeout, will release"] [nodeID=5979] [channel=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datacoord/handler.go:62] [GetDataVChanPositions] [collectionID=449976277507568832] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [numOfSegments=3]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [posTs=454666197896790019] [posTime=2024/12/17 05:38:21.889 +00:00]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datacoord/channel_checker.go:113] ["timer started"] ["watch state"=ToRelease] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0] ["check interval"=5m0s]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datanode/event_manager.go:112] ["DataNode received a PUT event with an end State"] [state=WatchFailure]
[2024/12/17 05:41:50.618 +00:00] [WARN] [datanode/flow_graph_manager.go:136] ["fail to create new DataSyncService"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [error="collection not match: invalid parameter[expected=449976277526694406][actual=449976277526694420]"] [errorVerbose="collection not match: invalid parameter[expected=449976277526694406][actual=449976277526694420]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:218\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection not match\nWraps: (3) invalid parameter[expected=449976277526694406][actual=449976277526694420]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.618 +00:00] [WARN] [datanode/event_manager.go:175] ["handle put event: new data sync service failed"] [vChanName=by-dev-rootcoord-dml_10_449976277526694406v0] [error="collection not match: invalid parameter[expected=449976277526694406][actual=449976277526694420]"] [errorVerbose="collection not match: invalid parameter[expected=449976277526694406][actual=449976277526694420]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:218\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection not match\nWraps: (3) invalid parameter[expected=449976277526694406][actual=449976277526694420]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datacoord/channel_checker.go:134] ["stop timer before timeout"] ["watch state"=ToWatch] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0] ["timeout interval"=5m0s] [runningTimerCount=4]
[2024/12/17 05:41:50.618 +00:00] [INFO] [datanode/event_manager.go:210] ["handle put event success"] [key=channelwatch/5979/by-dev-rootcoord-dml_1_449976277507568832v0] [state=WatchFailure] [vChanName=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datanode/event_manager.go:210] ["handle put event success"] [key=channelwatch/5979/by-dev-rootcoord-dml_10_449976277526694406v0] [state=WatchFailure] [vChanName=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datacoord/channel_checker.go:167] ["stop timer for channel"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [timerCount=3]
[2024/12/17 05:41:50.620 +00:00] [WARN] [datacoord/channel_manager.go:656] ["datanode watch channel failed or timeout, will release"] [nodeID=5979] [channel=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datanode/event_manager.go:112] ["DataNode received a PUT event with an end State"] [state=WatchFailure]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datacoord/handler.go:62] [GetDataVChanPositions] [collectionID=449976277526694406] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [numOfSegments=3]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [posTs=454666197896790019] [posTime=2024/12/17 05:38:21.889 +00:00]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_1_449976277507568832v0] ["watch state"=ToRelease]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datanode/data_node.go:293] ["try to release flowgraph"] [vChanName=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datacoord/channel_checker.go:113] ["timer started"] ["watch state"=ToRelease] [nodeID=5979] [channelName=by-dev-rootcoord-dml_10_449976277526694406v0] ["check interval"=5m0s]
[2024/12/17 05:41:50.620 +00:00] [INFO] [datacoord/channel_checker.go:134] ["stop timer before timeout"] ["watch state"=ToWatch] [nodeID=5979] [channelName=by-dev-rootcoord-dml_10_449976277526694406v0] ["timeout interval"=5m0s] [runningTimerCount=4]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_10_449976277526694406v0] ["watch state"=ToRelease]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [progress=0]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datanode/data_node.go:293] ["try to release flowgraph"] [vChanName=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [progress=0]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datanode/event_manager.go:112] ["DataNode received a PUT event with an end State"] [state=ReleaseSuccess]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datanode/event_manager.go:210] ["handle put event success"] [key=channelwatch/5979/by-dev-rootcoord-dml_1_449976277507568832v0] [state=ReleaseSuccess] [vChanName=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datacoord/channel_checker.go:167] ["stop timer for channel"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [timerCount=3]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datacoord/channel_manager.go:675] ["datanode release channel successfully, will reassign"] [nodeID=5979] [channel=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.622 +00:00] [INFO] [datacoord/channel_checker.go:134] ["stop timer before timeout"] ["watch state"=ToRelease] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0] ["timeout interval"=5m0s] [runningTimerCount=3]
[2024/12/17 05:41:50.623 +00:00] [INFO] [datacoord/policy.go:459] ["AverageReassignPolicy working"] [avaNodesCount=0] [toAssignChannelNum=1] [avaNodesChannelSum=0]
[2024/12/17 05:41:50.623 +00:00] [WARN] [datacoord/policy.go:464] ["there is no available nodes when reassigning, return"]
[2024/12/17 05:41:50.623 +00:00] [WARN] [datacoord/channel_manager.go:821] ["failed to reassign channel to other nodes, assigning to the original DataNode"] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.623 +00:00] [INFO] [datacoord/channel_manager.go:827] ["channel manager reassigning channels"] ["old node ID"=5979] [updates="["{type=Add,nodeID=5979,channels=\"[by-dev-rootcoord-dml_1_449976277507568832v0]\"}"]"]
[2024/12/17 05:41:50.623 +00:00] [INFO] [datanode/event_manager.go:112] ["DataNode received a PUT event with an end State"] [state=ReleaseSuccess]
[2024/12/17 05:41:50.623 +00:00] [INFO] [datacoord/handler.go:62] [GetDataVChanPositions] [collectionID=449976277507568832] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [numOfSegments=3]
[2024/12/17 05:41:50.623 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [posTs=454666197896790019] [posTime=2024/12/17 05:38:21.889 +00:00]
[2024/12/17 05:41:50.623 +00:00] [INFO] [datanode/event_manager.go:210] ["handle put event success"] [key=channelwatch/5979/by-dev-rootcoord-dml_10_449976277526694406v0] [state=ReleaseSuccess] [vChanName=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.623 +00:00] [INFO] [datacoord/channel_checker.go:113] ["timer started"] ["watch state"=ToWatch] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0] ["check interval"=5m0s]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_1_449976277507568832v0] ["watch state"=ToWatch]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datacoord/channel_checker.go:167] ["stop timer for channel"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [timerCount=3]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datacoord/channel_manager.go:675] ["datanode release channel successfully, will reassign"] [nodeID=5979] [channel=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datacoord/channel_checker.go:134] ["stop timer before timeout"] ["watch state"=ToRelease] [nodeID=5979] [channelName=by-dev-rootcoord-dml_10_449976277526694406v0] ["timeout interval"=5m0s] [runningTimerCount=3]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datacoord/policy.go:459] ["AverageReassignPolicy working"] [avaNodesCount=0] [toAssignChannelNum=1] [avaNodesChannelSum=0]
[2024/12/17 05:41:50.624 +00:00] [WARN] [datacoord/policy.go:464] ["there is no available nodes when reassigning, return"]
[2024/12/17 05:41:50.624 +00:00] [WARN] [datacoord/channel_manager.go:821] ["failed to reassign channel to other nodes, assigning to the original DataNode"] [nodeID=5979] [channelName=by-dev-rootcoord-dml_10_449976277526694406v0]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datacoord/channel_manager.go:827] ["channel manager reassigning channels"] ["old node ID"=5979] [updates="["{type=Add,nodeID=5979,channels=\"[by-dev-rootcoord-dml_10_449976277526694406v0]\"}"]"]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datacoord/handler.go:62] [GetDataVChanPositions] [collectionID=449976277526694406] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [numOfSegments=3]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [posTs=454666197896790019] [posTime=2024/12/17 05:38:21.889 +00:00]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datacoord/channel_checker.go:113] ["timer started"] ["watch state"=ToWatch] [nodeID=5979] [channelName=by-dev-rootcoord-dml_10_449976277526694406v0] ["check interval"=5m0s]
[2024/12/17 05:41:50.624 +00:00] [INFO] [datanode/data_sync_service.go:231] ["recover growing segments from checkpoints"] [vChannelName=by-dev-rootcoord-dml_1_449976277507568832v0] [segmentID=449976277507568832] [numRows=454664283063168584]
[2024/12/17 05:41:50.625 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [progress=0]
[2024/12/17 05:41:50.625 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_10_449976277526694406v0] ["watch state"=ToWatch]
[2024/12/17 05:41:50.625 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [progress=0]
[2024/12/17 05:41:50.626 +00:00] [INFO] [datanode/data_sync_service.go:231] ["recover growing segments from checkpoints"] [vChannelName=by-dev-rootcoord-dml_10_449976277526694406v0] [segmentID=449976277526694406] [numRows=454664283062151717]
[2024/12/17 05:41:50.645 +00:00] [WARN] [datanode/channel_meta.go:453] ["failed to deserialize stats"] [segmentID=454664316132224866] [error="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":15,"b":["vFoLrJOkqlhGB_jCBDqztoY0qgEcVkJYiKAHioJO4sHISkptTgzh6AIZt0BACSY4CKcEoAN6BhaIcgsCgoqUcA==","L6b7vX-oJ1__zG58WjuNYv8qwraxPeUi4uyw8UckJdSO-P20vLYlzqYWXGSSL7ik2kl0wGc-htqasgTs5SpZNw==","IxmjWR9TtwLs5-7V_MDip4gPxc1oNsCtkzN2i72uXkl0cy4Mp2qweyXxfenFEMZ4eroseN_jrJ_Od-3EzYljhw=="]},"pkType":21,"maxPk":"426840","minPk":"426765"}]"] [errorVerbose="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":15,"b":["vFoLrJOkqlhGB_jCBDqztoY0qgEcVkJYiKAHioJO4sHISkptTgzh6AIZt0BACSY4CKcEoAN6BhaIcgsCgoqUcA==","L6b7vX-oJ1__zG58WjuNYv8qwraxPeUi4uyw8UckJdSO-P20vLYlzqYWXGSSL7ik2kl0wGc-htqasgTs5SpZNw==","IxmjWR9TtwLs5-7V_MDip4gPxc1oNsCtkzN2i72uXkl0cy4Mp2qweyXxfenFEMZ4eroseN_jrJ_Od-3EzYljhw=="]},"pkType":21,"maxPk":"426840","minPk":"426765"}]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/storage.(*StatsReader).GetPrimaryKeyStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:258\n | github.com/milvus-io/milvus/internal/storage.DeserializeStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:290\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).loadStats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:451\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).initPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:475\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).InitPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:331\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:242\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string\nWraps: (3) invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":15,"b":["vFoLrJOkqlhGB_jCBDqztoY0qgEcVkJYiKAHioJO4sHISkptTgzh6AIZt0BACSY4CKcEoAN6BhaIcgsCgoqUcA==","L6b7vX-oJ1__zG58WjuNYv8qwraxPeUi4uyw8UckJdSO-P20vLYlzqYWXGSSL7ik2kl0wGc-htqasgTs5SpZNw==","IxmjWR9TtwLs5-7V_MDip4gPxc1oNsCtkzN2i72uXkl0cy4Mp2qweyXxfenFEMZ4eroseN_jrJ_Od-3EzYljhw=="]},"pkType":21,"maxPk":"426840","minPk":"426765"}]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.645 +00:00] [ERROR] [datanode/channel_meta.go:244] ["failed to init bloom filter"] [segmentID=454664316132224866] [error="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":15,"b":["vFoLrJOkqlhGB_jCBDqztoY0qgEcVkJYiKAHioJO4sHISkptTgzh6AIZt0BACSY4CKcEoAN6BhaIcgsCgoqUcA==","L6b7vX-oJ1__zG58WjuNYv8qwraxPeUi4uyw8UckJdSO-P20vLYlzqYWXGSSL7ik2kl0wGc-htqasgTs5SpZNw==","IxmjWR9TtwLs5-7V_MDip4gPxc1oNsCtkzN2i72uXkl0cy4Mp2qweyXxfenFEMZ4eroseN_jrJ_Od-3EzYljhw=="]},"pkType":21,"maxPk":"426840","minPk":"426765"}]"] [errorVerbose="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":15,"b":["vFoLrJOkqlhGB_jCBDqztoY0qgEcVkJYiKAHioJO4sHISkptTgzh6AIZt0BACSY4CKcEoAN6BhaIcgsCgoqUcA==","L6b7vX-oJ1__zG58WjuNYv8qwraxPeUi4uyw8UckJdSO-P20vLYlzqYWXGSSL7ik2kl0wGc-htqasgTs5SpZNw==","IxmjWR9TtwLs5-7V_MDip4gPxc1oNsCtkzN2i72uXkl0cy4Mp2qweyXxfenFEMZ4eroseN_jrJ_Od-3EzYljhw=="]},"pkType":21,"maxPk":"426840","minPk":"426765"}]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/storage.(*StatsReader).GetPrimaryKeyStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:258\n | github.com/milvus-io/milvus/internal/storage.DeserializeStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:290\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).loadStats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:451\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).initPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:475\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).InitPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:331\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:242\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string\nWraps: (3) invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":15,"b":["vFoLrJOkqlhGB_jCBDqztoY0qgEcVkJYiKAHioJO4sHISkptTgzh6AIZt0BACSY4CKcEoAN6BhaIcgsCgoqUcA==","L6b7vX-oJ1__zG58WjuNYv8qwraxPeUi4uyw8UckJdSO-P20vLYlzqYWXGSSL7ik2kl0wGc-htqasgTs5SpZNw==","IxmjWR9TtwLs5-7V_MDip4gPxc1oNsCtkzN2i72uXkl0cy4Mp2qweyXxfenFEMZ4eroseN_jrJ_Od-3EzYljhw=="]},"pkType":21,"maxPk":"426840","minPk":"426765"}]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [stack="github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n\t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:244\ngithub.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\ngithub.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\ngithub.com/panjf2000/ants/v2.(*goWorker).run.func1\n\t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67"]
[2024/12/17 05:41:50.645 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_1_449976277507568832v0] [segmentID=454664283063168583] [numRows=178]
[2024/12/17 05:41:50.645 +00:00] [WARN] [datanode/channel_meta.go:215] ["failed to addSegment, collection mismatch"] ["current collection ID"=449976277507568845] ["expected collection ID"=449976277507568832]
[2024/12/17 05:41:50.645 +00:00] [INFO] [datanode/data_sync_service.go:260] ["recover sealed segments form checkpoints"] [vChannelName=by-dev-rootcoord-dml_10_449976277526694406v0] [segmentID=454664283062151705] [numRows=407]
[2024/12/17 05:41:50.645 +00:00] [WARN] [datanode/channel_meta.go:215] ["failed to addSegment, collection mismatch"] ["current collection ID"=449976277526694420] ["expected collection ID"=449976277526694406]
[2024/12/17 05:41:50.645 +00:00] [WARN] [datanode/flow_graph_manager.go:136] ["fail to create new DataSyncService"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [error="collection not match: invalid parameter[expected=449976277507568832][actual=449976277507568845]"] [errorVerbose="collection not match: invalid parameter[expected=449976277507568832][actual=449976277507568845]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:218\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection not match\nWraps: (3) invalid parameter[expected=449976277507568832][actual=449976277507568845]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.645 +00:00] [WARN] [datanode/event_manager.go:175] ["handle put event: new data sync service failed"] [vChanName=by-dev-rootcoord-dml_1_449976277507568832v0] [error="collection not match: invalid parameter[expected=449976277507568832][actual=449976277507568845]"] [errorVerbose="collection not match: invalid parameter[expected=449976277507568832][actual=449976277507568845]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:218\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection not match\nWraps: (3) invalid parameter[expected=449976277507568832][actual=449976277507568845]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.645 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454664283063168583] [collectionID=449976277507568832] [partitionID=449976277507568845] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [startPosition=] [endPosition=] [recoverTs=454666197896790019] [importing=false]
[2024/12/17 05:41:50.645 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454664283063168583] [statsBinLogsLen=1]
[2024/12/17 05:41:50.652 +00:00] [INFO] [datacoord/channel_checker.go:167] ["stop timer for channel"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [timerCount=3]
[2024/12/17 05:41:50.652 +00:00] [WARN] [datacoord/channel_manager.go:656] ["datanode watch channel failed or timeout, will release"] [nodeID=5979] [channel=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.652 +00:00] [INFO] [datacoord/handler.go:62] [GetDataVChanPositions] [collectionID=449976277507568832] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [numOfSegments=3]
[2024/12/17 05:41:50.652 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [posTs=454666197896790019] [posTime=2024/12/17 05:38:21.889 +00:00]
[2024/12/17 05:41:50.652 +00:00] [INFO] [datacoord/channel_checker.go:113] ["timer started"] ["watch state"=ToRelease] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0] ["check interval"=5m0s]
[2024/12/17 05:41:50.652 +00:00] [INFO] [datanode/event_manager.go:112] ["DataNode received a PUT event with an end State"] [state=WatchFailure]
[2024/12/17 05:41:50.653 +00:00] [INFO] [datacoord/channel_checker.go:134] ["stop timer before timeout"] ["watch state"=ToWatch] [nodeID=5979] [channelName=by-dev-rootcoord-dml_1_449976277507568832v0] ["timeout interval"=5m0s] [runningTimerCount=4]
[2024/12/17 05:41:50.653 +00:00] [INFO] [datanode/event_manager.go:210] ["handle put event success"] [key=channelwatch/5979/by-dev-rootcoord-dml_1_449976277507568832v0] [state=WatchFailure] [vChanName=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.657 +00:00] [INFO] [datacoord/channel_manager.go:750] ["tickle update, timer delay"] [channel=by-dev-rootcoord-dml_1_449976277507568832v0] [progress=0]
[2024/12/17 05:41:50.657 +00:00] [INFO] [datanode/event_manager.go:123] ["DataNode is handling watchInfo PUT event"] [key=by-dev/meta/channelwatch/5979/by-dev-rootcoord-dml_1_449976277507568832v0] ["watch state"=ToRelease]
[2024/12/17 05:41:50.657 +00:00] [INFO] [datanode/data_node.go:293] ["try to release flowgraph"] [vChanName=by-dev-rootcoord-dml_1_449976277507568832v0]
[2024/12/17 05:41:50.657 +00:00] [WARN] [datanode/channel_meta.go:453] ["failed to deserialize stats"] [segmentID=454664283062151705] [error="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]"] [errorVerbose="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/storage.(*StatsReader).GetPrimaryKeyStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:258\n | github.com/milvus-io/milvus/internal/storage.DeserializeStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:290\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).loadStats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:451\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).initPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:475\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).InitPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:331\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:242\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:268\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string\nWraps: (3) invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.658 +00:00] [ERROR] [datanode/channel_meta.go:244] ["failed to init bloom filter"] [segmentID=454664283062151705] [error="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]"] [errorVerbose="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/storage.(*StatsReader).GetPrimaryKeyStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:258\n | github.com/milvus-io/milvus/internal/storage.DeserializeStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:290\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).loadStats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:451\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).initPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:475\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).InitPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:331\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:242\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:268\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string\nWraps: (3) invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [stack="github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n\t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:244\ngithub.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func2\n\t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:268\ngithub.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\ngithub.com/panjf2000/ants/v2.(*goWorker).run.func1\n\t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67"]
[2024/12/17 05:41:50.658 +00:00] [WARN] [datanode/flow_graph_manager.go:136] ["fail to create new DataSyncService"] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [error="collection not match: invalid parameter[expected=449976277526694406][actual=449976277526694420]"] [errorVerbose="collection not match: invalid parameter[expected=449976277526694406][actual=449976277526694420]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:218\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection not match\nWraps: (3) invalid parameter[expected=449976277526694406][actual=449976277526694420]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.658 +00:00] [WARN] [datanode/event_manager.go:175] ["handle put event: new data sync service failed"] [vChanName=by-dev-rootcoord-dml_10_449976277526694406v0] [error="collection not match: invalid parameter[expected=449976277526694406][actual=449976277526694420]"] [errorVerbose="collection not match: invalid parameter[expected=449976277526694406][actual=449976277526694420]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:218\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:240\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) collection not match\nWraps: (3) invalid parameter[expected=449976277526694406][actual=449976277526694420]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.658 +00:00] [INFO] [datanode/channel_meta.go:220] ["adding segment"] [type=Flushed] [segmentID=454664283062151705] [collectionID=449976277526694406] [partitionID=449976277526694420] [channel=by-dev-rootcoord-dml_10_449976277526694406v0] [startPosition=] [endPosition=] [recoverTs=454666197896790019] [importing=false]
[2024/12/17 05:41:50.658 +00:00] [INFO] [datanode/channel_meta.go:386] ["begin to init pk bloom filter"] [segmentID=454664283062151705] [statsBinLogsLen=1]
[2024/12/17 05:41:50.659 +00:00] [WARN] [datanode/channel_meta.go:453] ["failed to deserialize stats"] [segmentID=454664283062151705] [error="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]"] [errorVerbose="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/storage.(*StatsReader).GetPrimaryKeyStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:258\n | github.com/milvus-io/milvus/internal/storage.DeserializeStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:290\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).loadStats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:451\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).initPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:475\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).InitPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:331\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:242\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:268\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string\nWraps: (3) invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/12/17 05:41:50.659 +00:00] [ERROR] [datanode/channel_meta.go:244] ["failed to init bloom filter"] [segmentID=454664283062151705] [error="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]"] [errorVerbose="json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string: invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n | github.com/milvus-io/milvus/internal/storage.(*StatsReader).GetPrimaryKeyStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:258\n | github.com/milvus-io/milvus/internal/storage.DeserializeStats\n | \t/go/src/github.com/milvus-io/milvus/internal/storage/stats.go:290\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).loadStats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:451\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).initPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:475\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).InitPKstats\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:331\n | github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:242\n | github.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func2\n | \t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:268\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) json: cannot unmarshal array into Go struct field bloomFilterJSON.b of type string\nWraps: (3) invalid parameter[expected=valid JSON][actual={"fieldID":100,"max":0,"min":0,"bfType":4,"bf":{"k":13,"b":["r7YiA8pNwjZyfggUcfSC3y9vlGJr9c8aLliLHhIycepAPLiRQDk3MeogCqoynyZ7dU8ziwGf5cHhUyLrQxCTrw==","fIHOQbrovuV-sNngisk7Sw491ndq3h8WALOOLJHb-ZyoPr--b6UdhBg006l-gyfZ4dWRjeYC8pOfmKfOCcacSA==","Q2yW0J2lf_yENh1EwkVIeo5EZqaA_zbSm2rc_lecc118yHODDDtvmZJjXOTqCR632Knfo6hQbP3LgZW76YdvQQ==","jBZbolH7Ora6JMgb4HVHDuCb3F4bIaasjtRToAcgsR9es94VE1EQTnNNQTk0ZTuwIG_jGJ62cYJZwAlm2TYiFQ==","5yEE_-5IME5IstMDT6N_MdSYsQa3YkXGV_Jl7GnLZJzHblSJdOWT2mXHgU7FR5WzDXimDJ68o2DkCVegvSQyaQ==","gK8y5DYzKbfq5EcO5SumtQKk-CkjRWrhaFjcjPChhAuO0MDUxqMrIzKkDhVjcKKqbEgaoAzSVhLWsiJFAGhKEA==","wVfj_3dXi-9X5wZDq_f9wcQvdZZlf6bXq99cWf9qJsEWzWLB48aaiXlVC49Zcq2NmUHef6nb5AgkSX306H_G2g==","MZgCiUrUKUACEASqouUtL3cpbUhFxCKXAsIBEtaLgzrAgFBZSaRDKkuZpY5IdQEXaxpkLQJZclOIDNMTjG_3RA==","dB5zJdgNJvYG5iTH2fUW9kjM6X-KKXoswtVPc5O6yRHEVl5Zbe4TNrQ9ZRS67xDAGEDQ325w_dpCiDVaIJIyvg==","yBcz6Xv-JjZrn2g4zONXGKk3n0pisCvZidqvXz_sFrviWxJazY5hT35TzyNK_bJQ-L3y7nW_EwWARs2Ih_1TuA==","JchEDML54lBWugSQ3gix9u48bdF88__nHm2MdtDtFXUMYXbjLgdnvjFPXacM0oPuEM-0-dwFh0zj2wBjKDd6dw==","gByRkrlquZLcXAWaZKoVQU6rRIFf2nayABgIVNGRZUUJNQKbCNFgJCiGd-Yg4QSgRpKfMCGIZATHr6DghnKS8w==","OCt1Vb9o-9Ed7LOa33YtWnXzCFdNxHjC7sYBe5992dYFGruWfAAXP68mYuXBd6_F8ZKWnTVJdfrWU1lvyzocOw==","fvEXCSGfYANkFYfAPQeuKmKj1D05hanBshSHrdZPapPL8a8CJ1jL2vKQKzdbawL66AkqE7rwBSMzVogWDD3SLw=="]},"pkType":21,"maxPk":"5572e51d129c40a89af36fb37f2e2819_99","minPk":"5572e51d129c40a89af36fb37f2e2819_0"}]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [stack="github.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n\t/go/src/github.com/milvus-io/milvus/internal/datanode/channel_meta.go:244\ngithub.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func2\n\t/go/src/github.com/milvus-io/milvus/internal/datanode/data_sync_service.go:268\ngithub.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\ngithub.com/panjf2000/ants/v2.(*goWorker).run.func1\n\t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67"]

`

ahuai · 2024-12-18T02:04:56Z

我发现好像是由于我的服务器配置比较低当我插入40w+条数据到milvus后(2张表各40w+) 这个milvus的服务就会无限重启。。

yanliang567 · 2024-12-18T09:41:39Z

yes, you should add more cpu and memory to the etcd and minio services. Aslo the milvus service need more cpu and memory if you have more collections and more/bigger datasets

/assign @ahuai
/unassign

ahuai · 2024-12-18T09:47:58Z

yes, you should add more cpu and memory to the etcd and minio services. Aslo the milvus service need more cpu and memory if you have more collections and more/bigger datasets

/assign @ahuai /unassign

Is it unnecessary to configure a 512M memory limit in Docker if my server currently has 2 CPU cores and 8GB of RAM?

ahuai · 2024-12-18T09:49:21Z

在我的docker-compose.yml配置文件里

deploy:
resources:
limits:
cpus: '2.0'
memory: '6G'
deploy:
resources:
limits:
cpus: '0.5'
memory: '512M'

这部分配置是不是多余的？我主要是怕主机内存不够用

xiaofan-luan · 2024-12-24T07:49:21Z

You can not rollback to 2.3 from 2.5 unless you truncate the data.

My guess yiou need to check whether you have enough memory to host the amount of your data.

ahuai added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 17, 2024

ahuai assigned yanliang567 Dec 17, 2024

sre-ci-robot assigned ahuai and unassigned yanliang567 Dec 18, 2024

yanliang567 added help wanted Extra attention is needed and removed kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: milvus-standalone always restart #38513

[Bug]: milvus-standalone always restart #38513

ahuai commented Dec 17, 2024

ahuai commented Dec 17, 2024

ahuai commented Dec 18, 2024 •

edited

Loading

yanliang567 commented Dec 18, 2024

ahuai commented Dec 18, 2024

ahuai commented Dec 18, 2024 •

edited

Loading

xiaofan-luan commented Dec 24, 2024

[Bug]: milvus-standalone always restart #38513

[Bug]: milvus-standalone always restart #38513

Comments

ahuai commented Dec 17, 2024

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

ahuai commented Dec 17, 2024

ahuai commented Dec 18, 2024 • edited Loading

yanliang567 commented Dec 18, 2024

ahuai commented Dec 18, 2024

ahuai commented Dec 18, 2024 • edited Loading

xiaofan-luan commented Dec 24, 2024

ahuai commented Dec 18, 2024 •

edited

Loading

ahuai commented Dec 18, 2024 •

edited

Loading