运行“隐私求交”时后台容器无法调度，作业失败 #1599

lyh251543 · 2024-11-27T07:14:27Z

lyh251543
Nov 27, 2024

环境：自建centos7.9虚拟机
内存：16G
版本：secretflow-allinone-linux-x86_64-v1.10.0
问题：
1、安装后运行联合圈人时隐私求交卡住无响应10多分钟
2、使用默认的csv数据，内置alice\bob节点
3、通过master容器查看pod发现alice和bob的任务调度容器启动失败“CreatecontainerError”
4、事件日志如下：
Events:
Type Reason Age From Message

Warning FailedScheduling 37s kuscia-scheduler 0/3 nodes are available: waiting for task resource. can not find related task resource, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
Normal Scheduled 35s kuscia-scheduler Successfully assigned alice/jidu-hkkceuxl-node-3-0 to root-kuscia-lite-alice-ceph-node
Warning MissingClusterDNS 34s (x2 over 35s) Agent pod: "jidu-hkkceuxl-node-3-0_alice(40231fc0-e3f5-442c-9163-2bc337f0ab56)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
5、k3s集群状态如下：
bash-5.2# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
etcd-0 Healthy
controller-manager Healthy ok
scheduler Unhealthy Get "https://127.0.0.1:10259/healthz": net/http: TLS handshake timeout
6、麻烦老师帮忙看下问题原因

bzzbzz7 · 2024-12-10T09:21:28Z

bzzbzz7
Dec 10, 2024

运到同样问题，如下是我的环境信息

内存：8C/16G
版本：secretflow-allinone-linux-x86_64-v1.10.0.tar.gz
docker：Docker version 20.10.10, build b485636
环境：
Virtualization: kvm
Operating System: CentOS Linux 8
CPE OS Name: cpe:/o:centos:centos:8
Kernel: Linux 4.18.0-305.3.1.el8.x86_64
Architecture: x86-64

error日志：
2024-12-09 14:34:34.824 ERROR controller/gateway.go:143 create gateway(name:root-kuscia-master-localhost-localdomain namespace:kuscia-system) fail: namespaces "kuscia-system" not found
2024-12-09 14:34:34.826 ERROR controller/gateway.go:114 sync gateway error: namespaces "kuscia-system" not found

2024-12-09 14:34:35.107 ERROR domain/authorization_resource.go:126 Update domain [kuscia-system] auth label error: Operation cannot be fulfilled on domains.kuscia.secretflow "kuscia-system": the object has been modified; please apply your changes to the latest version and try again
2024-12-09 14:34:35.107 WARN domain/controller.go:406 Create domain kuscia-system auth failed: Operation cannot be fulfilled on domains.kuscia.secretflow "kuscia-system": the object has been modified; please apply your changes to the latest version and try again

2024-12-09 14:35:10.547 ERROR controller/handshake.go:422 DestReplyHandshake for [alice-kuscia-system] failed, detail-> invalid source domain [alice] publickey in domainroute [alice-kuscia-system], error: public key is empty

2024-12-09 14:42:45.261 INFO queue/queue.go:126 Finish processing item: queue id[domain-controller], key[alice] (1.567425ms)
E1209 14:42:52.521014 7 leaderelection.go:330] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-12-09 14:42:52.521 INFO nlog/nlog.go:77 E1209 14:42:52.521014 7 leaderelection.go:330] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-12-09 14:42:52.521 INFO nlog/nlog.go:77 E1209 14:42:52.521014 7 leaderelection.go:330] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2024-12-09 14:42:52.521 INFO nlog/nlog.go:77 E1209 14:42:52.521014 7 leaderelection.go:330] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

1 reply

wangzul Dec 12, 2024
Collaborator

运到同样问题，如下是我的环境信息

内存：8C/16G 版本：secretflow-allinone-linux-x86_64-v1.10.0.tar.gz docker：Docker version 20.10.10, build b485636 环境： Virtualization: kvm Operating System: CentOS Linux 8 CPE OS Name: cpe:/o:centos:centos:8 Kernel: Linux 4.18.0-305.3.1.el8.x86_64 Architecture: x86-64

error日志： 2024-12-09 14:34:34.824 ERROR controller/gateway.go:143 create gateway(name:root-kuscia-master-localhost-localdomain namespace:kuscia-system) fail: namespaces "kuscia-system" not found 2024-12-09 14:34:34.826 ERROR controller/gateway.go:114 sync gateway error: namespaces "kuscia-system" not found

2024-12-09 14:34:35.107 ERROR domain/authorization_resource.go:126 Update domain [kuscia-system] auth label error: Operation cannot be fulfilled on domains.kuscia.secretflow "kuscia-system": the object has been modified; please apply your changes to the latest version and try again 2024-12-09 14:34:35.107 WARN domain/controller.go:406 Create domain kuscia-system auth failed: Operation cannot be fulfilled on domains.kuscia.secretflow "kuscia-system": the object has been modified; please apply your changes to the latest version and try again

2024-12-09 14:35:10.547 ERROR controller/handshake.go:422 DestReplyHandshake for [alice-kuscia-system] failed, detail-> invalid source domain [alice] publickey in domainroute [alice-kuscia-system], error: public key is empty

2024-12-09 14:42:45.261 INFO queue/queue.go:126 Finish processing item: queue id[domain-controller], key[alice] (1.567425ms) E1209 14:42:52.521014 7 leaderelection.go:330] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 2024-12-09 14:42:52.521 INFO nlog/nlog.go:77 E1209 14:42:52.521014 7 leaderelection.go:330] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 2024-12-09 14:42:52.521 INFO nlog/nlog.go:77 E1209 14:42:52.521014 7 leaderelection.go:330] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 2024-12-09 14:42:52.521 INFO nlog/nlog.go:77 E1209 14:42:52.521014 7 leaderelection.go:330] error retrieving resource lock kube-system/kube-scheduler: Get "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

可以将你的问题在SecretPad重新提问。避免打扰
https://github.com/secretflow/secretpad/issues

wangzul · 2024-12-12T03:22:23Z

wangzul
Dec 12, 2024
Collaborator

环境：自建centos7.9虚拟机内存：16G 版本：secretflow-allinone-linux-x86_64-v1.10.0 问题： 1、安装后运行联合圈人时隐私求交卡住无响应10多分钟 2、使用默认的csv数据，内置alice\bob节点 3、通过master容器查看pod发现alice和bob的任务调度容器启动失败“CreatecontainerError” 4、事件日志如下： Events: Type Reason Age From Message

Warning FailedScheduling 37s kuscia-scheduler 0/3 nodes are available: waiting for task resource. can not find related task resource, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. Normal Scheduled 35s kuscia-scheduler Successfully assigned alice/jidu-hkkceuxl-node-3-0 to root-kuscia-lite-alice-ceph-node Warning MissingClusterDNS 34s (x2 over 35s) Agent pod: "jidu-hkkceuxl-node-3-0_alice(40231fc0-e3f5-442c-9163-2bc337f0ab56)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy. 5、k3s集群状态如下： bash-5.2# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-0 Healthy controller-manager Healthy ok scheduler Unhealthy Get "https://127.0.0.1:10259/healthz": net/http: TLS handshake timeout 6、麻烦老师帮忙看下问题原因

该问题已在secretflow/kuscia#463 跟进。

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SecretFlow

运行“隐私求交”时后台容器无法调度，作业失败 #1599

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 1 reply

{{title}}

{{title}}

{{title}}

Select a reply

SecretFlow

运行“隐私求交”时后台容器无法调度，作业失败 #1599

lyh251543 Nov 27, 2024

Replies: 2 comments · 1 reply

bzzbzz7 Dec 10, 2024

wangzul Dec 12, 2024 Collaborator

wangzul Dec 12, 2024 Collaborator

lyh251543
Nov 27, 2024

Replies: 2 comments 1 reply

bzzbzz7
Dec 10, 2024

wangzul Dec 12, 2024
Collaborator

wangzul
Dec 12, 2024
Collaborator