Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes AWS Stuck -TLS Handshake Error in Kube-api - In Version 1.3.0 #28888

Closed
talk2vino opened this issue Jul 13, 2016 · 5 comments
Closed

Kubernetes AWS Stuck -TLS Handshake Error in Kube-api - In Version 1.3.0 #28888

talk2vino opened this issue Jul 13, 2016 · 5 comments
Assignees
@justinsb
Labels
area/kube-proxy sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.

Comments

@talk2vino
Copy link
Contributor

Environment: Kubernetes in AWS

Previously i tried 30+ times in version Kubernetes 1.2.4 on AWS . It works flawless all the times.

Today when i tried the same with fresh new cluster with kubernetes version 1.3.0 . The cluster initialisation loops forever for 30 minutes

Creating autoscaling group
 0 minions started; waiting
 0 minions started; waiting
 0 minions started; waiting
 0 minions started; waiting
 5 minions started; ready
Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

kube version

kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

I go and checked each of the nodes with "docker ps" , and all of the nodes responded.

The below are the kube-apiserver logs from Master.

tail -f /var/log/kube-apiserver.log 
I0707 10:46:01.682375       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal/status: (3.478826ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.57:39608]
I0707 10:46:03.321912       7 logs.go:41] http: TLS handshake error from 182.74.106.3:37435: tls: client offered an unsupported, maximum protocol version of 301
I0707 10:46:03.684879       7 handlers.go:165] GET /api/v1/nodes: (2.205527ms) 200 [[pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format] 172.20.0.57:56892]
I0707 10:46:03.688692       7 handlers.go:165] GET /api/v1/namespaces/kube-system/pods/heapster-v1.1.0-3166934156-x3pqp: (1.511805ms) 200 [[pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format] 172.20.0.57:56892]
I0707 10:46:03.719907       7 handlers.go:165] GET /api/v1/nodes: (1.735749ms) 200 [[pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format] 172.20.0.57:56894]
I0707 10:46:03.723690       7 handlers.go:165] GET /api/v1/namespaces/kube-system/pods/heapster-v1.1.0-3166934156-x3pqp: (1.531456ms) 200 [[pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format] 172.20.0.57:56894]
I0707 10:46:04.832582       7 handlers.go:165] GET /healthz: (98.701µs) 200 [[Go-http-client/1.1] 127.0.0.1:55365]
I0707 10:46:05.082949       7 handlers.go:165] GET /api/v1/nodes: (1.584727ms) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/node-controller] 127.0.0.1:55358]
I0707 10:46:05.244386       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-58.us-west-2.compute.internal: (1.078198ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.58:34755]
I0707 10:46:05.252279       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-58.us-west-2.compute.internal/status: (3.069901ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.58:34755]
I0707 10:46:05.962474       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-54.us-west-2.compute.internal: (1.352055ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.54:55878]
I0707 10:46:05.968562       7 logs.go:41] http: TLS handshake error from 182.74.106.3:37436: tls: client offered an unsupported, maximum protocol version of 301
I0707 10:46:05.971862       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-54.us-west-2.compute.internal/status: (3.127346ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.54:55878]
I0707 10:46:07.210960       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-56.us-west-2.compute.internal: (1.406006ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.56:57991]
I0707 10:46:07.221951       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-56.us-west-2.compute.internal/status: (3.563008ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.56:57991]
I0707 10:46:08.619418       7 logs.go:41] http: TLS handshake error from 182.74.106.3:37437: tls: client offered an unsupported, maximum protocol version of 301
I0707 10:46:08.640468       7 handlers.go:165] GET /api/v1/watch/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=15&timeoutSeconds=573: (9m33.000413463s) 200 [[kube-apiserver/v1.3.0 (linux/amd64) kubernetes/2831379] 127.0.0.1:51381]
I0707 10:46:09.977067       7 handlers.go:165] GET /api/v1/nodes: (1.621331ms) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.978726       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-54.us-west-2.compute.internal: (917.251µs) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.980799       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-54.us-west-2.compute.internal/status: (1.555254ms) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.982049       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-55.us-west-2.compute.internal: (811.358µs) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.983941       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-55.us-west-2.compute.internal/status: (1.375196ms) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.985281       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-56.us-west-2.compute.internal: (894.413µs) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.987236       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-56.us-west-2.compute.internal/status: (1.474856ms) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.988493       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal: (856.196µs) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.990389       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal/status: (1.414768ms) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.991670       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-58.us-west-2.compute.internal: (851.331µs) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:09.993575       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-58.us-west-2.compute.internal/status: (1.359682ms) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/route-controller] 127.0.0.1:55358]
I0707 10:46:10.085520       7 handlers.go:165] GET /api/v1/nodes: (1.539579ms) 200 [[kube-controller-manager/v1.3.0 (linux/amd64) kubernetes/2831379/node-controller] 127.0.0.1:55358]
I0707 10:46:10.648457       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-55.us-west-2.compute.internal: (1.396064ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.55:47727]
I0707 10:46:10.658403       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-55.us-west-2.compute.internal/status: (5.005459ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.55:47727]
I0707 10:46:11.271671       7 logs.go:41] http: TLS handshake error from 182.74.106.3:37438: tls: client offered an unsupported, maximum protocol version of 301
I0707 10:46:11.684834       7 handlers.go:165] GET /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal: (1.328298ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.57:39608]
I0707 10:46:11.693274       7 handlers.go:165] PUT /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal/status: (2.990194ms) 200 [[kubelet/v1.3.0 (linux/amd64) kubernetes/2831379] 172.20.0.57:39608]

In particular

http: TLS handshake error from 182.74.106.3:38370: tls: client offered an unsupported, maximum protocol version of 301
 logs.go:41] http: TLS handshake error from 182.74.106.3:38371: tls: client offered an unsupported, maximum protocol version of 301
 http: TLS handshake error from 182.74.106.3:38372: tls: client offered an unsupported, maximum protocol version of 301
logs.go:41] http: TLS handshake error from 182.74.106.3:38373: tls: client offered an unsupported, maximum protocol version of 301

While debugging the util.sh stuck at

++ curl --insecure --user admin:fzZzkn18jGAv7Q66 --max-time 5 --fail --output /dev/null --silent https://52.36.190.215/healthz

Due to the TLS Handshake error

It is not happend on the previous releases. 1.2.4
@talk2vino
Copy link
Contributor Author

talk2vino commented Jul 13, 2016 •
edited
Loading

Its work when i use TLSv1.2 in curl

curl --insecure --user admin:bwteKU6yD3RsFzWI --max-time 5 --silent https://52.42.15.182/healthz --tlsv1.2
ok

The Default util.sh doesn't have tls argument.Without that the Kubernetes Version 1.3.0 cannot able to functional on AWS.

I can see the changelog 1.3.0 added

Security/Auth
L7 LB controller and disk attach controllers run on master, so nodes do not need those privileges.
Setting TLS1.2 minimum
kubectl create secret tls command
Webhook Token Authenticator
beta PodSecurityPolicy objects limits use of security-sensitive features by pods.

Do this need to update in the aws/utils.sh in package right ?.

@dims dims mentioned this issue Jul 13, 2016
Closed
@dims
Copy link
Member

dims commented Jul 13, 2016 •
edited
Loading

@talk2vino yep. filed a PR. Thanks for your help in tracking this down.

@k8s-github-robot k8s-github-robot added area/kube-proxy sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels Aug 3, 2016
dims added a commit to dims/kubernetes that referenced this issue Aug 5, 2016
@dims
[WIP] Use TLS 1.2 to access healthz API
1eb2348 
In the following PR, TLS was set to 1.2 as the minimum
because TLS1.0 and TLS1.1 are vulnerable:
kubernetes#26169

However the scripts that used curl were not updated to match
the TLS version.

Since --tlsv1.2 was introduced in curl 7.34.0, we should check
the version before using the option.

Fixes kubernetes#28888
@justinsb justinsb self-assigned this Nov 17, 2016
@xingxing122
Copy link

hi，I also encountered this problem, how do you solve

@xingxing122
Copy link

log：
Feb 23 18:18:26 dcms-dev-master kube-apiserver: I0223 18:18:26.695481 14103 logs.go:41] http: TLS handshake error from 172.16.0.2:46216: EOF

@talk2vino talk2vino mentioned this issue Feb 28, 2017
Closed
@smarterclayton
Copy link
Contributor

Closing due to age, reopen if this is re-creatable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@justinsb justinsb

Labels
area/kube-proxy sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dims @justinsb @smarterclayton @talk2vino @k8s-github-robot @xingxing122