Upgrade etcd to v2.0.5 on the master #5801

satnam6502 · 2015-03-23T18:27:59Z

This changes our Salt setup to install etcd v2.0.5 on the master.
Running tests to verify this actually works.

zmerlynn · 2015-03-23T19:10:01Z

cluster/saltbase/salt/etcd/init.sls

@@ -10,7 +10,7 @@
 #    shasum <tar>
 # 6. Update this file with new tar version and new hash

-{% set etcd_version="v2.0.0" %}
+{% set etcd_version="v2.0.5" %}
 {% set etcd_tar_url="https://storage.googleapis.com/kubernetes-release/etcd/etcd-%s-linux-amd64.tar.gz"
  | format(etcd_version)  %}
 {% set etcd_tar_hash="sha1=b3cd41d1748bf882a58a98c9585fd5849b943811" %}


It seems unlikely that v2.0.5 has the same hash?

Good point. Fixed.

zmerlynn · 2015-03-23T20:28:17Z

I assume you ran e2es on this?

satnam6502 · 2015-03-23T20:30:25Z

I am still testing to make sure that e2e works and I am creating a cluster by hand to make sure everything is fine and to manually inspect that the master is indeed running v2.0.5. Please do not merge until I have verified that everything is OK. Thank you.

satnam6502 · 2015-03-23T22:00:02Z

kube-up.sh works. The master is upgraded:

Linux kubernetes-master 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt4-3~bpo70+1 (2015-02-12) x86_64
=== GCE Kubernetes node setup complete ===
satnam@kubernetes-master:~$ etcd --version
etcd version 2.0.5

I can do things like kubectl get pods (which work).
Now running e2e tests.

satnam6502 · 2015-03-23T22:10:55Z

I am not convinced this is working yet. When I query the Elasticsearch service through the proxy interface I sometimes get:

{
  "kind": "Status",
  "creationTimestamp": null,
  "apiVersion": "v1beta1",
  "status": "Failure",
  "message": "501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]",
  "code": 500
}

and sometimes I get the expected result:

{
  "status" : 200,
  "name" : "Gideon Mace",
  "cluster_name" : "kubernetes_logging",
  "version" : {
    "number" : "1.4.4",
    "build_hash" : "c88f77ffc81301dfa9dfd81ca2232f09588bd512",
    "build_timestamp" : "2015-02-19T13:05:36Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.3"
  },
  "tagline" : "You Know, for Search"
}

@vishh and I were talking earlier about accessing cluster services via the proxy vs. ELB.

satnam6502 · 2015-03-23T22:51:23Z

"Only" 3 e2e failures. The usual suspects.

Summarizing 3 Failures:

[Fail] Shell [It] tests that .guestbook.sh.swp passes 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/shell.go:67

[Fail] kubectl guestbook [It] should create and stop a working application 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/kubectl.go:277

[Fail] Shell [It] tests that guestbook.sh passes 
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/shell.go:67

Ran 29 of 31 Specs in 1605.932 seconds
FAIL! -- 26 Passed | 3 Failed | 2 Pending | 0 Skipped F0323 15:40:47.965084   28545 driver.go:87] At least one test failed

satnam6502 · 2015-03-23T23:06:38Z

I made another cluster and it seems to work fine. I can list pods. Elasticsearch consistently returns the correct value. The Kibana dashboard works.
The serve_hostnames soak test functions for 1 pod pernode.

$ ./serve_hostnames --per_node=1
I0323 16:04:19.407214   10021 serve_hostnames.go:53] Starting serve_hostnames soak test with queries=1000 and perNode=1 upTo=-1
I0323 16:04:19.752859   10021 serve_hostnames.go:78] Nodes found on this cluster:
I0323 16:04:19.752930   10021 serve_hostnames.go:80] 0: kubernetes-minion-1h7n.c.kubernetes-satnam.internal
I0323 16:04:19.752972   10021 serve_hostnames.go:80] 1: kubernetes-minion-20bl.c.kubernetes-satnam.internal
I0323 16:04:19.753007   10021 serve_hostnames.go:80] 2: kubernetes-minion-khea.c.kubernetes-satnam.internal
I0323 16:04:19.753045   10021 serve_hostnames.go:80] 3: kubernetes-minion-x25i.c.kubernetes-satnam.internal
I0323 16:04:19.753126   10021 serve_hostnames.go:86] Using namespace serve-hostnames-6217 for this test.
I0323 16:04:19.753172   10021 serve_hostnames.go:89] Creating service serve-hostnames
I0323 16:04:19.810457   10021 serve_hostnames.go:123] Creating pod serve-hostnames-6217/serve-hostname-0-0 on node kubernetes-minion-1h7n.c.kubernetes-satnam.internal
I0323 16:04:19.867720   10021 serve_hostnames.go:123] Creating pod serve-hostnames-6217/serve-hostname-1-0 on node kubernetes-minion-20bl.c.kubernetes-satnam.internal
I0323 16:04:19.925172   10021 serve_hostnames.go:123] Creating pod serve-hostnames-6217/serve-hostname-2-0 on node kubernetes-minion-khea.c.kubernetes-satnam.internal
I0323 16:04:19.980908   10021 serve_hostnames.go:123] Creating pod serve-hostnames-6217/serve-hostname-3-0 on node kubernetes-minion-x25i.c.kubernetes-satnam.internal
I0323 16:04:20.038474   10021 serve_hostnames.go:158] Waiting for the serve-hostname pods to be ready
I0323 16:04:30.175948   10021 serve_hostnames.go:175] serve-hostnames-6217/serve-hostname-0-0 is running
I0323 16:04:35.269694   10021 serve_hostnames.go:175] serve-hostnames-6217/serve-hostname-1-0 is running
I0323 16:04:35.313927   10021 serve_hostnames.go:175] serve-hostnames-6217/serve-hostname-2-0 is running
I0323 16:04:35.358347   10021 serve_hostnames.go:175] serve-hostnames-6217/serve-hostname-3-0 is running
I0323 16:05:20.698515   10021 serve_hostnames.go:197] {
  "kind": "Status",
  "creationTimestamp": null,
  "apiVersion": "v1beta1",
  "status": "Failure",
  "message": "no endpoints available for serve-hostnames",
  "code": 500
}: 15  
I0323 16:05:20.698644   10021 serve_hostnames.go:197] serve-hostname-2-0: 254  
I0323 16:05:20.698674   10021 serve_hostnames.go:197] serve-hostname-1-0: 251  
I0323 16:05:20.698701   10021 serve_hostnames.go:197] serve-hostname-0-0: 240  
I0323 16:05:20.698727   10021 serve_hostnames.go:197] serve-hostname-3-0: 240  
I0323 16:05:20.698765   10021 serve_hostnames.go:208] Iteration 0 took 45.340382144s for 1000 queries
I0323 16:06:05.297976   10021 serve_hostnames.go:197] serve-hostname-2-0: 274  
I0323 16:06:05.298023   10021 serve_hostnames.go:197] serve-hostname-0-0: 249  
I0323 16:06:05.298039   10021 serve_hostnames.go:197] serve-hostname-3-0: 228  
I0323 16:06:05.298055   10021 serve_hostnames.go:197] serve-hostname-1-0: 249  
I0323 16:06:05.298076   10021 serve_hostnames.go:208] Iteration 1 took 44.599261154s for 1000 queries

I propose that we merge this upgrade to etcd.

satnam6502 · 2015-03-23T23:09:15Z

@kelseyhightower FYI we are about to upgrade to etcd v2.0.5 and I will also upgrade to the latest go-etcd when I can defeat Godeps.

zmerlynn · 2015-03-23T23:09:46Z

This is more testing than most things get. I'm sniffing at Jenkins now.

Upgrade etcd to v2.0.5 on the master

googlebot added the cla: yes label Mar 23, 2015

roberthbailey assigned zmerlynn Mar 23, 2015

zmerlynn reviewed Mar 23, 2015
View reviewed changes

satnam6502 force-pushed the etcd branch 2 times, most recently from b40c1ec to fa563d0 Compare March 23, 2015 20:27

Upgrade etcd to v2.0.5 on the master

69f867d

satnam6502 force-pushed the etcd branch from fa563d0 to 69f867d Compare March 23, 2015 20:52

zmerlynn added a commit that referenced this pull request Mar 23, 2015

Merge pull request #5801 from satnam6502/etcd

8065fb5

Upgrade etcd to v2.0.5 on the master

zmerlynn merged commit 8065fb5 into kubernetes:master Mar 23, 2015

satnam6502 mentioned this pull request Mar 24, 2015

Revert "Upgrade etcd to v2.0.5 on the master" #5862

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade etcd to v2.0.5 on the master #5801

Upgrade etcd to v2.0.5 on the master #5801

satnam6502 commented Mar 23, 2015

zmerlynn Mar 23, 2015

satnam6502 Mar 23, 2015

zmerlynn commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

zmerlynn commented Mar 23, 2015

Upgrade etcd to v2.0.5 on the master #5801

Upgrade etcd to v2.0.5 on the master #5801

Conversation

satnam6502 commented Mar 23, 2015

zmerlynn Mar 23, 2015

Choose a reason for hiding this comment

satnam6502 Mar 23, 2015

Choose a reason for hiding this comment

zmerlynn commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

satnam6502 commented Mar 23, 2015

zmerlynn commented Mar 23, 2015