Can we make sure the existing configuration stored in etcd and backup by pd wouldn't be destroyed? Otherwise, we have to send an announcement for this. I did a quick test with this change, but found all prescheduled pods are gone.

From docker inspect, I did see the binding for such volume: /mnt/master-pd/var/etcd:/var/etcd.

$ ./cluster/kubectl.sh get pods hostnames-k4uqg
current-context: "kubernetes"
Running: ./cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods hostnames-k4uqg
F0213 16:49:55.999296    7860 get.go:164] pod "hostnames-k4uqg" not found

From Kubelet, I found kubelet kills it due to unwanted.

Can we upgrade to 2.0.3 while we're at it?

Another issue I forget to mention is that you have to kill previous etcd process, otherwise, docker container of etcd won't run due to port conflicts too. I couldn't see any place you kill previous process.

For above etcd data loss issue, we guess somewhere we did format etcd pd. We need to figure out.

@brendandburns Actually we build etcd image from 2.0.3+ with some latest fixes from development: etcd-io/etcd#2253

@ArtfulCoder Here is the script we mount master-pd: kubernetes/cluster/gce/templates/format-and-mount-pd.sh

Can we verify this on a vagrant provider before merge to make sure nothing crazy happens there and that the cluster starts as expected?

@zmerlynn : Do we need to worry about the upgrade path yet ? Do we do live pushes of new kubernetes versions for GKE customers who are using kubernetes ?

coreos folks just build a new docker image based on v2.0.3 for us yesterday. Please update your manifest file to use the latest image built from them:

https://quay.io/repository/coreos/etcd?tag=v2.0.3

@dchen1107
@thockin
@brendandburns

The updated PR incorporates the following changes based on the feedback.

• using etcd 2.0.3
• etcd.manifest updated to use nicer port names
• etcd.manifest does not use port 7001
• removed monit since etcd will be monitored via the kubelet process on master
• removed the /var/etcd symlink creation command.

Known issues with this PR:

• push to upgrade does not work if original cluster has etcd directly running on master. But once we have a cluster running with etcd in docker, the push works.
• Could not verify with vagrant. I could not bring up vagrant with the head version (without my changes)

I have some time tomorrow to test this on vagrant.

@ArtfulCoder I patched my local client, and tried to figure out why the data is lost when converting etcd to a docker container.

Here are a couple of issues:

1. We didn't kill previous running monit and etcd server. We talked about this before. I manually kill those two processes; after this, I found the data is not lost.

2. Looks like you are not using coreos's etcd 2.0.3 image I posted above, and rebuild one by yourself. With this image, etcd image is failed due to "unknown wal version". At this stage, the data is not lost.

root@kubernetes-master:/var/log# docker ps -a
CONTAINER ID        IMAGE                   COMMAND                CREATED              STATUS                          PORTS                                            NAMES
f52cbb83708b        kubernetes/etcd:2.0.3   "/usr/local/bin/etcd   5 seconds ago        Exited (1) 4 seconds ago                                                         k8s_etcd-container.35824ab6_etcd-server.default.file_etcdmanifestdr6v86kneumqqa5i8i5pk1d1u1gcmd_ea74a221  

root@kubernetes-master:/var/log# docker logs f52cbb83708b
2015/02/20 22:11:34 netutil: Resolving kubernetes-master:2380 to 10.240.10.168:2380
2015/02/20 22:11:34 etcd: listening for peers on http://localhost:2380
2015/02/20 22:11:34 etcd: listening for peers on http://localhost:7001
2015/02/20 22:11:34 etcd: listening for client requests on http://0.0.0.0:4001
2015/02/20 22:11:34 etcd: stopping listening for client requests on http://0.0.0.0:4001
2015/02/20 22:11:34 etcd: stopping listening for peers on http://localhost:7001
2015/02/20 22:11:34 etcd: stopping listening for peers on http://localhost:2380
2015/02/20 22:11:34 etcd: unknown wal version in data dir /var/etcd

3. I upgraded /etc/kubernetes/manifests/etcd.manifest with

    image: quay.io/coreos/etcd:v2.0.3 

Now etcd is running as container.

root@kubernetes-master:/var/log# docker ps -a
CONTAINER ID        IMAGE                        COMMAND                CREATED             STATUS                      PORTS                                            NAMES
a617d25b7374        quay.io/coreos/etcd:v2.0.3   "/etcd /usr/local/bi   52 minutes ago      Up 52 minutes                                                                k8s_etcd-container.a9774c74_etcd-server.default.file_etcdmanifestdr6v86kneumqqa5i8i5pk1d1u1gcmd_9f3f474c   

4. But now I can not connect to my server. I think there must be a misconfigurations. Again the data is not lost. But the issue is that I cannot connect to etcd server anymore. Please note that we ran the image built by ourselves earlier with the same configurations, and it works except previous data is lost.
   cc/ @xiang90, please shed some lights on this issue.

root@kubernetes-master:/home/dawnchen# etcdctl -C="http://`hostname -i`:4001" ls
Error:  cannot sync with the cluster using endpoints http://10.240.10.168:4001

root@kubernetes-master:/home/dawnchen# docker inspect a617d25b7374
[{
    "AppArmorProfile": "",
    "Args": [
        "/usr/local/bin/etcd",
        "--addr",
        "10.240.10.168:4001",
        "--bind-addr",
        "0.0.0.0:4001",
        "--data-dir",
        "/var/etcd",
        "--initial-advertise-peer-urls",
        "http://kubernetes-master:2380",
        "--name",
        "kubernetes-master",
        "--initial-cluster",
        "kubernetes-master=http://kubernetes-master:2380"
    ],
    "Config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "/usr/local/bin/etcd",
            "--addr",
            "10.240.10.168:4001",
            "--bind-addr",
            "0.0.0.0:4001",
            "--data-dir",
            "/var/etcd",
            "--initial-advertise-peer-urls",
            "http://kubernetes-master:2380",
            "--name",
            "kubernetes-master",
            "--initial-cluster",
            "kubernetes-master=http://kubernetes-master:2380"
        ],
...

All above steps I performed so for, I verified data volume back up etcd, and the data is not lost. I suspect why before the data is lost we saw earlier might be due to version mismatch and etcd server discard those records? We need @Xiang99 comfirm this.

I was able to fix the upgrade path issue.
I have updated the pull request.
On Feb 2nd, the etcd data dir was changed to /var/etcd/data from /var/etcd
I hadn't realized this change since I had deleted the file and didnt see merge conflicts.

At this point, the only change that needs to be verified is that my PR works for vagrant.

@dchen1107
@xiang90

The commit where the data dir change was made is here:
commit 9a29d9f
Author: Justin SB justin@fathomdb.com
Date: Mon Feb 2 08:06:00 2015 -0800

Etcd data dir must be empty

Otherwise we get the error "unknown wal version in data dir /var/etcd"

When this is the homedir for etcd, e.g. .bashrc confuses it

Haven't read the PR yet. Have you fixed issue of killing previous etcd process and monit proccess yet?

@ArtfulCoder @dchen1107 etcd data dir cannot have "garbage". etcd does check that, it will return unknow if so.

I have to update the push script to kill etcd running on host. I will update the PR soon.

With flannel depending on etcd for IP address reservation, how is the etcd pod IP assigned?

The master is not participating in the overlay at this point. Etcd is
using a host-port to be accessed. You raise a good point though - if/when
the master participates in the overlay, we have a chicken-egg problem.
Also host ports suck for performance....

On Mon, Feb 23, 2015 at 7:14 AM, josselin-c notifications@github.com
wrote:

With flannel depending on etcd for IP address reservation, how is the etcd
pod IP assigned?

Reply to this email directly or view it on GitHub
#4442 (comment)
.

With or without flannel, we are going to have a chicken-egg issue if we tried to get rid of host port mapping for those "static" pods like etcd server here. This PR just puts etcd into a pod like what we had done for cadvisor, and a separate issue #4128 filed to handle master node configuration, including networking (cbr0 vs flannel).

Again, I don't think flannel issue or chicken-egg issue is blocker for this one. Or I miss something here?

To recap IRL discussions:

Assume we run etcd in a pod:

If the master doesn't run flannel: can it see and reach pods that are on
the flannel network? and vice-versa

If the master does run flannel: well, where does flannel get its etcd from?

On Mon, Feb 23, 2015 at 2:48 PM, Dawn Chen notifications@github.com wrote:

With or without flannel, we are going to have a chicken-egg issue if we
tried to get rid of host port mapping for those "static" pods like etcd
server here. This PR just puts etcd into a pod like what we had done for
cadvisor, and a separate issue #4128
#4128 filed to
handle master node configuration, including networking (cbr0 vs flannel).

Again, I don't think flannel issue or chicken-egg issue is blocker for
this one. Or I miss something here?

Reply to this email directly or view it on GitHub
#4442 (comment)
.

cc/ @kelseyhightower

One proposal on solving such chicken-egg issue is introducing static configuration to flannel. So that when flannel starts up, it won't query etcd for the configuration; Once etcd server is up as docker container, flannel will sync-up its "static" subnets with etcd. According @kelseyhightower, this is a very small patch to flannel, we can verify it this week.

@errordeveloper what about #4483 ? Is weave also concerned?

@dchen1107 @thockin I've put together a PR that would add static configuration and subnet leases to flannel. flannel-io/flannel#131

I've added some initial docs as well:

https://github.com/kelseyhightower/flannel/blob/static-configuration/Documentation/static-configuration.md

Is this PR active? If not please close it according to https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/devel/pull-requests.md

This is active. We leave it not merging because there was a discussion on breakage day. We plan to merge this along with other breakage prs. We just heard from @thockin yesterday the decision is no breakage day. We are going to move on with backward compatibility support.

Patched the latest commit and did a upgrade test:

1. etcd server is successfully running as a docker container.
2. etcd_server has permission issue mount that data volume
   related etcd.log:

# cat etcd.log 
2015/03/27 22:37:57 netutil: Resolving kubernetes-master:2380 to 10.240.118.117:2380
2015/03/27 22:37:57 etcd: listening for peers on http://localhost:2380
2015/03/27 22:37:57 etcd: listening for peers on http://localhost:7001
2015/03/27 22:37:57 etcd: listening for client requests on http://0.0.0.0:4001
2015/03/27 22:37:57 netutil: Resolving kubernetes-master:2380 to 10.240.118.117:2380
2015/03/27 22:37:57 etcdserver: name = kubernetes-master
2015/03/27 22:37:57 etcdserver: data dir = /var/etcd/data
2015/03/27 22:37:57 etcdserver: member dir = /var/etcd/data/member
2015/03/27 22:37:57 etcdserver: heartbeat = 100ms
2015/03/27 22:37:57 etcdserver: election = 1000ms
2015/03/27 22:37:57 etcdserver: snapshot count = 10000
2015/03/27 22:37:57 etcdserver: advertise client URLs = http://10.240.118.117:4001
2015/03/27 22:37:57 etcdserver: initial advertise peer URLs = http://10.240.118.117:2380
2015/03/27 22:37:57 etcdserver: initial cluster = kubernetes-master=http://kubernetes-master:2380
2015/03/27 22:37:57 etcdserver: start member 804adb5313283e27 in cluster 8831aec77f300152
2015/03/27 22:37:57 raft: 804adb5313283e27 became follower at term 0
2015/03/27 22:37:57 raft: newRaft 804adb5313283e27 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2015/03/27 22:37:57 raft: 804adb5313283e27 became follower at term 1
2015/03/27 22:37:57 etcdserver: added local member 804adb5313283e27 [http://kubernetes-master:2380] to cluster 8831aec77f300152
2015/03/27 22:37:59 raft: 804adb5313283e27 is starting a new election at term 1
2015/03/27 22:37:59 raft: 804adb5313283e27 became candidate at term 2
2015/03/27 22:37:59 raft: 804adb5313283e27 received vote from 804adb5313283e27 at term 2
2015/03/27 22:37:59 raft: 804adb5313283e27 became leader at term 2
2015/03/27 22:37:59 raft.node: 804adb5313283e27 elected leader 804adb5313283e27 at term 2
2015/03/27 22:37:59 etcdserver: published {Name:kubernetes-master ClientURLs:[http://10.240.118.117:4001]} to cluster 8831aec77f300152
2015/03/27 22:46:27 etcdserver: failed to purge wal file open /var/etcd/data/member/wal: permission denied

I manually changed kubelet allowed priviledge, and changed manifest with priviledge=true, still run into the same issue.

In this case, the previous existing pods are missing obviously. kubectl get pods only show static pods.

@ArtfulCoder is looking into it.

With the latest patch, I verified it and works fine. LGTM

#5011

I believe this change broke head. (e2e-gce 3921 onward stopped being able to turn up clusters)

BTW, sorry for leaving little detail before the revert, but I didn't do too much prodding. Clusters were getting here and hanging:

Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

I haven't trying privately applying this PR and seeing what the issue was, but after reverting, we're back green.

Dawn and I had manually applied the PR on GCE and it seemed fine

Sent from my iPhone

On Mar 28, 2015, at 12:38 PM, Zach Loafman notifications@github.com wrote:

BTW, sorry for leaving little detail before the revert, but I didn't do too much prodding. Clusters were getting here and hanging:

Waiting for cluster initialization.

This will continually check to see if the API for kubernetes is reachable.
This might loop forever if there was some uncaught error during start
up.

..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
I haven't trying privately applying this PR and seeing what the issue was, but after reverting, we're back green.

—
Reply to this email directly or view it on GitHub.

You'll be happy to know that my first attempt to re-apply this PR came back with a successful cluster. :P

(But I saw it spin at least once privately as well, so Jenkins isn't crazy.)

@ArtfulCoder: I can't explain this, but I even went all the way through hack/e2e-from-release.sh, which is about as close to Jenkins as you can get, and this PR looks possibly fine. We could possibly try re-introducing it on the theory that maybe the build itself failed, but I'm not comfortable with that answer. Regardless, we should perhaps wait until after we cut 0.14.0 on Monday to give it some soak?

The bug might be in how I derive the address etcd will use and the address that spider we thinks that etcd is on. I should have etcd on localhost and then have apiserver looking localhost for etcd.
That way there is no scope for error in etcd discoverability.

Sent from my iPhone

On Mar 28, 2015, at 3:05 PM, Zach Loafman notifications@github.com wrote:

@ArtfulCoder: I can't explain this, but I even went all the way through hack/e2e-from-release.sh, which is about as close to Jenkins as you can get, and this PR looks possibly fine. We could possibly try re-introducing it on the theory that maybe the build itself failed, but I'm not comfortable with that answer. Regardless, we should perhaps wait until after we cut 0.14.0 on Monday to give it some soak?

—
Reply to this email directly or view it on GitHub.

Spider=apiserver :)
Phone auto-complete isn't that great :)

Sent from my iPhone

On Mar 28, 2015, at 3:18 PM, Abhi Shah abshah@google.com wrote:

The bug might be in how I derive the address etcd will use, and the address that apiserver thinks that etcd is on. I should have etcd on localhost and then have apiserver looking localhost for etcd.
That way there is no scope for error in etcd discoverability.

Sent from my iPhone

On Mar 28, 2015, at 3:05 PM, Zach Loafman notifications@github.com wrote:

@ArtfulCoder: I can't explain this, but I even went all the way through hack/e2e-from-release.sh, which is about as close to Jenkins as you can get, and this PR looks possibly fine. We could possibly try re-introducing it on the theory that maybe the build itself failed, but I'm not comfortable with that answer. Regardless, we should perhaps wait until after we cut 0.14.0 on Monday to give it some soak?

—
Reply to this email directly or view it on GitHub.

@zmerlynn @ArtfulCoder Let's re-introduce this PR back on Monday with another round of fully testing. Have a good weekend.

Sounds good

Sent from my iPhone

On Mar 28, 2015, at 3:54 PM, Dawn Chen notifications@github.com wrote:

@zmerlynn @ArtfulCoder Let's re-introduce this PR back on Monday with another round of fully testing. Have a good weekend.

—
Reply to this email directly or view it on GitHub.

#6200 should fix the transient errors that we were seeing.