Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test docker nightly build against GCI image and Kubernetes head #25215

Closed
dchen1107 opened this issue May 5, 2016 · 13 comments
Closed

Test docker nightly build against GCI image and Kubernetes head #25215

dchen1107 opened this issue May 5, 2016 · 13 comments
Assignees
@Random-Liu @wonderfly
Labels
area/test lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@dchen1107
Copy link
Member

dchen1107 commented May 5, 2016 •
edited
Loading

We have ~20 test jenkins builds are running against various GCI image, and Kubernetes builds already. @wonderfly kindly agreed to add another one to test docker nightly build for us. This is relatively easy comparing to today's debian-based containerVM image, and less error-prone.

@liangchenye This is the first step to unblock the work of automating docker validation process which you have been working on. Beyond this, you need to:

  • Introduce node e2e test suite for automating docker validation. You can re-use the cloud_init config introduced by @wonderfly
  • Include your runtime conformance test suites to the newly introduced docker validation suite.
  • Convert our current docker performance tests written by @Random-Liu to the new test suite.
  • Add docker restart / recovery, resource tracking tests.

After the first suite is done, we can expand the test suite to cover other os distro easily.
@dchen1107 dchen1107 added area/test sig/node Categorizes an issue or PR as relevant to SIG Node. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels May 5, 2016
@dchen1107
Copy link
Member Author

cc/ @timstclair

@wonderfly suggested an alternative approach, and will provide the detail information here.

@wonderfly
Copy link
Contributor

We definitely want to automate the docker validation tests that you guys wrote/will write, and run them on a continuous basis. We, GCI, internally have been doing continuous testing (at a much smaller scale) with docker HEAD, but it turned out that docker HEAD is rather flaky that made our test results more like noise. So instead, @dchen1107 suggested that we could do it with docker's prereleases (RC releases). I think that makes sense: we avoid the noise from docker's HEAD flakiness, yet still get a chance to find bugs and upstream fixes before its official release.

As of the implementation on GCI, docker is built in to our images. We don't do runtime download/installation as other distros do. But we have an experimental nightly builder that builds a GCI image with the latest docker release. I think we just need to update it to use docker prereleases, and supply the images it builds to the continuous docker validations tests. It is a little non-intuitive in the sense that to test a new docker version we'd have to build a new image, but it's essentially the same thing as requested in comment#1: a combination of GCI HEAD build, k8s HEAD build, and latest docker prerelease.

@Random-Liu
Copy link
Member

@wonderfly SGTM! :)

@wonderfly
Copy link
Contributor

Sorry this took a little longer. The GCI image side change was completed a week ago. I had to work on something else so didn't get a chance to sort out the k8s side change yet. I expect to send it out for review early next week.

wonderfly added a commit to wonderfly/kubernetes that referenced this issue Jun 3, 2016
@wonderfly
Prep for continuous Docker validation test
a3d712c 
We want to continuously validate Docker releases (kubernetes#25215), on GCI. This change
adds a new test config variable, `KUBE_GCI_DOCKER_VERSION`, through which we can
specify which version of Docker we want to run on the master and nodes. This
change also patches the Jenkins e2e-runner with the ability to fetch the latest
Docker (pre)release, and sets the aforementioned variable accordingly.
@wonderfly wonderfly mentioned this issue Jun 3, 2016
Merged
@wonderfly
Copy link
Contributor

#26813 is out to fix this. It has been in the submit queue for a week and I have no idea when it'll get merged.

@wonderfly
Copy link
Contributor

The PR is marked as P3 so it's getting pushed back over and over again. Can anybody with the authority adjust its priority? Or if it's not super urgent then this issue should be a P3 too. 😄

k8s-github-robot pushed a commit that referenced this issue Jun 18, 2016
Merge pull request #26813 from wonderfly/choose_docker_version
48b255d 
Automatic merge from submit-queue

Prep for continuous Docker validation test

```release-note
Add a test config variable to specify desired Docker version to run on GCI.
```
We want to continuously validate Docker releases (#25215), on GCI. This change
adds a new test config variable, `KUBE_GCI_DOCKER_VERSION`, through which we can
specify which version of Docker we want to run on the master and nodes. This
change also patches the Jenkins e2e-runner with the ability to fetch the latest
Docker (pre)release, and sets the aforementioned variable accordingly.

Tested on my local Jenkins instance that was able to start a cluster with the latest Docker version (different from installed version) running on both master and nodes.

@dchen1107 Can you review?

cc/ @andyzheng0831 for changes in `cluster/gce/gci/helper.sh`, and @ixdy @spxtr for changes to the Jenkins e2e-runner

cc/ @kubernetes/goog-image
@dchen1107 dchen1107 mentioned this issue Jun 20, 2016
Merged
@wonderfly
Copy link
Contributor

It took a few iterations but we just had our first green docker validation build (checkout the internal Jenkins instance), and you can see the GCI image version, k8s commit number, and docker version (1.12.0-rc2), that it was run against.

So I think my share of the issue is done. I will continue helping you guys for questions, but this should be assigned to someone else now. @dchen1107 I don't have the permission to edit assignee. Can you?

@Random-Liu Random-Liu self-assigned this Jun 28, 2016
@Random-Liu
Copy link
Member

@wonderfly Thanks a lot! I've assigned myself. And will leave unassign to @dchen1107. :)

@Random-Liu Random-Liu mentioned this issue Jun 29, 2016
Merged
k8s-github-robot pushed a commit that referenced this issue Jul 12, 2016
Merge pull request #28213 from Random-Liu/docker-validation-node-e2e
b6ffcf7 
Automatic merge from submit-queue

Node E2E: Prep for continuous Docker validation node e2e test

Based on #28516, for #25215.

#26813 added support to run e2e test on gci preview image and newest docker version.
This PR added the same support to node e2e test.

The main dependencies of node e2e test are `docker`, `kubelet`, `etcd` and `apiserver`.
Currently, node e2e test builds `kubelet` and `apiserver` locally, and copies them into `/tmp` directory in VM instance. GCI also has built-in `docker`. So the only dependency missing is `etcd`.

This PR injected a simple cloud-init script when creating instance to install `etcd` during node startup.

@andyzheng0831 for the cloud init script.
@wonderfly for the gci instance setup.
@pwittrock for the node e2e test change.

/cc @dchen1107 

[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()
@Random-Liu
Copy link
Member

@dchen1107 Can we close this one?

@wonderfly
Copy link
Contributor

Let's close this now?

@fejta-bot
Copy link

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 18, 2017
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 17, 2018
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Random-Liu Random-Liu

@wonderfly wonderfly

Labels
area/test lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Random-Liu @wonderfly @dchen1107 @k8s-ci-robot @fejta-bot