diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2d1491087feba..d02fe22323ae2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,7 +21,7 @@ Follow either of the two links above to access the appropriate CLA and instructi ## Protocols for Collaborative Development -Please read [this doc](docs/collab.md) for information on how we're running development for the project. +Please read [this doc](docs/devel/collab.md) for information on how we're running development for the project. ## Adding dependencies diff --git a/README.md b/README.md index d83451cc4a5ab..424015b90159e 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ Check out examples of Kubernetes in action, and community projects in the larger * [Example of dynamic updates](examples/update-demo/README.md) * [Cluster monitoring with heapster and cAdvisor](https://github.com/GoogleCloudPlatform/heapster) * [Community projects](https://github.com/GoogleCloudPlatform/kubernetes/wiki/Kubernetes-Community) -* [Development guide](docs/development.md) +* [Development guide](docs/devel/development.md) Or fork and start hacking! diff --git a/docs/devel/collab.md b/docs/devel/collab.md new file mode 100644 index 0000000000000..c464404895dd5 --- /dev/null +++ b/docs/devel/collab.md @@ -0,0 +1,37 @@ +# On Collaborative Development + +Kubernetes is open source, but many of the people working on it do so as their day job. In order to avoid forcing people to be "at work" effectively 24/7, we want to establish some semi-formal protocols around development. Hopefully these rules make things go more smoothly. If you find that this is not the case, please complain loudly. + +## Patches welcome + +First and foremost: as a potential contributor, your changes and ideas are welcome at any hour of the day or night, weekdays, weekends, and holidays. Please do not ever hesitate to ask a question or send a PR. + +## Timezones and calendars + +For the time being, most of the people working on this project are in the US and on Pacific time. Any times mentioned henceforth will refer to this timezone. Any references to "work days" will refer to the US calendar. + +## Code reviews + +All changes must be code reviewed. For non-maintainers this is obvious, since you can't commit anyway. But even for maintainers, we want all changes to get at least one review, preferably from someone who knows the areas the change touches. For non-trivial changes we may want two reviewers. The primary reviewer will make this decision and nominate a second reviewer, if needed. Except for trivial changes, PRs should sit for at least a 2 hours to allow for wider review. + +Most PRs will find reviewers organically. If a maintainer intends to be the primary reviewer of a PR they should set themselves as the assignee on GitHub and say so in a reply to the PR. Only the primary reviewer of a change should actually do the merge, except in rare cases (e.g. they are unavailable in a reasonable timeframe). + +If a PR has gone 2 work days without an owner emerging, please poke the PR thread and ask for a reviewer to be assigned. + +Except for rare cases, such as trivial changes (e.g. typos, comments) or emergencies (e.g. broken builds), maintainers should not merge their own changes. + +Expect reviewers to request that you avoid [common go style mistakes](https://code.google.com/p/go-wiki/wiki/CodeReviewComments) in your PRs. + +## Assigned reviews + +Maintainers can assign reviews to other maintainers, when appropriate. The assignee becomes the shepherd for that PR and is responsible for merging the PR once they are satisfied with it or else closing it. The assignee might request reviews from non-maintainers. + +## Merge hours + +Maintainers will do merges between the hours of 7:00 am Monday and 7:00 pm (19:00h) Friday. PRs that arrive over the weekend or on holidays will only be merged if there is a very good reason for it and if the code review requirements have been met. + +There may be discussion an even approvals granted outside of the above hours, but merges will generally be deferred. + +## Holds + +Any maintainer or core contributor who wants to review a PR but does not have time immediately may put a hold on a PR simply by saying so on the PR discussion and offering an ETA measured in single-digit days at most. Any PR that has a hold shall not be merged until the person who requested the hold acks the review, withdraws their hold, or is overruled by a preponderance of maintainers. diff --git a/docs/devel/development.md b/docs/devel/development.md new file mode 100644 index 0000000000000..f750c6116e64f --- /dev/null +++ b/docs/devel/development.md @@ -0,0 +1,179 @@ +# Development Guide + +# Releases and Official Builds + +Official releases are built in Docker containers. Details are [here](build/README.md). You can do simple builds and development with just a local Docker installation. If want to build go locally outside of docker, please continue below. + +## Go development environment + +Kubernetes is written in [Go](http://golang.org) programming language. If you haven't set up Go development environment, please follow [this instruction](http://golang.org/doc/code.html) to install go tool and set up GOPATH. Ensure your version of Go is at least 1.3. + +## Put kubernetes into GOPATH + +We highly recommend to put kubernetes' code into your GOPATH. For example, the following commands will download kubernetes' code under the current user's GOPATH (Assuming there's only one directory in GOPATH.): + +``` +$ echo $GOPATH +/home/user/goproj +$ mkdir -p $GOPATH/src/github.com/GoogleCloudPlatform/ +$ cd $GOPATH/src/github.com/GoogleCloudPlatform/ +$ git clone git@github.com:GoogleCloudPlatform/kubernetes.git +``` + +The commands above will not work if there are more than one directory in ``$GOPATH``. + +(Obviously, clone your own fork of Kubernetes if you plan to do development.) + +## godep and dependency management + +Kubernetes uses [godep](https://github.com/tools/godep) to manage dependencies. It is not strictly required for building Kubernetes but it is required when managing dependencies under the Godeps/ tree, and is required by a number of the build and test scripts. Please make sure that ``godep`` is installed and in your ``$PATH``. + +### Installing godep +There are many ways to build and host go binaries. Here is an easy way to get utilities like ```godep``` installed: + +1. Ensure that [mercurial](http://mercurial.selenic.com/wiki/Download) is installed on your system. (some of godep's dependencies use the mercurial +source control system). Use ```apt-get install mercurial``` or ```yum install mercurial``` on Linux, or [brew.sh](http://brew.sh) on OS X, or download +directly from mercurial. +2. Create a new GOPATH for your tools and install godep: +``` +export GOPATH=$HOME/go-tools +mkdir -p $GOPATH +go get github.com/tools/godep +``` + +3. Add $GOPATH/bin to your path. Typically you'd add this to your ~/.profile: +``` +export GOPATH=$HOME/go-tools +export PATH=$PATH:$GOPATH/bin +``` + +### Using godep +Here is a quick summary of `godep`. `godep` helps manage third party dependencies by copying known versions into Godeps/_workspace. You can use `godep` in three ways: + +1. Use `godep` to call your `go` commands. For example: `godep go test ./...` +2. Use `godep` to modify your `$GOPATH` so that other tools know where to find the dependencies. Specifically: `export GOPATH=$GOPATH:$(godep path)` +3. Use `godep` to copy the saved versions of packages into your `$GOPATH`. This is done with `godep restore`. + +We recommend using options #1 or #2. + +## Hooks + +Before committing any changes, please link/copy these hooks into your .git +directory. This will keep you from accidentally committing non-gofmt'd go code. + +``` +cd kubernetes +ln -s hooks/prepare-commit-msg .git/hooks/prepare-commit-msg +ln -s hooks/commit-msg .git/hooks/commit-msg +``` + +## Unit tests + +``` +cd kubernetes +hack/test-go.sh +``` + +Alternatively, you could also run: + +``` +cd kubernetes +godep go test ./... +``` + +If you only want to run unit tests in one package, you could run ``godep go test`` under the package directory. For example, the following commands will run all unit tests in package kubelet: + +``` +$ cd kubernetes # step into kubernetes' directory. +$ cd pkg/kubelet +$ godep go test +# some output from unit tests +PASS +ok github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet 0.317s +``` + +## Coverage +``` +cd kubernetes +godep go tool cover -html=target/c.out +``` + +## Integration tests + +You need an etcd somewhere in your PATH. To install etcd, run: + +``` +cd kubernetes +hack/install-etcd.sh +sudo ln -s $(pwd)/third_party/etcd/bin/etcd /usr/bin/etcd +``` + +``` +cd kubernetes +hack/test-integration.sh +``` + +## End-to-End tests + +You can run an end-to-end test which will bring up a master and two minions, perform some tests, and then tear everything down. Make sure you have followed the getting started steps for your chosen cloud platform (which might involve changing the `KUBERNETES_PROVIDER` environment variable to something other than "gce". +``` +cd kubernetes +hack/e2e-test.sh +``` + +Pressing control-C should result in an orderly shutdown but if something goes wrong and you still have some VMs running you can force a cleanup with the magical incantation: +``` +hack/e2e-test.sh 1 1 1 +``` + +## Testing out flaky tests +[Instructions here](docs/devel/flaky-tests.md) + +## Add/Update dependencies + +Kubernetes uses [godep](https://github.com/tools/godep) to manage dependencies. To add or update a package, please follow the instructions on [godep's document](https://github.com/tools/godep). + +To add a new package ``foo/bar``: + +- Make sure the kubernetes' root directory is in $GOPATH/github.com/GoogleCloudPlatform/kubernetes +- Run ``godep restore`` to make sure you have all dependancies pulled. +- Download foo/bar into the first directory in GOPATH: ``go get foo/bar``. +- Change code in kubernetes to use ``foo/bar``. +- Run ``godep save ./...`` under kubernetes' root directory. + +To update a package ``foo/bar``: + +- Make sure the kubernetes' root directory is in $GOPATH/github.com/GoogleCloudPlatform/kubernetes +- Run ``godep restore`` to make sure you have all dependancies pulled. +- Update the package with ``go get -u foo/bar``. +- Change code in kubernetes accordingly if necessary. +- Run ``godep update foo/bar`` under kubernetes' root directory. + +## Keeping your development fork in sync + +One time after cloning your forked repo: + +``` +git remote add upstream https://github.com/GoogleCloudPlatform/kubernetes.git +``` + +Then each time you want to sync to upstream: + +``` +git fetch upstream +git rebase upstream/master +``` + +## Regenerating the API documentation + +``` +cd kubernetes/api +sudo docker build -t kubernetes/raml2html . +sudo docker run --name="docgen" kubernetes/raml2html +sudo docker cp docgen:/data/kubernetes.html . +``` + +View the API documentation using htmlpreview (works on your fork, too): +``` +http://htmlpreview.github.io/?https://github.com/GoogleCloudPlatform/kubernetes/blob/master/api/kubernetes.html +``` diff --git a/docs/devel/flaky-tests.md b/docs/devel/flaky-tests.md new file mode 100644 index 0000000000000..d2cc8fadaf4cc --- /dev/null +++ b/docs/devel/flaky-tests.md @@ -0,0 +1,52 @@ +# Hunting flaky tests in Kubernetes +Sometimes unit tests are flaky. This means that due to (usually) race conditions, they will occasionally fail, even though most of the time they pass. + +We have a goal of 99.9% flake free tests. This means that there is only one flake in one thousand runs of a test. + +Running a test 1000 times on your own machine can be tedious and time consuming. Fortunately, there is a better way to achieve this using Kubernetes. + +_Note: these instructions are mildly hacky for now, as we get run once semantics and logging they will get better_ + +There is a testing image ```brendanburns/flake``` up on the docker hub. We will use this image to test our fix. + +Create a replication controller with the following config: +```yaml +id: flakeController +desiredState: + replicas: 24 + replicaSelector: + name: flake + podTemplate: + desiredState: + manifest: + version: v1beta1 + id: "" + volumes: [] + containers: + - name: flake + image: brendanburns/flake + env: + - name: TEST_PACKAGE + value: pkg/tools + - name: REPO_SPEC + value: https://github.com/GoogleCloudPlatform/kubernetes + restartpolicy: {} + labels: + name: flake +labels: + name: flake +``` + +```./cluster/kubecfg.sh -c controller.yaml create replicaControllers``` + +This will spin up 100 instances of the test. They will run to completion, then exit, the kubelet will restart them, eventually you will have sufficient +runs for your purposes, and you can stop the replication controller: + +```sh +./cluster/kubecfg.sh stop flakeController +./cluster/kubecfg.sh rm flakeController +``` + +Now examine the machines with ```docker ps -a``` and look for tasks that exited with non-zero exit codes (ignore those that exited -1, since that's what happens when you stop the replica controller) + +Happy flake hunting! diff --git a/docs/devel/releasing.dot b/docs/devel/releasing.dot new file mode 100644 index 0000000000000..fe8124c36da71 --- /dev/null +++ b/docs/devel/releasing.dot @@ -0,0 +1,113 @@ +// Build it with: +// $ dot -Tsvg releasing.dot >releasing.svg + +digraph tagged_release { + size = "5,5" + // Arrows go up. + rankdir = BT + subgraph left { + // Group the left nodes together. + ci012abc -> pr101 -> ci345cde -> pr102 + style = invis + } + subgraph right { + // Group the right nodes together. + version_commit -> dev_commit + style = invis + } + { // Align the version commit and the info about it. + rank = same + // Align them with pr101 + pr101 + version_commit + // release_info shows the change in the commit. + release_info + } + { // Align the dev commit and the info about it. + rank = same + // Align them with 345cde + ci345cde + dev_commit + dev_info + } + // Join the nodes from subgraph left. + pr99 -> ci012abc + pr102 -> pr100 + // Do the version node. + pr99 -> version_commit + dev_commit -> pr100 + tag -> version_commit + pr99 [ + label = "Merge PR #99" + shape = box + fillcolor = "#ccccff" + style = "filled" + fontname = "Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" + ]; + ci012abc [ + label = "012abc" + shape = circle + fillcolor = "#ffffcc" + style = "filled" + fontname = "Consolas, Liberation Mono, Menlo, Courier, monospace" + ]; + pr101 [ + label = "Merge PR #101" + shape = box + fillcolor = "#ccccff" + style = "filled" + fontname = "Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" + ]; + ci345cde [ + label = "345cde" + shape = circle + fillcolor = "#ffffcc" + style = "filled" + fontname = "Consolas, Liberation Mono, Menlo, Courier, monospace" + ]; + pr102 [ + label = "Merge PR #102" + shape = box + fillcolor = "#ccccff" + style = "filled" + fontname = "Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" + ]; + version_commit [ + label = "678fed" + shape = circle + fillcolor = "#ccffcc" + style = "filled" + fontname = "Consolas, Liberation Mono, Menlo, Courier, monospace" + ]; + dev_commit [ + label = "456dcb" + shape = circle + fillcolor = "#ffffcc" + style = "filled" + fontname = "Consolas, Liberation Mono, Menlo, Courier, monospace" + ]; + pr100 [ + label = "Merge PR #100" + shape = box + fillcolor = "#ccccff" + style = "filled" + fontname = "Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" + ]; + release_info [ + label = "pkg/version/base.go:\ngitVersion = \"v0.5\";" + shape = none + fontname = "Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" + ]; + dev_info [ + label = "pkg/version/base.go:\ngitVersion = \"v0.5-dev\";" + shape = none + fontname = "Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" + ]; + tag [ + label = "$ git tag -a v0.5" + fillcolor = "#ffcccc" + style = "filled" + fontname = "Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" + ]; +} + diff --git a/docs/devel/releasing.md b/docs/devel/releasing.md new file mode 100644 index 0000000000000..4cdf88273c680 --- /dev/null +++ b/docs/devel/releasing.md @@ -0,0 +1,152 @@ +# Releasing Kubernetes + +This document explains how to create a Kubernetes release (as in version) and +how the version information gets embedded into the built binaries. + +## Origin of the Sources + +Kubernetes may be built from either a git tree (using `hack/build-go.sh`) or +from a tarball (using either `hack/build-go.sh` or `go install`) or directly by +the Go native build system (using `go get`). + +When building from git, we want to be able to insert specific information about +the build tree at build time. In particular, we want to use the output of `git +describe` to generate the version of Kubernetes and the status of the build +tree (add a `-dirty` prefix if the tree was modified.) + +When building from a tarball or using the Go build system, we will not have +access to the information about the git tree, but we still want to be able to +tell whether this build corresponds to an exact release (e.g. v0.3) or is +between releases (e.g. at some point in development between v0.3 and v0.4). + +## Version Number Format + +In order to account for these use cases, there are some specific formats that +may end up representing the Kubernetes version. Here are a few examples: + +- **v0.5**: This is official version 0.5 and this version will only be used + when building from a clean git tree at the v0.5 git tag, or from a tree + extracted from the tarball corresponding to that specific release. +- **v0.5-15-g0123abcd4567**: This is the `git describe` output and it indicates + that we are 15 commits past the v0.5 release and that the SHA1 of the commit + where the binaries were built was `0123abcd4567`. It is only possible to have + this level of detail in the version information when building from git, not + when building from a tarball. +- **v0.5-15-g0123abcd4567-dirty** or **v0.5-dirty**: The extra `-dirty` prefix + means that the tree had local modifications or untracked files at the time of + the build, so there's no guarantee that the source code matches exactly the + state of the tree at the `0123abcd4567` commit or at the `v0.5` git tag + (resp.) +- **v0.5-dev**: This means we are building from a tarball or using `go get` or, + if we have a git tree, we are using `go install` directly, so it is not + possible to inject the git version into the build information. Additionally, + this is not an official release, so the `-dev` prefix indicates that the + version we are building is after `v0.5` but before `v0.6`. (There is actually + an exception where a commit with `v0.5-dev` is not present on `v0.6`, see + later for details.) + +## Injecting Version into Binaries + +In order to cover the different build cases, we start by providing information +that can be used when using only Go build tools or when we do not have the git +version information available. + +To be able to provide a meaningful version in those cases, we set the contents +of variables in a Go source file that will be used when no overrides are +present. + +We are using `pkg/version/base.go` as the source of versioning in absence of +information from git. Here is a sample of that file's contents: + +``` + var ( + gitVersion string = "v0.4-dev" // version from git, output of $(git describe) + gitCommit string = "" // sha1 from git, output of $(git rev-parse HEAD) + ) +``` + +This means a build with `go install` or `go get` or a build from a tarball will +yield binaries that will identify themselves as `v0.4-dev` and will not be able +to provide you with a SHA1. + +To add the extra versioning information when building from git, the +`hack/build-go.sh` script will gather that information (using `git describe` and +`git rev-parse`) and then create a `-ldflags` string to pass to `go install` and +tell the Go linker to override the contents of those variables at build time. It +can, for instance, tell it to override `gitVersion` and set it to +`v0.4-13-g4567bcdef6789-dirty` and set `gitCommit` to `4567bcdef6789...` which +is the complete SHA1 of the (dirty) tree used at build time. + +## Handling Official Versions + +Handling official versions from git is easy, as long as there is an annotated +git tag pointing to a specific version then `git describe` will return that tag +exactly which will match the idea of an official version (e.g. `v0.5`). + +Handling it on tarballs is a bit harder since the exact version string must be +present in `pkg/version/base.go` for it to get embedded into the binaries. But +simply creating a commit with `v0.5` on its own would mean that the commits +coming after it would also get the `v0.5` version when built from tarball or `go +get` while in fact they do not match `v0.5` (the one that was tagged) exactly. + +To handle that case, creating a new release should involve creating two adjacent +commits where the first of them will set the version to `v0.5` and the second +will set it to `v0.5-dev`. In that case, even in the presence of merges, there +will be a single comit where the exact `v0.5` version will be used and all +others around it will either have `v0.4-dev` or `v0.5-dev`. + +The diagram below illustrates it. + +![Diagram of git commits involved in the release](./releasing.png) + +After working on `v0.4-dev` and merging PR 99 we decide it is time to release +`v0.5`. So we start a new branch, create one commit to update +`pkg/version/base.go` to include `gitVersion = "v0.5"` and `git commit` it. + +We test it and make sure everything is working as expected. + +Before sending a PR for it, we create a second commit on that same branch, +updating `pkg/version/base.go` to include `gitVersion = "v0.5-dev"`. That will +ensure that further builds (from tarball or `go install`) on that tree will +always include the `-dev` prefix and will not have a `v0.5` version (since they +do not match the official `v0.5` exactly.) + +We then send PR 100 with both commits in it. + +Once the PR is accepted, we can use `git tag -a` to create an annotated tag +*pointing to the one commit* that has `v0.5` in `pkg/version/base.go` and push +it to GitHub. (Unfortunately GitHub tags/releases are not annotated tags, so +this needs to be done from a git client and pushed to GitHub using SSH.) + +## Parallel Commits + +While we are working on releasing `v0.5`, other development takes place and +other PRs get merged. For instance, in the example above, PRs 101 and 102 get +merged to the master branch before the versioning PR gets merged. + +This is not a problem, it is only slightly inaccurate that checking out the tree +at commit `012abc` or commit `345cde` or at the commit of the merges of PR 101 +or 102 will yield a version of `v0.4-dev` *but* those commits are not present in +`v0.5`. + +In that sense, there is a small window in which commits will get a +`v0.4-dev` or `v0.4-N-gXXX` label and while they're indeed later than `v0.4` +but they are not really before `v0.5` in that `v0.5` does not contain those +commits. + +Unfortunately, there is not much we can do about it. On the other hand, other +projects seem to live with that and it does not really become a large problem. + +As an example, Docker commit a327d9b91edf has a `v1.1.1-N-gXXX` label but it is +not present in Docker `v1.2.0`: + +``` + $ git describe a327d9b91edf + v1.1.1-822-ga327d9b91edf + + $ git log --oneline v1.2.0..a327d9b91edf + a327d9b91edf Fix data space reporting from Kb/Mb to KB/MB + + (Non-empty output here means the commit is not present on v1.2.0.) +``` + diff --git a/docs/devel/releasing.png b/docs/devel/releasing.png new file mode 100644 index 0000000000000..935628deddc5b Binary files /dev/null and b/docs/devel/releasing.png differ diff --git a/docs/devel/releasing.svg b/docs/devel/releasing.svg new file mode 100644 index 0000000000000..f703e6e2ac969 --- /dev/null +++ b/docs/devel/releasing.svg @@ -0,0 +1,113 @@ + + + + + + +tagged_release + + +ci012abc + +012abc + + +pr101 + +Merge PR #101 + + +ci012abc->pr101 + + + + +ci345cde + +345cde + + +pr101->ci345cde + + + + +pr102 + +Merge PR #102 + + +ci345cde->pr102 + + + + +pr100 + +Merge PR #100 + + +pr102->pr100 + + + + +version_commit + +678fed + + +dev_commit + +456dcb + + +version_commit->dev_commit + + + + +dev_commit->pr100 + + + + +release_info +pkg/version/base.go: +gitVersion = "v0.5"; + + +dev_info +pkg/version/base.go: +gitVersion = "v0.5-dev"; + + +pr99 + +Merge PR #99 + + +pr99->ci012abc + + + + +pr99->version_commit + + + + +tag + +$ git tag -a v0.5 + + +tag->version_commit + + + + +