Skip to content

Commit

Permalink
Use rsync to get source into build container
Browse files Browse the repository at this point in the history 
We also add "version" to all docker images and containers

This version is to be incremented manually when we change the shape of the build
image (like changing the golang version or the set of volumes in the data
container).  This will delete all older versions of images and containers when
the version is different.
  • Loading branch information
@jbeda
jbeda committed Oct 4, 2016
1 parent 22b7d90 commit dc586ea
Show file tree
Hide file tree
Showing 12 changed files with 519 additions and 266 deletions.
3 changes: 3 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ network_closure.sh

.tags*

# Version file for dockerized build
.dockerized-kube-version-defs

# Web UI
/www/master/node_modules/
/www/master/npm-debug.log
Expand Down
1 change: 1 addition & 0 deletions build/BUILD_IMAGE_VERSION
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3
113 changes: 67 additions & 46 deletions build/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,19 @@ Building Kubernetes is easy if you take advantage of the containerized build env

## Requirements

1. Docker, using one of the two following configurations:
1. Docker, using one of the following configurations:
1. **Mac OS X** You can either use Docker for Mac or docker-machine. See installation instructions [here](https://docs.docker.com/installation/mac/).
**Note**: You will want to set the Docker VM to have at least 3GB of initial memory or building will likely fail. (See: [#11852]( http://issue.k8s.io/11852)) and do not `make quick-release` from `/tmp/` (See: [#14773]( https://github.com/kubernetes/kubernetes/issues/14773))
2. **Linux with local Docker** Install Docker according to the [instructions](https://docs.docker.com/installation/#installation) for your OS. The scripts here assume that they are using a local Docker server and that they can "reach around" docker and grab results directly from the file system.
2. [Python](https://www.python.org)
3. **Optional** [Google Cloud SDK](https://developers.google.com/cloud/sdk/)
**Note**: You will want to set the Docker VM to have at least 3GB of initial memory or building will likely fail. (See: [#11852]( http://issue.k8s.io/11852)).
2. **Linux with local Docker** Install Docker according to the [instructions](https://docs.docker.com/installation/#installation) for your OS.
3. **Remote Docker engine** Use a big machine in the cloud to build faster. This is a little trickier so look at the section later on.
2. **Optional** [Google Cloud SDK](https://developers.google.com/cloud/sdk/)

You must install and configure Google Cloud SDK if you want to upload your release to Google Cloud Storage and may safely omit this otherwise.

## Overview

While it is possible to build Kubernetes using a local golang installation, we have a build process that runs in a Docker container. This simplifies initial set up and provides for a very consistent build and test environment.

There is also early support for building Docker "run" containers

## Key scripts

The following scripts are found in the `build/` directory. Note that all scripts must be run from the Kubernetes root directory.
Expand All @@ -29,10 +27,68 @@ The following scripts are found in the `build/` directory. Note that all scripts
* `build/run.sh make test`: Run all unit tests
* `build/run.sh make test-integration`: Run integration test
* `build/run.sh make test-cmd`: Run CLI tests
* `build/copy-output.sh`: This will copy the contents of `_output/dockerized/bin` from any the Docker container to the local `_output/dockerized/bin`.
* `build/make-clean.sh`: Clean out the contents of `_output/dockerized`, remove any container images and the data container
* `build/shell.sh`: Drop into a `bash` shell in a build container with a snapshot of the current repo code.
* `build/release.sh`: Build everything, test it, and (optionally) upload the results to a GCS bucket.
* `build/copy-output.sh`: This will copy the contents of `_output/dockerized/bin` from the Docker container to the local `_output/dockerized/bin`. It will also copy out specific file patterns that are generated as part of the build process. This is run automatically as part of `build/run.sh`.
* `build/make-clean.sh`: Clean out the contents of `_output`, remove any locally built container images and remove the data container.
* `/build/shell.sh`: Drop into a `bash` shell in a build container with a snapshot of the current repo code.

## Basic Flow

The scripts directly under `build/` are used to build and test. They will ensure that the `kube-build` Docker image is built (based on `build/build-image/Dockerfile`) and then execute the appropriate command in that container. These scripts will both ensure that the right data is cached from run to run for incremental builds and will copy the results back out of the container.

The `kube-build` container image is built by first creating a "context" directory in `_output/images/build-image`. It is done there instead of at the root of the Kubernetes repo to minimize the amount of data we need to package up when building the image.

There are 3 different containers instances that are run from this image. The first is a "data" container to store all data that needs to persist across to support incremental builds. Next there is an "rsync" container that is used to transfer data in and out to the data container. Lastly there is a "build" container that is used for actually doing build actions. The data container persists across runs while the rsync and build containers are deleted after each use.

`rsync` is used transparently behind the scenes to efficiently move data in and and of the container. This will use an ephemeral port picked by Docker. You can modify this by setting the `KUBE_RSYNC_PORT` env variable.

All Docker names are suffixed with a hash derived from the file path (to allow concurrent usage on things like CI machines) and a version number. When the version number changes all state is cleared and clean build is started. This allows the build infrastructure to be changed and signal to CI systems that old artifacts need to be deleted.

## Proxy Settings

If you are behind a proxy and you are letting these scripts use `docker-machine` to set up your local VM for you on macOS, you need to export proxy settings for kubernetes build, the following environment variables should be defined.

```
export KUBERNETES_HTTP_PROXY=http://username:password@proxyaddr:proxyport
export KUBERNETES_HTTPS_PROXY=https://username:password@proxyaddr:proxyport
```

Optionally, you can specify addresses of no proxy for kubernetes build, for example

```
export KUBERNETES_NO_PROXY=127.0.0.1
```

If you are using sudo to make kubernetes build for example make quick-release, you need run `sudo -E make quick-release` to pass the environment variables.

## Really Remote Docker Engine

It is possible to use a Docker Engine that is running remotely (under your desk or in the cloud). Docker must be configured to connect to that machine and the local rsync port must be forwarded (via SSH or nc) from localhost to the remote machine.

To do this easily with GCE and `docker-machine`, do something like this:
```
# Create the remote docker machine on GCE. This is a pretty beefy machine with SSD disk.
KUBE_BUILD_VM=k8s-build
KUBE_BUILD_GCE_PROJECT=<project>
docker-machine create \
--driver=google \
--google-project=${KUBE_BUILD_GCE_PROJECT} \
--google-zone=us-west1-a \
--google-machine-type=n1-standard-8 \
--google-disk-size=50 \
--google-disk-type=pd-ssd \
${KUBE_BUILD_VM}
# Set up local docker to talk to that machine
eval $(docker-machine env ${KUBE_BUILD_VM})
# Pin down the port that rsync will be exposed on on the remote machine
export KUBE_RSYNC_PORT=8370
# forward local 8730 to that machine so that rsync works
docker-machine ssh ${KUBE_BUILD_VM} -L ${KUBE_RSYNC_PORT}:localhost:8730 -N &
```

Look at `docker-machine stop`, `docker-machine start` and `docker-machine rm` to manage this VM.

## Releasing

Expand Down Expand Up @@ -63,41 +119,6 @@ Env Variable | Default | Description
`KUBE_GCS_NO_CACHING` | `y` | Disable HTTP caching of GCS release artifacts. By default GCS will cache public objects for up to an hour. When doing "devel" releases this can cause problems.
`KUBE_GCS_DOCKER_REG_PREFIX` | `docker-reg` | *Experimental* When uploading docker images, the bucket that backs the registry.

## Basic Flow

The scripts directly under `build/` are used to build and test. They will ensure that the `kube-build` Docker image is built (based on `build/build-image/Dockerfile`) and then execute the appropriate command in that container. If necessary (for Mac OS X), the scripts will also copy results out.

The `kube-build` container image is built by first creating a "context" directory in `_output/images/build-image`. It is done there instead of at the root of the Kubernetes repo to minimize the amount of data we need to package up when building the image.

Everything in `build/build-image/` is meant to be run inside of the container. If it doesn't think it is running in the container it'll throw a warning. While you can run some of that stuff outside of the container, it wasn't built to do so.

When building final release tars, they are first staged into `_output/release-stage` before being tar'd up and put into `_output/release-tars`.

## Proxy Settings


If you are behind a proxy, you need to export proxy settings for kubernetes build, the following environment variables should be defined.

```
export KUBERNETES_HTTP_PROXY=http://username:password@proxyaddr:proxyport
export KUBERNETES_HTTPS_PROXY=https://username:password@proxyaddr:proxyport
```

Optionally, you can specify addresses of no proxy for kubernetes build, for example

```
export KUBERNETES_NO_PROXY=127.0.0.1
```

If you are using sudo to make kubernetes build for example make quick-release, you need run `sudo -E make quick-release` to pass the environment variables.

## TODOs

These are in no particular order

* [X] Harmonize with scripts in `hack/`. How much do we support building outside of Docker and these scripts?
* [X] Deprecate/replace most of the stuff in the hack/
* [ ] Finish support for the Dockerized runtime. Issue [#19](http://issue.k8s.io/19). A key issue here is to make this fast/light enough that we can use it for development workflows.


[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/build/README.md?pixel)]()
20 changes: 8 additions & 12 deletions build/build-image/Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
# This file creates a standard build environment for building Kubernetes
FROM gcr.io/google_containers/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG

ADD localtime /etc/localtime

# Mark this as a kube-build container
RUN touch /kube-build-image

Expand All @@ -25,19 +27,13 @@ RUN chmod -R a+rwx /usr/local/go/pkg ${K8S_PATCHED_GOROOT}/pkg
# of operations.
ENV HOME /go/src/k8s.io/kubernetes
WORKDIR ${HOME}
# We have to mkdir the dirs we need, or else Docker will create them when we
# mount volumes, and it will create them with root-only permissions. The
# explicit chmod of _output is required, but I can't really explain why.
RUN mkdir -p ${HOME} ${HOME}/_output \
&& chmod -R a+rwx ${HOME} ${HOME}/_output

# Propagate the git tree version into the build image
ADD kube-version-defs /kube-version-defs
RUN chmod a+r /kube-version-defs
ENV KUBE_GIT_VERSION_FILE /kube-version-defs

# Make output from the dockerized build go someplace else
ENV KUBE_OUTPUT_SUBPATH _output/dockerized

# Upload Kubernetes source
ADD kube-source.tar.gz /go/src/k8s.io/kubernetes/
# Pick up version stuff here as we don't copy our .git over.
ENV KUBE_GIT_VERSION_FILE ${HOME}/.dockerized-kube-version-defs

ADD rsyncd.password /
RUN chmod a+r /rsyncd.password
ADD rsyncd.sh /
83 changes: 83 additions & 0 deletions build/build-image/rsyncd.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
#!/bin/bash

# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This script will set up and run rsyncd to allow data to move into and out of
# our dockerized build system. This is used for syncing sources and changes of
# sources into the docker-build-container. It is also used to transfer built binaries
# and generated files back out.
#
# When run as root (rare) it'll preserve the file ids as sent from the client.
# Usually it'll be run as non-dockerized UID/GID and end up translating all file
# ownership to that.


set -o errexit
set -o nounset
set -o pipefail

# The directory that gets sync'd
VOLUME=${HOME}

# Assume that this is running in Docker on a bridge. Allow connections from
# anything on the local subnet.
ALLOW=$(ip route | awk '/^default via/ { reg = "^[0-9./]+ dev "$5 } ; $0 ~ reg { print $1 }')

CONFDIR="/tmp/rsync.k8s"
PIDFILE="${CONFDIR}/rsyncd.pid"
CONFFILE="${CONFDIR}/rsyncd.conf"
SECRETS="${CONFDIR}/rsyncd.secrets"

mkdir -p "${CONFDIR}"

if [[ -f "${PIDFILE}" ]]; then
PID=$(cat "${PIDFILE}")
echo "Cleaning up old PID file: ${PIDFILE}"
kill $PID &> /dev/null || true
rm "${PIDFILE}"
fi

PASSWORD=$(</rsyncd.password)

cat <<EOF >"${SECRETS}"
k8s:${PASSWORD}
EOF
chmod go= "${SECRETS}"

USER_CONFIG=
if [[ "$(id -u)" == "0" ]]; then
USER_CONFIG=" uid = 0"$'\n'" gid = 0"
fi

cat <<EOF >"${CONFFILE}"
pid file = ${PIDFILE}
use chroot = no
log file = /dev/stdout
reverse lookup = no
munge symlinks = no
port = 8730
[k8s]
numeric ids = true
$USER_CONFIG
hosts deny = *
hosts allow = ${ALLOW}
auth users = k8s
secrets file = ${SECRETS}
read only = false
path = ${VOLUME}
filter = - /.make/ - /.git/ - /_tmp/
EOF

exec /usr/bin/rsync --no-detach --daemon --config="${CONFFILE}" "$@"
Loading

0 comments on commit dc586ea

Please sign in to comment.