Add streaming command execution & port forwarding

Add streaming command execution & port forwarding via HTTP connection upgrades (currently using SPDY).
satnam6502 · Feb 20, 2015 · 5bd0e9a · 5bd0e9a
1 parent 25d38c1
commit 5bd0e9a
Show file tree

Hide file tree

Showing 45 changed files with 4,438 additions and 156 deletions.
diff --git a/cmd/gendocs/gen_kubectl_docs.go b/cmd/gendocs/gen_kubectl_docs.go
@@ -164,7 +164,8 @@ func main() {
 	// Set environment variables used by kubectl so the output is consistent,
 	// regardless of where we run.
 	os.Setenv("HOME", "/home/username")
-	kubectl := cmd.NewFactory(nil).NewKubectlCommand(ioutil.Discard)
+	//TODO os.Stdin should really be something like ioutil.Discard, but a Reader
+	kubectl := cmd.NewFactory(nil).NewKubectlCommand(os.Stdin, ioutil.Discard, ioutil.Discard)
 	genMarkdown(kubectl, "", docsDir)
 	for _, c := range kubectl.Commands() {
 		genMarkdown(c, "kubectl", docsDir)

diff --git a/cmd/genman/gen_kubectl_man.go b/cmd/genman/gen_kubectl_man.go
@@ -62,7 +62,8 @@ func main() {
 	// Set environment variables used by kubectl so the output is consistent,
 	// regardless of where we run.
 	os.Setenv("HOME", "/home/username")
-	kubectl := cmd.NewFactory(nil).NewKubectlCommand(ioutil.Discard)
+	//TODO os.Stdin should really be something like ioutil.Discard, but a Reader
+	kubectl := cmd.NewFactory(nil).NewKubectlCommand(os.Stdin, ioutil.Discard, ioutil.Discard)
 	genMarkdown(kubectl, "", docsDir)
 	for _, c := range kubectl.Commands() {
 		genMarkdown(c, "kubectl", docsDir)

diff --git a/cmd/kubectl/kubectl.go b/cmd/kubectl/kubectl.go
@@ -25,7 +25,7 @@ import (
 
 func main() {
 	runtime.GOMAXPROCS(runtime.NumCPU())
-	cmd := cmd.NewFactory(nil).NewKubectlCommand(os.Stdout)
+	cmd := cmd.NewFactory(nil).NewKubectlCommand(os.Stdin, os.Stdout, os.Stderr)
 	if err := cmd.Execute(); err != nil {
 		os.Exit(1)
 	}

diff --git a/docs/design/command_execution_port_forwarding.md b/docs/design/command_execution_port_forwarding.md
@@ -0,0 +1,144 @@
+# Container Command Execution & Port Forwarding in Kubernetes
+
+## Abstract
+
+This describes an approach for providing support for:
+
+- executing commands in containers, with stdin/stdout/stderr streams attached
+- port forwarding to containers
+
+## Background
+
+There are several related issues/PRs:
+
+- [Support attach](https://github.com/GoogleCloudPlatform/kubernetes/issues/1521)
+- [Real container ssh](https://github.com/GoogleCloudPlatform/kubernetes/issues/1513)
+- [Provide easy debug network access to services](https://github.com/GoogleCloudPlatform/kubernetes/issues/1863)
+- [OpenShift container command execution proposal](https://github.com/openshift/origin/pull/576)
+
+## Motivation
+
+Users and administrators are accustomed to being able to access their systems
+via SSH to run remote commands, get shell access, and do port forwarding.
+
+Supporting SSH to containers in Kubernetes is a difficult task. You must
+specify a "user" and a hostname to make an SSH connection, and `sshd` requires
+real users (resolvable by NSS and PAM). Because a container belongs to a pod,
+and the pod belongs to a namespace, you need to specify namespace/pod/container
+to uniquely identify the target container. Unfortunately, a
+namespace/pod/container is not a real user as far as SSH is concerned. Also,
+most Linux systems limit user names to 32 characters, which is unlikely to be
+large enough to contain namespace/pod/container. We could devise some scheme to
+map each namespace/pod/container to a 32-character user name, adding entries to
+`/etc/passwd` (or LDAP, etc.) and keeping those entries fully in sync all the
+time. Alternatively, we could write custom NSS and PAM modules that allow the
+host to resolve a namespace/pod/container to a user without needing to keep
+files or LDAP in sync.
+
+As an alternative to SSH, we are using a multiplexed streaming protocol that
+runs on top of HTTP. There are no requirements about users being real users,
+nor is there any limitation on user name length, as the protocol is under our
+control. The only downside is that standard tooling that expects to use SSH
+won't be able to work with this mechanism, unless adapters can be written.
+
+## Constraints and Assumptions
+
+- SSH support is not currently in scope
+- CGroup confinement is ultimately desired, but implementing that support is not currently in scope
+- SELinux confinement is ultimately desired, but implementing that support is not currently in scope
+
+## Use Cases
+
+- As a user of a Kubernetes cluster, I want to run arbitrary commands in a container, attaching my local stdin/stdout/stderr to the container
+- As a user of a Kubernetes cluster, I want to be able to connect to local ports on my computer and have them forwarded to ports in the container
+
+## Process Flow
+
+### Remote Command Execution Flow
+1. The client connects to the Kubernetes Master to initiate a remote command execution
+request
+2. The Master proxies the request to the Kubelet where the container lives
+3. The Kubelet executes nsenter + the requested command and streams stdin/stdout/stderr back and forth between the client and the container
+
+### Port Forwarding Flow
+1. The client connects to the Kubernetes Master to initiate a remote command execution
+request
+2. The Master proxies the request to the Kubelet where the container lives
+3. The client listens on each specified local port, awaiting local connections
+4. The client connects to one of the local listening ports
+4. The client notifies the Kubelet of the new connection
+5. The Kubelet executes nsenter + socat and streams data back and forth between the client and the port in the container
+
+
+## Design Considerations
+
+### Streaming Protocol
+
+The current multiplexed streaming protocol used is SPDY. This is not the
+long-term desire, however. As soon as there is viable support for HTTP/2 in Go,
+we will switch to that.
+
+### Master as First Level Proxy
+
+Clients should not be allowed to communicate directly with the Kubelet for
+security reasons. Therefore, the Master is currently the only suggested entry
+point to be used for remote command execution and port forwarding. This is not
+necessarily desirable, as it means that all remote command execution and port
+forwarding traffic must travel through the Master, potentially impacting other
+API requests.
+
+In the future, it might make more sense to retrieve an authorization token from
+the Master, and then use that token to initiate a remote command execution or
+port forwarding request with a load balanced proxy service dedicated to this
+functionality. This would keep the streaming traffic out of the Master.
+
+### Kubelet as Backend Proxy
+
+The kubelet is currently responsible for handling remote command execution and
+port forwarding requests. Just like with the Master described above, this means
+that all remote command execution and port forwarding streaming traffic must
+travel through the Kubelet, which could result in a degraded ability to service
+other requests.
+
+In the future, it might make more sense to use a separate service on the node.
+
+Alternatively, we could possibly inject a process into the container that only
+listens for a single request, expose that process's listening port on the node,
+and then issue a redirect to the client such that it would connect to the first
+level proxy, which would then proxy directly to the injected process's exposed
+port. This would minimize the amount of proxying that takes place.
+
+### Scalability
+
+There are at least 2 different ways to execute a command in a container:
+`docker exec` and `nsenter`. While `docker exec` might seem like an easier and
+more obvious choice, it has some drawbacks.
+
+#### `docker exec`
+
+We could expose `docker exec` (i.e. have Docker listen on an exposed TCP port
+on the node), but this would require proxying from the edge and securing the
+Docker API. `docker exec` calls go through the Docker daemon, meaning that all
+stdin/stdout/stderr traffic is proxied through the Daemon, adding an extra hop.
+Additionally, you can't isolate 1 malicious `docker exec` call from normal
+usage, meaning an attacker could initiate a denial of service or other attack
+and take down the Docker daemon, or the node itself.
+
+We expect remote command execution and port forwarding requests to be long
+running and/or high bandwidth operations, and routing all the streaming data
+through the Docker daemon feels like a bottleneck we can avoid.
+
+#### `nsenter`
+
+The implementation currently uses `nsenter` to run commands in containers,
+joining the appropriate container namespaces. `nsenter` runs directly on the
+node and is not proxied through any single daemon process.
+
+### Security
+
+Authentication and authorization hasn't specifically been tested yet with this
+functionality. We need to make sure that users are not allowed to execute
+remote commands or do port forwarding to containers they aren't allowed to
+access.
+
+Additional work is required to ensure that multiple command execution or port forwarding connections from different clients are not able to see each other's data. This can most likely be achieved via SELinux labeling and unique process contexts.
diff --git a/docs/kubectl-exec.md b/docs/kubectl-exec.md
@@ -0,0 +1,60 @@
+## kubectl exec
+
+Execute a command in a container.
+
+### Synopsis
+
+Execute a command in a container.
+Examples:
+  $ kubectl exec -p 123456-7890 -c ruby-container date
+  <returns output from running 'date' in ruby-container from pod 123456-7890>
+
+  $ kubectl exec -p 123456-7890 -c ruby-container -i -t -- bash -il
+  <switches to raw terminal mode, sends stdin to 'bash' in ruby-container from
+   pod 123456-780 and sends stdout/stderr from 'bash' back to the client
+
+kubectl exec -p <pod> -c <container> -- <command> [<args...>]
+
+### Options
+
+```
+  -c, --container="": Container name
+  -p, --pod="": Pod name
+  -i, --stdin=false: Pass stdin to the container
+  -t, --tty=false: Stdin is a TTY
+```
+
+### Options inherrited from parent commands
+
+```
+      --alsologtostderr=false: log to standard error as well as files
+      --api-version="": The API version to use when talking to the server
+  -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https.
+      --certificate-authority="": Path to a cert. file for the certificate authority.
+      --client-certificate="": Path to a client key file for TLS.
+      --client-key="": Path to a client key file for TLS.
+      --cluster="": The name of the kubeconfig cluster to use
+      --context="": The name of the kubeconfig context to use
+  -h, --help=false: help for kubectl
+      --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
+      --kubeconfig="": Path to the kubeconfig file to use for CLI requests.
+      --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace
+      --log_dir=: If non-empty, write log files in this directory
+      --log_flush_frequency=5s: Maximum number of seconds between log flushes
+      --logtostderr=true: log to standard error instead of files
+      --match-server-version=false: Require server version to match client version
+      --namespace="": If present, the namespace scope for this CLI request.
+      --password="": Password for basic authentication to the API server.
+  -s, --server="": The address and port of the Kubernetes API server
+      --stderrthreshold=2: logs at or above this threshold go to stderr
+      --token="": Bearer token for authentication to the API server.
+      --user="": The name of the kubeconfig user to use
+      --username="": Username for basic authentication to the API server.
+      --v=0: log level for V logs
+      --validate=false: If true, use a schema to validate the input before sending it
+      --vmodule=: comma-separated list of pattern=N settings for file-filtered logging
+```
+
+### SEE ALSO
+* [kubectl](kubectl.md)
+
diff --git a/docs/kubectl-port-forward.md b/docs/kubectl-port-forward.md
@@ -0,0 +1,64 @@
+## kubectl port-forward
+
+Forward 1 or more local ports to a pod.
+
+### Synopsis
+
+Forward 1 or more local ports to a pod.
+Examples:
+  $ kubectl port-forward -p mypod 5000 6000
+  <listens on ports 5000 and 6000 locally, forwarding data to/from ports 5000
+   and 6000 in the pod>
+
+	$ kubectl port-forward -p mypod 8888:5000
+	<listens on port 8888 locally, forwarding to 5000 in the pod>
+
+	$ kubectl port-forward -p mypod :5000
+	<listens on a random port locally, forwarding to 5000 in the pod>
+
+	$ kubectl port-forward -p mypod 0:5000
+	<listens on a random port locally, forwarding to 5000 in the pod>
+
+
+kubectl port-forward -p <pod> [<local port>:]<remote port> [<port>...]
+
+### Options
+
+```
+  -p, --pod="": Pod name
+```
+
+### Options inherrited from parent commands
+
+```
+      --alsologtostderr=false: log to standard error as well as files
+      --api-version="": The API version to use when talking to the server
+  -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https.
+      --certificate-authority="": Path to a cert. file for the certificate authority.
+      --client-certificate="": Path to a client key file for TLS.
+      --client-key="": Path to a client key file for TLS.
+      --cluster="": The name of the kubeconfig cluster to use
+      --context="": The name of the kubeconfig context to use
+  -h, --help=false: help for kubectl
+      --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
+      --kubeconfig="": Path to the kubeconfig file to use for CLI requests.
+      --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace
+      --log_dir=: If non-empty, write log files in this directory
+      --log_flush_frequency=5s: Maximum number of seconds between log flushes
+      --logtostderr=true: log to standard error instead of files
+      --match-server-version=false: Require server version to match client version
+      --namespace="": If present, the namespace scope for this CLI request.
+      --password="": Password for basic authentication to the API server.
+  -s, --server="": The address and port of the Kubernetes API server
+      --stderrthreshold=2: logs at or above this threshold go to stderr
+      --token="": Bearer token for authentication to the API server.
+      --user="": The name of the kubeconfig user to use
+      --username="": Username for basic authentication to the API server.
+      --v=0: log level for V logs
+      --validate=false: If true, use a schema to validate the input before sending it
+      --vmodule=: comma-separated list of pattern=N settings for file-filtered logging
+```
+
+### SEE ALSO
+* [kubectl](kubectl.md)
+
diff --git a/docs/kubectl.md b/docs/kubectl.md
@@ -54,6 +54,8 @@ kubectl
 * [kubectl-log](kubectl-log.md)
 * [kubectl-rollingupdate](kubectl-rollingupdate.md)
 * [kubectl-resize](kubectl-resize.md)
+* [kubectl-exec](kubectl-exec.md)
+* [kubectl-port-forward](kubectl-port-forward.md)
 * [kubectl-run-container](kubectl-run-container.md)
 * [kubectl-stop](kubectl-stop.md)
 * [kubectl-expose](kubectl-expose.md)

diff --git a/docs/man/man1/kubectl-exec.1 b/docs/man/man1/kubectl-exec.1
@@ -0,0 +1,52 @@
+.TH "KUBERNETES" "1" " kubernetes User Manuals" "Eric Paris" "Jan 2015"  ""
+
+
+.SH NAME
+.PP
+kubectl exec \- Execute a command in a container.
+
+
+.SH SYNOPSIS
+.PP
+\fBkubectl exec\fP [OPTIONS]
+
+
+.SH DESCRIPTION
+.PP
+Execute a command in a container.
+Examples:
+  $ kubectl exec \-p 123456\-7890 \-c ruby\-container date
+  <returns output from running 'date' in ruby-container from pod 123456-7890>
+
+.PP
+$ kubectl exec \-p 123456\-7890 \-c ruby\-container \-i \-t \-\- bash \-il
+  <switches to raw terminal mode, sends stdin to 'bash' in ruby\-container from
+   pod 123456\-780 and sends stdout/stderr from 'bash' back to the client
+
+
+.SH OPTIONS
+.PP
+\fB\-c\fP, \fB\-\-container\fP=""
+    Container name
+
+.PP
+\fB\-p\fP, \fB\-\-pod\fP=""
+    Pod name
+
+.PP
+\fB\-i\fP, \fB\-\-stdin\fP=false
+    Pass stdin to the container
+
+.PP
+\fB\-t\fP, \fB\-\-tty\fP=false
+    Stdin is a TTY
+
+
+.SH SEE ALSO
+.PP
+\fBkubectl(1)\fP,
+
+
+.SH HISTORY
+.PP
+January 2015, Originally compiled by Eric Paris (eparis at redhat dot com) based on the kubernetes source material, but hopefully they have been automatically generated since!