Proposal: Network Driver

THIS PROPOSAL IS A WORK IN PROGRESS

This proposal brings new functionality and new interfaces to Docker's
networking implementation. Some things are still in flux, but our goal is to
get this approved as there is a sizeable effort internally to make this happen.
We need reviewers and comments on this proposal from the community.

You can see some of the work here
but this is a work in progress, and its implementation (and the maturity of it)
should not reflect the state of this document.

Concepts

Many concepts will be used throughout this and several proposals that are in
process. You may see these repeated with slightly different verbiage or may
contain at-length descriptions of how these interfaces will be implemented.

Here's a basic diagram about how the networking itself operates:

The following includes a description of the interface and its name. The
sub-bullets provide practical examples of these components with the existing
implementation of Docker as an example.

• Driver: executable code which is triggered when certain operations are
  requested.
• Extension: A collection of Drivers that supply different portions of
  functionality to Docker.
• State: A key/value store of parameterized data for consumption and mutation by a Driver.
  • libpack is our initial state
    implementation, but we are also evaluating
    ecc and others.
• Sandbox: An isolated environment
  • libcontainer's functionality -- for now, this is more or less a standard docker container.
• Endpoint: An addressable endpoint used for communication over a specific network. Endpoints join exactly one network and are expected to create a method of network communication for a container. Endpoints are garbage collected when they no longer belong to any Sandboxes.
  • veth pair
• Network: A collection of endpoints that are able to communicate to each other. These networks are intended to be isolated from each other and do not cross communicate. Networks house endpoints which can communicate with each other.
  • Our ethernet bridge, iptables rules we use.

Container Network Model (or CNM)

The container network model is a few axioms about how docker wishes to supply
interoperation between networks and containers.

1. All containers on a specific network can communicate with each other freely.
2. Multiple networks are the way to segment traffic between containers and
   should be supported by all drivers.
3. Multiple endpoints per container are the way to join a container to
   multiple networks.
4. An endpoint is added to a sandbox to provide it with network connectivity.

This has a few consequences:

• Network-based service discovery will replace on-disk and ENV discovery. This
  allows discovery to be scoped and more flexible for external implementations.
  Implementation of this is still TBD.
• Links will be deprecated or only exist on the default network.

Notion of a Default Network

Since Docker is a tool heavily used by both operations and development
personnel to different goals respective to their skill sets, it is critical to
have a functioning "out of the box" implementation for people to use with ease.
Docker will create a "default" network (named default) for the use of basic
networking.

Networks and Endpoints

Endpoints are a part of a Network. The network is (at least in the simplebridge implementation) isolated, but drivers may implement the notion of a network however they choose.

Docker's new system will allow for N networks, but a 'default' network will be
created as a convenience for users, and with the default driver it will
function similarly to the existing network solution now.

Multiple endpoints can be created for a single container, and bound to them at
the same time. The endpoints may live on different networks and may all belong
to one container.

Again, Endpoints as a part of different networks should not be able to communicate with each other in the default implementation. It's expected that network operators would program any bridging between two networks.

Workflow for a Network Driver

At boot, a network driver will be given a replay of its state; this will allow
the driver to return to being in sync with the state of the system, to create
new networks, etc. How replays are handled by drivers is intentionally
undefined.

A network can be requested to be created. In the workflow below, the network
assumes to be created already.

A network driver will be asked at docker run (in order, for a given network):

1. To create an endpoint within the network
2. To join an endpoint to a sandbox

The driver will also provide port mapping/expose functionality (see below for
API) and communicate with service discovery (TBD).

Network API

Driver abstract interface:

type Driver interface {
  // Restore from state
  Restore(netstate state.State) error

  // Create a new network with the network id and supplied parameters
  AddNetwork(netid string, params []string) error

  // Remove a network using the network ID.
  RemoveNetwork(netid string) error

  // Retrieve a network by ID.
  GetNetwork(id string) (Network, error)

  // List returns the IDs of available networks
  ListNetworks() []string
}

Network abstract interface:

(note that Link and Unlink here are merely for convenience and do not require
an alternative implementation)

// A network is a perimeter of IP connectivity between network services.
type Network interface {
  // Id returns the network's globally unique identifier
  Id() string

  // List of endpoints that belong to this network.
  Endpoints() []Endpoint

  // Link makes the specified sandbox reachable as a named endpoint on the network.
  // If the endpoint already exists, the call will either fail (replace=false), or
  // unlink the previous endpoint.
  //
  // For example mynet.Link(mysandbox, "db", true) will make mysandbox available as
  // "db" on mynet, and will replace the other previous endpoint, if any.
  //
  // The same sandbox can be linked to multiple networks.
  // The same sandbox can be linked to the same network as multiple endpoints.
  Link(s sandbox.Sandbox, name string, replace bool) (Endpoint, error)

  // Unlink removes the specified endpoint, unlinking the corresponding sandbox from the
  // network.
  Unlink(name string) error
}

Endpoint abstract interface:

// An endpoint represents a particular member of a network, registered under a certain name
// and reachable over IP by other endpoints on the same network.
type Endpoint interface {
  // The name of the endpoint.
  Name() string

  // Expose a port over the network. Publish it as the port to the host if
  // requested.
  Expose(portspec string, publish bool) error

  // The network this endpoint belongs to.
  Network() Network
}

docker net tool

docker net is the vehicle we're pushing to manipulate networks. The
basic commands are described below:

• create: create a network
• destroy: destroy a network
• join: join a container to a network
• leave: remove a container from a network

There will be a forthcoming UI extension to docker run which also selects a
network at run time. This interface is currently to be determined.

Our initial implementation

Our implementation is very similar to docker's existing implementation, which
we are dubbing simplebridge. This driver creates a bridge for each network,
and a veth pair for each endpoint. Networks may contain a set of vxlan peers
which will be attached to the bridge to ensure network connectivity to get
multi-host links.