Plugins: experimental support for new plugin management #23446

tiborvass · 2016-06-10T22:50:10Z

Fixes #20363

NOTE: this PR modifies vendor and the vendor check is expected to fail. Once we agree on design we'll do the proper vendoring.

This patch introduces a new experimental engine-level plugin management
with a new API and command line. Plugins can be distributed via a Docker registry,
and their lifecycle is managed by the engine.
This makes plugins a first-class construct.

For more background, have a look at issue #20363.

Documentation is in a separate commit. If you want to understand how the new plugin
system works, you can start by reading the documentation.

Note: backwards compatibility with existing plugins is maintained, albeit they won't
benefit from the advantages of the new system.

subfuzion · 2016-06-10T23:26:39Z

This sounds great. This seems like the general solution to the kind of thing I was hoping for when I wrote my thoughts on a generic logging driver that would let you specify the image to use for custom logging. My particular use case was to add Kafka logging support without having to wait for an official driver. Is that the right kind of use case this PR can help with? That would be awesome.

thaJeztah · 2016-06-11T12:00:43Z

@subfuzion I don't think this adds support for new type of plugins (yet), so no logging-driver plugins yet

thaJeztah · 2016-06-11T12:01:12Z

When denying the initial install, subsequent installs fail;

$ docker plugin install aragunathan/no-remove
Plugin "aragunathan/no-remove:latest" requested the following privileges:
 - Networking: host
 - Mounting host path: /data
Do you grant the above permissions? [y/N]  (pressed ENTER here, as it's the default)
Permission denied while installing plugin aragunathan/no-remove:latest

$ docker plugin install aragunathan/no-remove
Error response from daemon: aragunathan/no-remove:latest exists

Logs are below; note that my "deny permissions" does not show up in the logs,
perhaps we should add a log-entry for that, so that it can be found back:

DEBU[0031] Calling POST /v1.24/plugins/aragunathan/no-remove:latest/pull
DEBU[0032] Trying to pull aragunathan/no-remove from https://registry-1.docker.io v2
DEBU[0033] Increasing token expiration to: 60 seconds
DEBU[0033] manifest: {
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.plugin.v0+json",
      "size": 872,
      "digest": "sha256:073da4dede0a523c610c892bdfe1bc7dc4d7d0cf38b946be7155db197b2f2274"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 5112374,
         "digest": "sha256:1c482de4cb9f561bc67ae257fc6462c77f66aa733d9d2d7e64432403af316f24"
      }
   ]
}
DEBU[0035] types.Plugin{Manifest:types.PluginManifest{ManifestVersion:"", Description:"", Documentation:"", Entrypoint:[]string(nil), Interface:types.PluginInterface{Types:[]types.InterfaceType(nil), Socket:""}, Network:types.PluginNetwork{Type:""}, Capabilities:[]string(nil), Mounts:[]struct { types.PluginSetting; types.PluginMount }(nil), Devices:[]struct { types.PluginSetting; types.PluginDevice }(nil), Env:[]struct { types.PluginSetting; types.PluginEnv }(nil), Args:[]struct { types.PluginSetting; types.PluginArg }(nil)}, Config:types.PluginConfig{Mounts:[]types.PluginMount(nil), Env:[]string(nil), Args:[]string(nil), Devices:[]types.PluginDevice(nil)}, Active:false, Name:"no-remove:latest", Tag:"", ID:""}
DEBU[0045] Calling GET /v1.24/info
DEBU[0045] Calling POST /v1.24/plugins/aragunathan/no-remove:latest/pull
DEBU[0045] plugin already exists
ERRO[0045] Handler for POST /v1.24/plugins/aragunathan/no-remove:latest/pull returned error: aragunathan/no-remove:latest exists

Doing remove, followed by install does not resolve this;

$ docker plugin rm aragunathan/no-remove:latest
aragunathan/no-remove:latest

$ docker plugin install aragunathan/no-remove
Error response from daemon: aragunathan/no-remove:latest exists

It looks like removing the plugin doesn't work in that case:

$ docker plugin rm aragunathan/no-remove:latest
aragunathan/no-remove:latest

$ docker plugin ls
NAME                    TAG                 ACTIVE
aragunathan/no-remove   latest              false

thaJeztah · 2016-06-11T22:11:19Z

Looks like CI is failing;

https://jenkins.dockerproject.org/job/Docker-PRs-experimental/19809/console

22:59:48 + go test -check.v -check.timeout=5m -test.timeout=360m github.com/docker/docker/integration-cli
23:00:17 # github.com/docker/docker/integration-cli
23:00:17 ./docker_cli_external_graphdriver_unix_test.go:80: unknown plugins.Plugin field 'Name' in struct literal

https://jenkins.dockerproject.org/job/Docker-PRs/28607/console

23:57:19 # github.com/docker/docker/plugin
23:57:19 plugin/manager.go:285: cannot convert spec (type specs.Spec) to type libcontainerd.Spec
23:57:19 plugin/manager.go:354: cannot use specs.Root literal (type specs.Root) as type windowsoci.Root in assignment
23:57:19 plugin/manager.go:379: cannot use s (type windowsoci.WindowsSpec) as type specs.Spec in return argument
23:57:19 plugin/manager.go:383: cannot use s (type windowsoci.WindowsSpec) as type specs.Spec in return argument
23:57:19 plugin/manager.go:387: cannot use m (type specs.Mount) as type windowsoci.Mount in append
23:57:19 plugin/manager.go:400: cannot use specs.Process literal (type specs.Process) as type windowsoci.Process in assignment
23:57:19 plugin/manager.go:402: cannot use s (type windowsoci.WindowsSpec) as type specs.Spec in return argument
23:58:05 Build step 'Execute shell' marked build as failure

cpuguy83 · 2016-06-12T16:42:28Z

docs/reference/commandline/plugin_inspect.md

+            "Type": "host"
+        },
+        "Capabilities": null,
+        "Mounts": [


We should reconcile this with moby/swarmkit#918 and #22373

anusha-ragunathan · 2016-06-13T16:19:23Z

cc @anusha-ragunathan

mlaventure · 2016-06-14T18:31:05Z

Super long read! But couldn't find anything that poped out, since this is going into experimental, LGTM (IANAM).

cpuguy83 · 2016-06-14T18:51:20Z

LGTM

icecrime · 2016-06-14T21:14:38Z

We need to remove those docs, as the feature is experimental.

@tiborvass Can we please gather them in a single document under the experimental/ docs tree?

Signed-off-by: Tibor Vass <tibor@docker.com>

This patch introduces a new experimental engine-level plugin management with a new API and command line. Plugins can be distributed via a Docker registry, and their lifecycle is managed by the engine. This makes plugins a first-class construct. For more background, have a look at issue moby#20363. Documentation is in a separate commit. If you want to understand how the new plugin system works, you can start by reading the documentation. Note: backwards compatibility with existing plugins is maintained, albeit they won't benefit from the advantages of the new system. Signed-off-by: Tibor Vass <tibor@docker.com> Signed-off-by: Anusha Ragunathan <anusha@docker.com>

thaJeztah · 2016-06-14T21:21:35Z

@tiborvass @icecrime you can also add an

advisory="experimental"

meta header; we added an option to the template to show a "this is an experimental feature" warning

icecrime · 2016-06-14T21:48:23Z

I think it's better to do as usual and have it in a single page under experimental/, but as you wish.

anusha-ragunathan · 2016-06-14T21:58:21Z

@thaJeztah : experimental advisory can be added to plugin docs. but the "support for advisory" PR is in docker/docs-base-1.12-integration. Do you know when it'll be merged to docker/docker?

thaJeztah · 2016-06-14T22:02:58Z

@anusha-ragunathan is see there was a PR to do that but closed, https://github.com/docker/docs-base/pull/263, perhaps the 1.12-docs-base is going to be used, let me check

thaJeztah · 2016-06-14T22:35:21Z

docs LGTM (for the RC)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

crosbymichael · 2016-06-14T23:29:14Z

cli/error.go

+type Errors []error
+
+func (errs Errors) Error() string {
+	if len(errs) < 1 {


nit: strings.Join() for this so you are not going from string to byte to string,etc

will fix in followup

HackToday · 2016-06-20T05:56:47Z

hi @thaJeztah and @tiborvass where can I found the documentation about this ?

albers · 2016-07-03T18:24:32Z

@thaJeztah I'm not sure how I should handle experimental features in bash completion.
There is no notion of an "experimental completion", so I fear that pulling support for experimental features into the completion might cause problems in release management.

If I do a default build from the sources, I will get no experimental features. Having them supported in the completion would mean an inconsistency.

My current policy is to ignore experimental features in bash completion. WDYT?

thaJeztah · 2016-07-03T19:04:51Z

@albers yes, I think we should indeed ignore the completion for now

albers · 2016-07-04T14:58:16Z

@thaJeztah yeah, I think maintaining two variants would not be adequate. Thanks for confirming.

GordonTheTurtle added the status/0-triage label Jun 10, 2016

tiborvass added status/1-design-review and removed status/0-triage labels Jun 10, 2016

subfuzion mentioned this pull request Jun 10, 2016

Logging driver plugins #18604

Closed

thaJeztah added the status/failing-ci Indicates that the PR in its current state fails the test suite label Jun 11, 2016

cpuguy83 reviewed Jun 12, 2016
View reviewed changes

icecrime added this to the 1.12.0 milestone Jun 13, 2016

tiborvass force-pushed the plugins-experimental branch 3 times, most recently from 384441c to bab6240 Compare June 14, 2016 00:04

tiborvass mentioned this pull request Jun 14, 2016

plugins: add new types and client methods docker/engine-api#272

Merged

tiborvass force-pushed the plugins-experimental branch 7 times, most recently from 1d854ed to 8e7bc6d Compare June 14, 2016 11:01

bboreham mentioned this pull request Jun 14, 2016

Better Plugin Infrastructure #20363

Closed

tiborvass force-pushed the plugins-experimental branch 2 times, most recently from 1401282 to 2cc6c47 Compare June 14, 2016 16:06

cpuguy83 added status/2-code-review and removed status/1-design-review labels Jun 14, 2016

icecrime removed the status/2-code-review label Jun 14, 2016

tiborvass force-pushed the plugins-experimental branch 2 times, most recently from e866f44 to f371170 Compare June 14, 2016 21:20

Tibor Vass added 2 commits June 14, 2016 14:20

plugins: vendor engine-api to import new plugin types

e5b7d36

Signed-off-by: Tibor Vass <tibor@docker.com>

tiborvass force-pushed the plugins-experimental branch from 8dd36f4 to e85ed69 Compare June 14, 2016 22:24

icecrime added status/4-merge and removed status/3-docs-review labels Jun 14, 2016

tiborvass force-pushed the plugins-experimental branch from e85ed69 to e79873c Compare June 14, 2016 22:49

docker plugin commandline reference

e79873c

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

crosbymichael reviewed Jun 14, 2016
View reviewed changes

cpuguy83 merged commit 6ed921d into moby:master Jun 15, 2016

nishanttotla mentioned this pull request Jun 23, 2016

Node inventory: dynamic moby/swarmkit#279

Closed

sdurrheimer mentioned this pull request Jul 3, 2016

Add zsh completion for 'docker plugin' commands #24273

Merged

kacole2 mentioned this pull request Jul 13, 2016

Run REX-Ray inside Docker container rexray/rexray#502

Closed

ohmystack mentioned this pull request Aug 2, 2016

Fix plugin lock when restoring containers #25005

Closed

tomdee mentioned this pull request Aug 12, 2016

Make Calico work with experimental docker plugins support in 1.12 projectcalico/libnetwork-plugin#61

Closed

tiborvass deleted the plugins-experimental branch July 17, 2019 00:34

thaJeztah mentioned this pull request Aug 2, 2022

api: swagger: add missing "force" query arg on plugin disable #43897

Merged

freaki2 approved these changes Sep 23, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plugins: experimental support for new plugin management #23446

Plugins: experimental support for new plugin management #23446

tiborvass commented Jun 10, 2016 •

edited

Loading

subfuzion commented Jun 10, 2016

thaJeztah commented Jun 11, 2016

thaJeztah commented Jun 11, 2016

thaJeztah commented Jun 11, 2016

cpuguy83 Jun 12, 2016

anusha-ragunathan commented Jun 13, 2016

mlaventure commented Jun 14, 2016

cpuguy83 commented Jun 14, 2016

icecrime commented Jun 14, 2016

thaJeztah commented Jun 14, 2016 •

edited by icecrime

Loading

icecrime commented Jun 14, 2016

anusha-ragunathan commented Jun 14, 2016

thaJeztah commented Jun 14, 2016

thaJeztah commented Jun 14, 2016

crosbymichael Jun 14, 2016

tiborvass Jun 14, 2016

HackToday commented Jun 20, 2016

albers commented Jul 3, 2016

thaJeztah commented Jul 3, 2016

albers commented Jul 4, 2016

Plugins: experimental support for new plugin management #23446

Plugins: experimental support for new plugin management #23446

Conversation

tiborvass commented Jun 10, 2016 • edited Loading

subfuzion commented Jun 10, 2016

thaJeztah commented Jun 11, 2016

thaJeztah commented Jun 11, 2016

thaJeztah commented Jun 11, 2016

cpuguy83 Jun 12, 2016

Choose a reason for hiding this comment

anusha-ragunathan commented Jun 13, 2016

mlaventure commented Jun 14, 2016

cpuguy83 commented Jun 14, 2016

icecrime commented Jun 14, 2016

thaJeztah commented Jun 14, 2016 • edited by icecrime Loading

icecrime commented Jun 14, 2016

anusha-ragunathan commented Jun 14, 2016

thaJeztah commented Jun 14, 2016

thaJeztah commented Jun 14, 2016

crosbymichael Jun 14, 2016

Choose a reason for hiding this comment

tiborvass Jun 14, 2016

Choose a reason for hiding this comment

HackToday commented Jun 20, 2016

albers commented Jul 3, 2016

thaJeztah commented Jul 3, 2016

albers commented Jul 4, 2016

tiborvass commented Jun 10, 2016 •

edited

Loading

thaJeztah commented Jun 14, 2016 •

edited by icecrime

Loading