parallel docker push of the same image to different registries doesn't work #22854
Closed
Description
When running docker push in parallel the behaviour seems to be to not push one layer more than once, which makes sense if we are pushing all the images to the same registry. For situations when we have multiple registries for high availability the parallel push doesn’t work and we need to do each push command separately, slowing down the CI process.
Closed issues checked before opening this one:
Parallelize Push and Pull
Can't push multiple images to same repo in parallel
docker push is not parallel
BUG REPORT INFORMATION
Output of docker version:
docker version
Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:42 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Wed Apr 27 00:34:42 2016
OS/Arch: linux/amd64
Output of docker info:
docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 13
Server Version: 1.11.1
Storage Driver: devicemapper
Pool Name: docker-202:1-1619906-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 541 MB
Data Space Total: 107.4 GB
Data Space Available: 5.021 GB
Metadata Space Used: 1.122 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.26 GiB
Name: ip-1-1-1-1
ID: 2NUJ:2XG2:XIVN:UQBX:MRSU:OA63:VGB6:SOHS:RAXF:QC3J:B4AC:5G3O
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Additional environment details (AWS, VirtualBox, physical, etc.):
Running on AWS EC2 instance launched using AMI: CentOS 7 (x86_64) with Updates HVM
Running the registries using the docker image: registry:2.3.0
Steps to reproduce the issue:
docker pull centos
docker tag centos docker-registry-us-east-1-management.domain-name.com/centos:latest
docker tag centos docker-registry-us-west-2-management.domain-name.com/centos:latest
docker tag centos docker-registry-eu-west-1-management.domain-name.com/centos:latest
# Run the push commands listed bellow in parallel:
docker push docker-registry-us-east-1-management.domain-name.com/centos:latest
docker push docker-registry-us-west-2-management.domain-name.com/centos:latest
docker push docker-registry-eu-west-1-management.domain-name.com/centos:latest
Describe the results you received:
One of the push commands succeeds but the other two fail:
docker push docker-registry-eu-west-1-management.domain-name.com/centos
The push refers to a repository [docker-registry-eu-west-1-management.domain-name.com/centos]
5f70bf18a086: Layer already exists
17a992d2fac2: Pushed
latest: digest: sha256:1a97702f27a63cfeda3f317ee22ef8b20d0f2eb01ee8a3068fe39cf2dd04b4ac size: 1125
docker push docker-registry-us-west-2-management.domain-name.com/centos
The push refers to a repository [docker-registry-us-west-2-management.domain-name.com/centos]
5f70bf18a086: Mounted from ubuntu
17a992d2fac2: Pushed
errors:
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry
Running the command separately works fine:
docker push docker-registry-us-west-2-management.domain-name.com/centos
The push refers to a repository [docker-registry-us-west-2-management.domain-name.com/centos]
5f70bf18a086: Layer already exists
17a992d2fac2: Layer already exists
latest: digest: sha256:1a97702f27a63cfeda3f317ee22ef8b20d0f2eb01ee8a3068fe39cf2dd04b4ac size: 1125
Logs from the failed push:
May 20 10:46:20 ip-1-1-1-1 docker: time="2016-05-20T10:46:20.918607499Z" level=warning msg="failed to upload schema2 manifest: errors:\ndigest invalid\nmanifest blob unknown: blob unknown to registry\ndigest invalid\nmanifest blob unknown: blob unknown to registry\ndigest invalid\nmanifest blob unknown: blob unknown to registry\ndigest invalid\nmanifest blob unknown: blob unknown to registry\ndigest invalid\nmanifest blob unknown: blob unknown to registry\ndigest invalid\nmanifest blob unknown: blob unknown to registry\ndigest invalid\nmanifest blob unknown: blob unknown to registry\ndigest invalid\nmanifest blob unknown: blob unknown to registry\n - falling back to schema1"
May 20 10:46:20 ip-1-1-1-1 docker: 2016/05/20 10:46:20 http: panic serving @: could not find ':' in digest:
May 20 10:46:20 ip-1-1-1-1 docker: goroutine 4559 [running]:
May 20 10:46:20 ip-1-1-1-1 docker: net/http.(*conn).serve.func1(0xc820b9f340, 0x7f6d4f6be658, 0xc820028278)
May 20 10:46:20 ip-1-1-1-1 docker: /usr/local/go/src/net/http/server.go:1287 +0xb5
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/distribution/digest.Digest.sepIndex(0x0, 0x0, 0x130c)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/distribution/digest/digest.go:135 +0xda
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/distribution/digest.Digest.Hex(0x0, 0x0, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/distribution/digest/digest.go:124 +0x37
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/distribution/manifest/schema1.(*configManifestBuilder).Build(0xc820fe1900, 0x7f6d4f656598, 0xc8212b6bc0, 0x0, 0x0, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/distribution/manifest/schema1/config_builder.go:137 +0x55b
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/distribution.manifestFromBuilder(0x7f6d4f655e18, 0xc8212b6bc0, 0x7f6d484548b8, 0xc820fe1900, 0xc821b2e180, 0x8, 0x8, 0x0, 0x0, 0x0, ...)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/distribution/push_v2.go:210 +0x18e
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/distribution.(*v2Pusher).pushV2Tag(0xc821436380, 0x7f6d4f655e18, 0xc8212b6bc0, 0x7f6d4f5d1b50, 0xc821268600, 0xc8219f0e60, 0x47, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/distribution/push_v2.go:170 +0x1421
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/distribution.(*v2Pusher).pushV2Repository(0xc821436380, 0x7f6d4f655e18, 0xc8212b6bc0, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/distribution/push_v2.go:85 +0x301
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/distribution.(*v2Pusher).Push(0xc821436380, 0x7f6d4f655e18, 0xc8212b6bc0, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/distribution/push_v2.go:70 +0x37e
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/distribution.Push(0x7f6d4f655e18, 0xc8212b6bc0, 0x7f6d4f5d1ba0, 0xc821268600, 0xc821902750, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/distribution/push.go:138 +0xc0e
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/daemon.(*Daemon).PushImage(0xc820596600, 0x7f6d4f5d1ba0, 0xc821268600, 0xc821293ce0, 0xc820012cc0, 0x7f6d4f655b58, 0xc8212b6b80, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:1070 +0x440
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/api/server/router/local.(*router).postImagesPush(0xc8209e00e0, 0x7f6d4f5d18d8, 0xc821293cb0, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260, 0xc821293b90, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/router/local/image.go:238 +0xa77
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/api/server/router/local.(*router).(github.com/docker/docker/api/server/router/local.postImagesPush)-fm(0x7f6d4f5d18d8, 0xc821293cb0, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260, 0xc821293b90, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/router/local/local.go:102 +0x74
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/api/server.versionMiddleware.func1(0x7f6d4f5d18d8, 0xc821293cb0, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260, 0xc821293b90, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/middleware.go:155 +0x83a
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/api/server.(*Server).corsMiddleware.func1(0x7f6d4f6be068, 0xc82006d440, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260, 0xc821293b90, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/middleware.go:134 +0xfa
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/api/server.(*Server).userAgentMiddleware.func1(0x7f6d4f6be068, 0xc82006d440, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260, 0xc821293b90, 0x0, 0x0)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/middleware.go:117 +0x4be
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/server.go:169 +0x2ad
May 20 10:46:20 ip-1-1-1-1 docker: net/http.HandlerFunc.ServeHTTP(0xc820b085e0, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260)
May 20 10:46:20 ip-1-1-1-1 docker: /usr/local/go/src/net/http/server.go:1422 +0x3a
May 20 10:46:20 ip-1-1-1-1 docker: github.com/gorilla/mux.(*Router).ServeHTTP(0xc820969360, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e
May 20 10:46:20 ip-1-1-1-1 docker: github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc820b8f000, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260)
May 20 10:46:20 ip-1-1-1-1 docker: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
May 20 10:46:20 ip-1-1-1-1 docker: net/http.serverHandler.ServeHTTP(0xc820068960, 0x7f6d4f5d17a8, 0xc820b9f3f0, 0xc821031260)
May 20 10:46:20 ip-1-1-1-1 docker: /usr/local/go/src/net/http/server.go:1862 +0x19e
May 20 10:46:20 ip-1-1-1-1 docker: net/http.(*conn).serve(0xc820b9f340)
May 20 10:46:20 ip-1-1-1-1 docker: /usr/local/go/src/net/http/server.go:1361 +0xbee
May 20 10:46:20 ip-1-1-1-1 docker: created by net/http.(*Server).Serve
May 20 10:46:20 ip-1-1-1-1 docker: /usr/local/go/src/net/http/server.go:1910 +0x3f6
Describe the results you expected:
When the client validates if a layer is already being sent it should take into account the registry url, as different URLs would require the send to still happend.
Additional information you deem important (e.g. issue happens only occasionally):
Issue happens whenever we have images with a few or all the layers the same and we try to push them in parallel to different registries.