Make image (layer) downloads faster by using pigz #35697

sargun · 2017-12-04T23:55:07Z

- What I did
The Golang built-in gzip library is serialized, and fairly slow
at decompressing. It also only decompresses on demand, versus
pipelining decompression. This fixes those problems.

- How I did it
This change switches to trying to use parallel gzip for gzip decompression as opposed to the
golang one. If it isn't available, it'll fall back to the default. This code path can also be disabled by environment variable.

- How to verify it
There are existing tests for this codepath to ensure correctness of this implementation. I ran some manual benchmarks, and I found I was able to get about 50% better performance as opposed to the build in library. These performance gains were primarily seen on images with layers about 10MB.

- Description for the changelog
Make image (layer) downloads faster by using pgzip

sargun · 2017-12-05T00:02:56Z

Any idea what:

00:02:07 2017/12/05 00:02:07 unrecognized import path "cloud.google.com/go" (parse https://cloud.google.com/go?go-get=1: no go-import meta tags)

Is due to?

thaJeztah · 2017-12-05T04:59:44Z

looks like this is implementing the same as #34788 (looks like that one stalled though)

sargun · 2017-12-05T05:02:50Z

Should I hate this behind an environment variable, so you can opt out of the library? The primary downside I know is that there is a memory overhead with this library.

…

Sent from my iPhone On Dec 4, 2017, at 21:00, Sebastiaan van Stijn <notifications@github.com> wrote: looks like this is implementing the same as #34788 <#34788> (looks like that one stalled though) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#35697 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAtyRNDunSsbTahMlr9qTJ9MRJN9MYsGks5s9M4LgaJpZM4Q1g4j> .

thaJeztah · 2017-12-05T05:17:54Z

We preferred to use the external binary instead of using the Go implementation (see the other pull request); unless there's compelling reasons that you know of.

sargun · 2017-12-05T11:09:36Z

@thaJeztah Switched out. It looks like LZMA / xz was already doing this, so I took this opportunity to share code between the two of them.

sargun · 2017-12-05T16:27:01Z

Once this is in, I'll add pigz as a recommendation to the packages here: https://github.com/docker/docker-ce-packaging.

sargun · 2017-12-05T17:59:20Z

@thaJeztah I don't see an easy way to install pigz in the container for CI. Any ideas?

sargun · 2017-12-06T01:25:16Z

@thaJeztah PTAL.

sargun · 2017-12-06T21:19:09Z

CC:
@ripcurld0
@jboero

thaJeztah · 2017-12-08T00:59:52Z

pkg/archive/archive.go

+
+	unpigzPath, err := exec.LookPath("unpigz")
+	if err != nil {
+		logrus.Debug("Cannot find unpigz")


Possibly change the error to be something like "Unpigz binary not found in PATH, falling back to regular gzip" (better suggestions welcome 😅 )

thaJeztah · 2017-12-08T01:01:37Z

Can you add the package to the Dockerfiles, so that it can be run in CI? (Or is it already there)?

thaJeztah · 2017-12-08T01:02:47Z

ping @AkihiroSuda @unclejack @stevvooe PTAL

AkihiroSuda · 2017-12-08T01:53:29Z

I don't think we should add an environment variable.

I suggest introducing a daemon flag --parallel-decompression=pigz|none and pass it to pkg/archive via some WithPigz DecompressOpt like

func DecompressStream(archive io.Reader, opts DecompressOpt...) (io.ReadCloser, error)

sargun · 2017-12-08T01:57:03Z

@AkihiroSuda Exposing that knob seems like overkill, and introduces a deprecation cycle if we ever want to revert to using the Golang parallel gzip library.

sargun · 2017-12-08T02:11:37Z

@AkihiroSuda If we do anything, I'd suggest (part of a different PR), --decompress-gz-with=gzip|unpigz|internal

AkihiroSuda · 2017-12-08T02:15:41Z

--decompress-gz-with=gzip|unpigz|internal

What is gzip here? Do you mean executing /bin/gzip?

sargun · 2017-12-08T02:16:29Z

@AkihiroSuda correct.

AkihiroSuda · 2017-12-08T02:17:33Z

How /bin/gzip is useful?

This change is in response to moby/moby#35697 It adds pigz to the recommended binaries that should be installed with docker-ce. Signed-off-by: Sargun Dhillon <sargun@sargun.me> Upstream-commit: 1ca014b Component: packaging

jboero · 2018-04-23T13:13:40Z

Wow I thought nobody liked my pigz proposal but then I just noticed a RHEL 7 install failed with pigz as required prerequisite:

ec2-user@ip- ~> sudo yum install pigz
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
No package pigz available.
Error: Nothing to do

I love that pigz is now used but my PR had no packaging dependency and would fall back to regular gzip if pigz was unavailable. D'oh!

Anybody in packaging please remove pigz from deps... RHEL 7 doesn't have a pigz in repos.

Thanks guys!

cpuguy83 · 2018-04-23T13:19:59Z

@jboero Agree having it as a requirement is not right, however there is no version of Docker for RHEL that has pigz support yet.

sargun · 2018-04-23T14:48:27Z

At least for the Debian packaging, it just falls back. In Debian, they have the idea of “recommends” — does RHEL not?

…

Sent from my iPhone On Apr 23, 2018, at 06:21, Brian Goff <notifications@github.com> wrote: @jboero <https://github.com/jboero> Agree having it as a requirement is not right, however there is no version of Docker for RHEL that has pigz support yet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35697 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAtyRG7MtMu4keQ8oc4AyGeqUl_262oFks5trdVGgaJpZM4Q1g4j> .

cpuguy83 · 2018-04-23T14:52:49Z

No, there is no support for "weak depdendencies" until rather recently on RPM, and not supported on RHEL. Docker CE is for CentOS, which does have a pigz package. On Mon, Apr 23, 2018 at 10:48 AM Sargun Dhillon <notifications@github.com> wrote:

…

At least for the Debian packaging, it just falls back. In Debian, they have the idea of “recommends” — does RHEL not? Sent from my iPhone On Apr 23, 2018, at 06:21, Brian Goff ***@***.***> wrote: @jboero <https://github.com/jboero> Agree having it as a requirement is not right, however there is no version of Docker for RHEL that has pigz support yet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35697 (comment)>, or mute the thread < https://github.com/notifications/unsubscribe-auth/AAtyRG7MtMu4keQ8oc4AyGeqUl_262oFks5trdVGgaJpZM4Q1g4j > . — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#35697 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwxZssdmd4oX3vzPvgNl1efMnjKiiUMks5trenPgaJpZM4Q1g4j> .

jboero · 2018-04-23T15:34:13Z

Ah of course. OK thanks guys. I'm glad they pushed for using external pigz instead of trying to rewrite everything from scratch in go. Well done.

jboero · 2018-04-23T16:19:59Z

Workaround - pigz is included in EPEL which works with RHEL (and CentOS)

lox · 2019-04-05T00:43:49Z

It's super not clear how to actually make use of this, do we need to ensure that unpigz is available on hosts to see the speed up?

sargun · 2019-04-05T00:48:25Z

@lox correct

This change is in response to moby/moby#35697 It adds pigz to the recommended binaries that should be installed with docker-ce. Signed-off-by: Sargun Dhillon <sargun@sargun.me> Upstream-commit: 1ca014b Component: packaging

GitHubDiom · 2020-11-10T11:29:03Z

Hey, how do you guys make package decompression parallel?
I'd like to test parallel decompression but can't seem to do it.

dockerinfo

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.0.2-1.el7.elrepo.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.757GiB
 Name: master
 ID: KGRO:7J7A:7TKD:UNUO:ZQ7A:QYEJ:DJK4:OK4S:7JBS:U5IG:LBS3:2RJ7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

after run docker run gcc

download is parallel
But the extraction is serial.

thaJeztah · 2020-11-10T12:07:14Z

@GitHubDiom this PR makes each individual "extract" run faster if pigz is installed, but layers still have to be extracted/applied in sequence; see #21814, #37957 for more details

GitHubDiom · 2020-11-10T13:53:27Z

@GitHubDiom this PR makes each individual "extract" run faster if pigz is installed, but layers still have to be extracted/applied in sequence; see #21814, #37957 for more details

Can docker pipeline the pull process for a specific layer?
As far as I know, three phases appear when pulling one layer, for example, download(Network-intensive) -> decompress (CPU-intensive) -> extract (IO-intensive),
where decompress phase means to decompress the image from Net-adaptor buffer to memory, and extract phase means to move the image from memory to disk( it is exactly? )

For instance, extracting the layer may start immediately after the beginning of the layer image has been decompressed.

thaJeztah · 2020-11-10T15:10:15Z

Theres some comments about decompress/extract in #21814 (comment) (and further)

Can increase Docker image extraction speed significantly: moby/moby#35697

See moby/moby#35697

and introduce pigz for layer extract speed up: moby/moby#35697 Signed-off-by: Jingkai He <jingkai@hey.com>

* bash script fixing Signed-off-by: Jingkai He <jingkai@hey.com> * use the latest focal image as the base image Signed-off-by: Jingkai He <jingkai@hey.com> * removed unnecessary dependencies and introduce pigz for layer extract speed up: moby/moby#35697 Signed-off-by: Jingkai He <jingkai@hey.com> * bring net-tools & iproute2 back Signed-off-by: Jingkai He <jingkai@hey.com> --------- Signed-off-by: Jingkai He <jingkai@hey.com>

GordonTheTurtle added the status/0-triage label Dec 4, 2017

sargun force-pushed the use-pgzip branch 4 times, most recently from 7c2c30f to c84bb1a Compare December 5, 2017 09:21

sargun changed the title ~~Make image (layer) downloads faster by using pgzip~~ Make image (layer) downloads faster by using pigz Dec 5, 2017

thaJeztah reviewed Dec 8, 2017

View reviewed changes

thaJeztah added status/1-design-review and removed status/0-triage labels Dec 8, 2017

sargun force-pushed the use-pgzip branch from c84bb1a to c41b6e6 Compare December 8, 2017 01:14

AkihiroSuda mentioned this pull request Dec 8, 2017

Added parallel gzip. #34788

Closed

AkihiroSuda mentioned this pull request Dec 8, 2017

archive: support parallel decompression (pigz) containerd/containerd#1896

Closed

sargun force-pushed the use-pgzip branch from c41b6e6 to fca9825 Compare December 8, 2017 02:07

AkihiroSuda mentioned this pull request Apr 6, 2018

[FEATURE REQUEST] Enable parallel gzip (pigz) option for image compression/decompression. #34787

Closed

thaJeztah mentioned this pull request May 22, 2018

Remove pigz requirement for CentOS docker/docker-ce-packaging#110

Merged

mxpv mentioned this pull request Sep 13, 2018

Support parallel decompression (pigz) containerd/containerd#2640

Merged

thaJeztah mentioned this pull request Oct 3, 2018

docker pull does download and extraction sequentially #37957

Closed

rajivml mentioned this pull request Mar 25, 2019

Docker Inside Docker (DinD) GoogleContainerTools/distroless#334

Closed

snorrk mentioned this pull request Apr 8, 2020

Enable multiprocessor decompression of layers docker/for-win#6271

Closed

fnkr added a commit to punktDe/ansible-docker that referenced this pull request Sep 21, 2021

TASK: Install pigz

1fb78d2

Can increase Docker image extraction speed significantly: moby/moby#35697

This was referenced Aug 23, 2022

Push image faster by using pigz to compress #44007

Closed

Push image faster by using pigz to compress #44008

Open

dcarrion87 mentioned this pull request Jan 26, 2023

HAI-1010 Add pigz to improve compression performance harrison-ai/cobalt-docker-rootless-nvidia-dind#6

Merged

Okeanos added a commit to Okeanos/dotfiles that referenced this pull request May 12, 2023

add pigz package to make Docker faster

bbf138b

See moby/moby#35697

peterzhuamazon mentioned this pull request Aug 23, 2023

[Bug]: Windows CI builds failing to find docker (Update to Run Windows inside docker containers) opensearch-project/opensearch-ci#281

Closed

jingkaihe added a commit to jingkaihe/concourse-dind that referenced this pull request Oct 6, 2023

removed unnecessary dependencies

cf6cdbd

and introduce pigz for layer extract speed up: moby/moby#35697 Signed-off-by: Jingkai He <jingkai@hey.com>

travier mentioned this pull request Sep 24, 2024

tracker: package additions/removals for F41 coreos/fedora-coreos-tracker#1785

Closed

thaJeztah mentioned this pull request Jan 8, 2025

pkg/ioutils: un-export ReadCloserWrapper #49237

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make image (layer) downloads faster by using pigz #35697

Make image (layer) downloads faster by using pigz #35697

sargun commented Dec 4, 2017 •

edited

Loading

sargun commented Dec 5, 2017

thaJeztah commented Dec 5, 2017

sargun commented Dec 5, 2017 via email

thaJeztah commented Dec 5, 2017

sargun commented Dec 5, 2017

sargun commented Dec 5, 2017

sargun commented Dec 5, 2017

sargun commented Dec 6, 2017

sargun commented Dec 6, 2017

thaJeztah Dec 8, 2017

thaJeztah commented Dec 8, 2017

thaJeztah commented Dec 8, 2017

AkihiroSuda commented Dec 8, 2017

sargun commented Dec 8, 2017

sargun commented Dec 8, 2017 •

edited

Loading

AkihiroSuda commented Dec 8, 2017

sargun commented Dec 8, 2017

AkihiroSuda commented Dec 8, 2017

jboero commented Apr 23, 2018 •

edited

Loading

cpuguy83 commented Apr 23, 2018

sargun commented Apr 23, 2018 via email

cpuguy83 commented Apr 23, 2018 via email

jboero commented Apr 23, 2018

jboero commented Apr 23, 2018

lox commented Apr 5, 2019

sargun commented Apr 5, 2019

GitHubDiom commented Nov 10, 2020 •

edited

Loading

thaJeztah commented Nov 10, 2020

GitHubDiom commented Nov 10, 2020 •

edited

Loading

thaJeztah commented Nov 10, 2020

Make image (layer) downloads faster by using pigz #35697

Make image (layer) downloads faster by using pigz #35697

Conversation

sargun commented Dec 4, 2017 • edited Loading

sargun commented Dec 5, 2017

thaJeztah commented Dec 5, 2017

sargun commented Dec 5, 2017 via email

thaJeztah commented Dec 5, 2017

sargun commented Dec 5, 2017

sargun commented Dec 5, 2017

sargun commented Dec 5, 2017

sargun commented Dec 6, 2017

sargun commented Dec 6, 2017

thaJeztah Dec 8, 2017

Choose a reason for hiding this comment

thaJeztah commented Dec 8, 2017

thaJeztah commented Dec 8, 2017

AkihiroSuda commented Dec 8, 2017

sargun commented Dec 8, 2017

sargun commented Dec 8, 2017 • edited Loading

AkihiroSuda commented Dec 8, 2017

sargun commented Dec 8, 2017

AkihiroSuda commented Dec 8, 2017

jboero commented Apr 23, 2018 • edited Loading

cpuguy83 commented Apr 23, 2018

sargun commented Apr 23, 2018 via email

cpuguy83 commented Apr 23, 2018 via email

jboero commented Apr 23, 2018

jboero commented Apr 23, 2018

lox commented Apr 5, 2019

sargun commented Apr 5, 2019

GitHubDiom commented Nov 10, 2020 • edited Loading

thaJeztah commented Nov 10, 2020

GitHubDiom commented Nov 10, 2020 • edited Loading

thaJeztah commented Nov 10, 2020

sargun commented Dec 4, 2017 •

edited

Loading

sargun commented Dec 8, 2017 •

edited

Loading

jboero commented Apr 23, 2018 •

edited

Loading

GitHubDiom commented Nov 10, 2020 •

edited

Loading

GitHubDiom commented Nov 10, 2020 •

edited

Loading