Skip to content

Commit

Permalink
Enable libz-sys for flate2
Browse files Browse the repository at this point in the history
* libz-sys now allows to build from cc instead of cmake. This patch enables this
  for the flate2 crate, which should be the fastest implementation of gzip
* Update LAST_RELEASE to v0.18.0
* Use libz-sys instead of gzip-zune-inflate when building release binaries

Old(gzip-zune-inflate):
Benchmark 2: backhand-dist
  Time (mean ± σ):     804.6 ms ±  12.8 ms    [User: 1047.7 ms, System: 267.5 ms]
  Range (min … max):   771.8 ms … 837.3 ms    50 runs

Benchmark 3: backhand-dist-musl
  Time (mean ± σ):      1.399 s ±  0.019 s    [User: 1.903 s, System: 0.330 s]
  Range (min … max):    1.358 s …  1.442 s    50 runs

New(flate::libz-sys):
Benchmark 2: backhand-dist
  Time (mean ± σ):     672.0 ms ±   5.9 ms    [User: 846.2 ms, System: 217.1 ms]
  Range (min … max):   659.2 ms … 693.9 ms    50 runs

Benchmark 3: backhand-dist-musl
  Time (mean ± σ):     786.2 ms ±   5.6 ms    [User: 1006.3 ms, System: 203.1 ms]
  Range (min … max):   776.5 ms … 808.8 ms    50 runs
  • Loading branch information
wcampbell0x2a committed Sep 2, 2024
1 parent 47e8bf6 commit 4b2ade7
Show file tree
Hide file tree
Showing 5 changed files with 52 additions and 10 deletions.
3 changes: 1 addition & 2 deletions .github/workflows/binaries.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,7 @@ jobs:
target: ${{ matrix.job.target }}

- run: $BUILD_CMD build -p backhand-cli --bin add-backhand --bin replace-backhand --features xz-static --locked --target ${{ matrix.job.target }} --profile=dist
# default features, but replace gzip with gzip-zune-inflate
- run: $BUILD_CMD build -p backhand-cli --bin unsquashfs-backhand --locked --target ${{ matrix.job.target }} --profile=dist --no-default-features --features zstd,xz-static,gzip-zune-inflate
- run: $BUILD_CMD build -p backhand-cli --bin unsquashfs-backhand --locked --target ${{ matrix.job.target }} --profile=dist --no-default-features --features zstd,xz-static,gzip
- name: archive
run: |
tar -czvf backhand-${{ matrix.job.target }}.tar.gz \
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### `backhand`
- Use feature `zlib-ng` for `flate2`, which is enabled when compression option `Gzip` is used. This enables the backend to use [zlib-ng](https://github.com/zlib-ng/zlib-ng), which is faster by default! ([#562](https://github.com/wcampbell0x2a/backhand/pull/562))
- Remove duplicated data when addding new files to a `FilesystemWriter`. This also applies this behavior to the `add` and `replace` binaries. This is controllable with `FilesystemWriter::set_no_duplicate_files`. ([#603](https://github.com/wcampbell0x2a/backhand/pull/603)), ([#594](https://github.com/wcampbell0x2a/backhand/pull/594))
- Increase speed of internal `HashMap`s, by switching to `xxhash` and just using the `inode` as the key in other places.
- Changed `SuperBlock::Flags` to be public.
Expand Down
48 changes: 44 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion backhand/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ rustdoc-args = ["--cfg", "docsrs"]
deku = "0.17.0"
tracing = { version = "0.1.40" }
thiserror = "1.0.63"
flate2 = { version = "1.0.33", optional = true }
flate2 = { version = "1.0.33", optional = true, features = ["zlib-ng"] }
zune-inflate = { version = "0.2.54", optional = true, default-features = false, features = ["zlib"] }
xz2 = { version = "0.1.7", optional = true }
rust-lzo = { version = "0.6.2", optional = true }
Expand All @@ -28,6 +28,8 @@ zstd-safe = { version = "7.2.1", optional = true }
document-features = { version = "0.2.10", optional = true }
xxhash-rust = { version = "0.8.12", features = ["xxh64"] }
solana-nohash-hasher = "0.2.1"
# Use the fastest implementation (libz-ng) for flate2 but remove dependence on CMake
libz-sys = { version = "1.1.20", features = ["zlib-ng-no-cmake-experimental-community-maintained"], default-features = false }

[features]
default = ["xz", "gzip", "zstd"]
Expand Down
6 changes: 3 additions & 3 deletions bench.bash
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
#!/bin/bash
set -ex

LAST_RELEASE="v0.17.0"
LAST_RELEASE="v0.18.0"

BACKHAND_LAST_RELEASE="./last-release/bin/unsquashfs-backhand"
BACKHAND="./target/dist/unsquashfs-backhand"
BACKHAND_MUSL="./target/x86_64-unknown-linux-musl/dist/unsquashfs-backhand"
UNSQUASHFS="/usr/bin/unsquashfs"

# Using dynamic linked xz for perf reasons and matching unsquashfs in this testing
FLAGS="--bins --locked --profile=dist --no-default-features --features xz --features gzip-zune-inflate --features zstd"
FLAGS="--bins --locked --profile=dist --no-default-features --features xz --features zstd --features gzip"

bench () {
echo ""
Expand Down Expand Up @@ -42,7 +42,7 @@ bench "backhand-test/test-assets/test_re815_xev160/870D97.squashfs" 0x0 2_re815
# xz
bench "backhand-test/test-assets/test_tplink_ax1800/img-1571203182_vol-ubi_rootfs.ubifs" 0x0 3_ax18000
# xz
#bench "test-assets/test_archlinux_iso_rootfs/airootfs.sfs" 0x0
bench "test-assets/test_archlinux_iso_rootfs/airootfs.sfs" 0x0
# xz
bench "backhand-test/test-assets/test_er605_v2_2/2611E3.squashfs" 0x0 4_er605
# gzip
Expand Down

0 comments on commit 4b2ade7

Please sign in to comment.