Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unpack_strategy/directory: use mv for nested unpack #18518

Merged
merged 1 commit into from
Oct 20, 2024
Merged

Conversation

cho-m
Copy link
Member

@cho-m cho-m commented Oct 5, 2024

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your changes? Here's an example.
  • Have you successfully run brew style with your changes locally?
  • Have you successfully run brew typecheck with your changes locally?
  • Have you successfully run brew tests with your changes locally?

mv should preserve hardlinks and allow faster unpack on the same filesystem. A secondary pass is done with cp to copy over attributes onto any existing directories.

We only run this for nested unpacks as most direct Directory strategy usage is for repositories where moving files breaks existing code.

This uses cp -pR for non-move as some potential user reported issues could be due to Apple's cp -l on specific macOS versions. Can consider re-adding cp -l with better handling for older macOS.


This improves performance when pouring bottles if user has brew installed on same filesystem as /tmp directory (~1.2-1.4x when tested via brew reinstall which means direct brew install).


macOS mv man page says:

     As the rename(2) call does not work across file systems, mv uses cp(1) and rm(1) to accomplish the move.  The
     effect is equivalent to:

           rm -f destination_path && \
           cp -pRP source_file destination && \
           rm -rf source_file

Which should essentially be similar to cp -pR when combined with other logic.

@cho-m
Copy link
Member Author

cho-m commented Oct 6, 2024

Will need to think about this a bit more in relation to how we are "unpacking" git repositories.

Copy link
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth a try once 🟢 and you're ready, thanks!

Library/Homebrew/unpack_strategy/directory.rb Outdated Show resolved Hide resolved
@cho-m cho-m force-pushed the unpack-with-mv branch 3 times, most recently from a803cfe to 9f79991 Compare October 19, 2024 03:49
@cho-m cho-m changed the title unpack_strategy/directory: use FileUtils.mv unpack_strategy/directory: use mv for nested unpack Oct 19, 2024
@cho-m cho-m marked this pull request as ready for review October 19, 2024 13:17
@cho-m cho-m force-pushed the unpack-with-mv branch 3 times, most recently from 6a33f07 to d26dec1 Compare October 19, 2024 18:28
`mv` should preserve hardlinks and allow faster unpack on the same
filesystem. A secondary pass is done with `cp` to copy over attributes
onto any existing directories.

We only run this for nested unpacks as most direct Directory strategy
usage is for repositories where moving files breaks existing code.

This uses `cp -pR` for non-move as some potential user reported issues
could be due to Apple's `cp -l` on specific macOS versions. Can
consider re-adding `cp -l` with better handling for older macOS.
@MikeMcQuaid
Copy link
Member

Thanks @cho-m! Good to merge when you are.

@cho-m
Copy link
Member Author

cho-m commented Oct 20, 2024

Will merge now and keep an eye out for any user issues.

Should speed up pour time for large bottle, e.g. for brew install texlive --display-times (2GB bottle, 4GB unpack), I see

  • Old: 131.294 s
  • New: 48.367 s

I did try removing nested part of extraction (directly extracting to cellar), which ran in 46.952 s, but not sure on any edge cases. Not significant enough to continue exploring.

@cho-m cho-m merged commit 35ebf4a into master Oct 20, 2024
27 checks passed
@cho-m cho-m deleted the unpack-with-mv branch October 20, 2024 23:56
@MikeMcQuaid
Copy link
Member

Wow, great speedup there @cho-m!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants