Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set POOL_FLAG_ADDFILEPROVIDESFILTERED only when not loading filelists #1741

Merged
merged 1 commit into from
Oct 2, 2024
Merged

Set POOL_FLAG_ADDFILEPROVIDESFILTERED only when not loading filelists #1741

merged 1 commit into from
Oct 2, 2024

Conversation

kontura
Copy link
Contributor

@kontura kontura commented Oct 1, 2024

When filelists are loaded and file provides are used it is much faster not to set POOL_FLAG_ADDFILEPROVIDESFILTERED because libsolv goes through all the files just once during pool_addfileprovides_queue.

This is based on: rpm-software-management/libdnf#1674

Specifically this is an improvement for users that use repos with file provides outside of primary.xml and download filelsits.xml.

@kontura
Set POOL_FLAG_ADDFILEPROVIDESFILTERED only when not loading filelists
db4e548 
When filelists are loaded and file provides are used it is much faster
not to set `POOL_FLAG_ADDFILEPROVIDESFILTERED` because libsolv goes
through all the files just once during `pool_addfileprovides_queue`.
@kontura kontura force-pushed the file-provides-dynamic branch from 7bd6792 to db4e548 Compare October 1, 2024 07:48
@m-blaha m-blaha self-assigned this Oct 1, 2024
@kontura kontura marked this pull request as draft October 2, 2024 07:17
@kontura
Copy link
Contributor Author

kontura commented Oct 2, 2024 •
edited
Loading

While working on the backport to dnf4 I noticed some differences in performance, it turns out dnf5 is probably storing the filelists solv cache twice.

When metadata are downloaded with filelists the primary solv file also contains filelists. This can be visible when comparing dnf4 and dnf5 solv file sizes from the same metadata. For example:

$ ls -lah /var/cache/libdnf5/fedora-845d89688cb28f31/solv/fedora.solv /var/cache/libdnf5/fedora-845d89688cb28f31/solv/fedora-filelists.solvx
-rw-r--r--. 1 root root 61M Oct  2 03:19 /var/cache/libdnf5/fedora-845d89688cb28f31/solv/fedora-filelists.solvx
-rw-r--r--. 1 root root 87M Oct  2 03:19 /var/cache/libdnf5/fedora-845d89688cb28f31/solv/fedora.solv

vs

$ ls -lah /var/cache/dnf/fedora.solv /var/cache/dnf/fedora-filenames.solvx
-rw-r--r--. 1 root root 61M Oct  2 02:30 /var/cache/dnf/fedora-filenames.solvx
-rw-r--r--. 1 root root 27M Oct  2 02:30 /var/cache/dnf/fedora.solv

@m-blaha
Copy link
Member

m-blaha commented Oct 2, 2024

I did some tests and with this patch, the performance issue with POOL_FLAG_ADDFILEPROVIDESFILTERED is gone. Good work!

I suggest to track the primary solv file problem as a separate issue, since it's not caused by this patch, and the fix may turn out to be difficult. What do you think?

@kontura
Copy link
Contributor Author

kontura commented Oct 2, 2024

I did some tests and with this patch, the performance issue with POOL_FLAG_ADDFILEPROVIDESFILTERED is gone. Good work!

I suggest to track the primary solv file problem as a separate issue, since it's not caused by this patch, and the fix may turn out to be difficult. What do you think?

Ok, that is a good suggestion.

@kontura kontura marked this pull request as ready for review October 2, 2024 08:32
m-blaha
m-blaha approved these changes Oct 2, 2024
View reviewed changes
Copy link
Member

@m-blaha m-blaha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@m-blaha m-blaha added this pull request to the merge queue Oct 2, 2024
Merged via the queue into rpm-software-management:main with commit 68eaef5 Oct 2, 2024
17 of 20 checks passed
@kontura kontura mentioned this pull request Oct 2, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@m-blaha m-blaha m-blaha approved these changes

Assignees

@m-blaha m-blaha

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kontura @m-blaha