Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup: build docs in parallel #92733

Merged
merged 1 commit into from
May 16, 2022
Merged

Conversation

hugovk
Copy link
Member

@hugovk hugovk commented May 12, 2022

sphinx-build has a -j N option:

Distribute the build over N processes in parallel, to make building on multiprocessor machines more effective. Note that not all parts and not all builders of Sphinx can be parallelized. If auto argument is given, Sphinx uses the number of CPUs as N.

https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-j

Test on a new 10-core Mac

make -C Doc clean venv; time make -C Doc html

Results of two runs before

make -C Doc html  64.83s user 0.84s system 98% cpu 1:06.79 total
make -C Doc html  64.61s user 0.84s system 99% cpu 1:06.04 total

Results of two runs after

make -C Doc html  73.28s user 3.09s system 219% cpu 34.826 total
make -C Doc html  74.05s user 3.09s system 221% cpu 34.791 total

That's nearly twice as fast: 1m06s -> 35s

Test on an old dual-core Mac

Results of one run before

make -C Doc html  204.87s user 5.07s system 94% cpu 3:41.28 total

Results of one runs before

make -C Doc html  278.01s user 13.18s system 196% cpu 2:28.38 total

That's about 1.5x faster: 3m42s -> 2m29s

Copy link
Member

@AA-Turner AA-Turner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Faster is better!

A

@AA-Turner
Copy link
Member

@hugovk Should we backport to 3.9 and 3.10 too? -j auto was added in Sphinx 1.7, so I don't believe there would be any compatability concerns.

A

@hugovk hugovk added needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes labels May 14, 2022
@hugovk
Copy link
Member Author

hugovk commented May 14, 2022

Yes, labels added.

@AlexWaygood AlexWaygood requested a review from JulienPalard May 14, 2022 21:38
@JulienPalard
Copy link
Member

I'll have to dig into this a bit later, but at first glance it can't be backported to 3.9. I don't remember all the details but in 3.9 we used a non-compatibile sphinx extention (in pyspecific.py). It has been fixed around 3.10.

That's why on docs.python.org we only use parallel builds since 3.10:

https://github.com/python/docsbuild-scripts/blob/30b98030d78db56b28f8590b877f3c33f79b913d/build_docs.py#L197

We'll also have to check how it goes with the already present -j4 in build_docs.py, (Yes I'll gladly drop the flag in build_docs.py to use the one from the cpython Makefile).

@hugovk
Copy link
Member Author

hugovk commented May 16, 2022

I'll have to dig into this a bit later, but at first glance it can't be backported to 3.9. I don't remember all the details but in 3.9 we used a non-compatibile sphinx extention (in pyspecific.py). It has been fixed around 3.10.

That's why on docs.python.org we only use parallel builds since 3.10:

python/docsbuild-scripts@30b9803/build_docs.py#L197

Thanks, let's keep it simple and drop the 3.9 backport, this change is mainly beneficial for developers building docs, and I expect that will mostly be main and newer versions.

We'll also have to check how it goes with the already present -j4 in build_docs.py, (Yes I'll gladly drop the flag in build_docs.py to use the one from the cpython Makefile).

Yep, what's a good way to check it?

@hugovk hugovk removed the needs backport to 3.9 only security fixes label May 16, 2022
@JulienPalard
Copy link
Member

Yep, what's a good way to check it?

By running make SPHINXOPTS="-j4" autobuild-dev-html which is mostly what docsbuild-scripts does.

And it goes well, so it looks good to me. It starts sphinx-build -b html -d build/doctrees -j auto -j4 -Ea -A daily=1 -W . build/html.

we'll just have to remove the -j4 from docsbuild scripts once its merged, but in the meantime sphinx is not surprised by the presence of two -js.

@miss-islington
Copy link
Contributor

Thanks @hugovk for the PR, and @JulienPalard for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 16, 2022
(cherry picked from commit a487623)

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
@miss-islington
Copy link
Contributor

Sorry, @hugovk and @JulienPalard, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker a487623c6b784847a8a1e47b4597b0ae2b8def87 3.10

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label May 16, 2022
@bedevere-bot
Copy link

GH-92847 is a backport of this pull request to the 3.11 branch.

miss-islington added a commit that referenced this pull request May 16, 2022
(cherry picked from commit a487623)

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
@hugovk hugovk deleted the speedup-docs-build branch May 16, 2022 12:19
hugovk added a commit to hugovk/cpython that referenced this pull request May 16, 2022
@bedevere-bot bedevere-bot removed the needs backport to 3.10 only security fixes label May 16, 2022
@bedevere-bot
Copy link

GH-92850 is a backport of this pull request to the 3.10 branch.

@hugovk
Copy link
Member Author

hugovk commented May 16, 2022

Flag drop from build_docs.py: python/docsbuild-scripts#129

ambv pushed a commit that referenced this pull request May 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants