Docs: Run `latexmk` in parallel when creating PDFs #123113

AA-Turner · 2024-08-17T18:07:27Z

See python/docsbuild-scripts#169 (comment) for an explanation. In short, we have accidentally been generating PDFs from .tex sources twice, with work needlessly duplicated.

This PR can be read commit-by-commit.

We change make to $(MAKE) to aid make in detecting submakes (https://www.gnu.org/software/make/manual/html_node/MAKE-Variable.html), add logging for when each part of the dist target starts and finishes, remove an obsolete sed command, run the PDF generation in parallel, reduce log verbosity, and finally ensure that we only generate the PDFs once.

A

📚 Documentation preview 📚: https://cpython-previews--123113.org.readthedocs.build/

The Makefile template switched to `latexmk` and stopped using `makeindex` on 2016-10-23, first released in Sphinx 1.6

picnixz

If you want to replace make by $(MAKE), I think you should replace all occurrences (there are some remaining in the _ensure-* targets).

Doc/Makefile

jfbu · 2024-08-17T18:50:54Z

It is a mystery to me why the second Latexmk caused such trouble. I try to reproduce this with a minimal project but can't. On second Latexmk it simply says that the targets are up-to-date. It would be very interesting to fetch the Latexmk output saying why it re-triggers xelatex.

Unfortunately Latexmk has no "log" file. One has to capture its console output.

AA-Turner · 2024-08-17T19:40:25Z

These two log files are from HEAD and this PR respectively, running time make autobuild-dev | tee autobuild-dev.pr123113.log, but with everything except the A4 LaTeX bit of the dist: target removed.

I do see the up-to-date here, so I'm not sure what was going on originally. The speed-ups are still present though, so I imagine the majority is simply from paralellism?

A

jfbu · 2024-08-17T20:28:13Z

Thanks a lot for the logs:

$ grep 'Running '\''xelatex' autobuild-dev.HEAD.log
Running 'xelatex   -recorder  "c-api.tex"'
Running 'xelatex   -recorder  "c-api.tex"'
Running 'xelatex   -recorder  "c-api.tex"'
Running 'xelatex   -recorder  "extending.tex"'
Running 'xelatex   -recorder  "extending.tex"'
Running 'xelatex   -recorder  "extending.tex"'
Running 'xelatex   -recorder  "faq.tex"'
Running 'xelatex   -recorder  "faq.tex"'
Running 'xelatex   -recorder  "faq.tex"'
Running 'xelatex   -recorder  "howto-annotations.tex"'
Running 'xelatex   -recorder  "howto-annotations.tex"'
Running 'xelatex   -recorder  "howto-annotations.tex"'
Running 'xelatex   -recorder  "howto-argparse.tex"'
Running 'xelatex   -recorder  "howto-argparse.tex"'
Running 'xelatex   -recorder  "howto-argparse.tex"'
Running 'xelatex   -recorder  "howto-clinic.tex"'
Running 'xelatex   -recorder  "howto-clinic.tex"'
Running 'xelatex   -recorder  "howto-cporting.tex"'
Running 'xelatex   -recorder  "howto-cporting.tex"'
Running 'xelatex   -recorder  "howto-curses.tex"'
Running 'xelatex   -recorder  "howto-curses.tex"'
Running 'xelatex   -recorder  "howto-curses.tex"'
Running 'xelatex   -recorder  "howto-descriptor.tex"'
Running 'xelatex   -recorder  "howto-descriptor.tex"'
Running 'xelatex   -recorder  "howto-descriptor.tex"'
Running 'xelatex   -recorder  "howto-enum.tex"'
Running 'xelatex   -recorder  "howto-enum.tex"'
Running 'xelatex   -recorder  "howto-enum.tex"'
Running 'xelatex   -recorder  "howto-free-threading-extensions.tex"'
Running 'xelatex   -recorder  "howto-free-threading-extensions.tex"'
Running 'xelatex   -recorder  "howto-free-threading-extensions.tex"'
Running 'xelatex   -recorder  "howto-functional.tex"'
Running 'xelatex   -recorder  "howto-functional.tex"'
Running 'xelatex   -recorder  "howto-functional.tex"'
Running 'xelatex   -recorder  "howto-gdb_helpers.tex"'
Running 'xelatex   -recorder  "howto-gdb_helpers.tex"'
Running 'xelatex   -recorder  "howto-gdb_helpers.tex"'
Running 'xelatex   -recorder  "howto-instrumentation.tex"'
Running 'xelatex   -recorder  "howto-instrumentation.tex"'
Running 'xelatex   -recorder  "howto-instrumentation.tex"'
Running 'xelatex   -recorder  "howto-ipaddress.tex"'
Running 'xelatex   -recorder  "howto-ipaddress.tex"'
Running 'xelatex   -recorder  "howto-ipaddress.tex"'
Running 'xelatex   -recorder  "howto-isolating-extensions.tex"'
Running 'xelatex   -recorder  "howto-isolating-extensions.tex"'
Running 'xelatex   -recorder  "howto-isolating-extensions.tex"'
Running 'xelatex   -recorder  "howto-logging-cookbook.tex"'
Running 'xelatex   -recorder  "howto-logging-cookbook.tex"'
Running 'xelatex   -recorder  "howto-logging-cookbook.tex"'
Running 'xelatex   -recorder  "howto-logging.tex"'
Running 'xelatex   -recorder  "howto-logging.tex"'
Running 'xelatex   -recorder  "howto-logging.tex"'
Running 'xelatex   -recorder  "howto-logging.tex"'
Running 'xelatex   -recorder  "howto-mro.tex"'
Running 'xelatex   -recorder  "howto-mro.tex"'
Running 'xelatex   -recorder  "howto-perf_profiling.tex"'
Running 'xelatex   -recorder  "howto-perf_profiling.tex"'
Running 'xelatex   -recorder  "howto-perf_profiling.tex"'
Running 'xelatex   -recorder  "howto-pyporting.tex"'
Running 'xelatex   -recorder  "howto-pyporting.tex"'
Running 'xelatex   -recorder  "howto-pyporting.tex"'
Running 'xelatex   -recorder  "howto-regex.tex"'
Running 'xelatex   -recorder  "howto-regex.tex"'
Running 'xelatex   -recorder  "howto-regex.tex"'
Running 'xelatex   -recorder  "howto-sockets.tex"'
Running 'xelatex   -recorder  "howto-sockets.tex"'
Running 'xelatex   -recorder  "howto-sockets.tex"'
Running 'xelatex   -recorder  "howto-sorting.tex"'
Running 'xelatex   -recorder  "howto-sorting.tex"'
Running 'xelatex   -recorder  "howto-sorting.tex"'
Running 'xelatex   -recorder  "howto-sorting.tex"'
Running 'xelatex   -recorder  "howto-timerfd.tex"'
Running 'xelatex   -recorder  "howto-timerfd.tex"'
Running 'xelatex   -recorder  "howto-unicode.tex"'
Running 'xelatex   -recorder  "howto-unicode.tex"'
Running 'xelatex   -recorder  "howto-unicode.tex"'
Running 'xelatex   -recorder  "howto-urllib2.tex"'
Running 'xelatex   -recorder  "howto-urllib2.tex"'
Running 'xelatex   -recorder  "howto-urllib2.tex"'
Running 'xelatex   -recorder  "installing.tex"'
Running 'xelatex   -recorder  "installing.tex"'
Running 'xelatex   -recorder  "installing.tex"'
Running 'xelatex   -recorder  "library.tex"'
Running 'xelatex   -recorder  "library.tex"'
Running 'xelatex   -recorder  "library.tex"'
Running 'xelatex   -recorder  "reference.tex"'
Running 'xelatex   -recorder  "reference.tex"'
Running 'xelatex   -recorder  "reference.tex"'
Running 'xelatex   -recorder  "tutorial.tex"'
Running 'xelatex   -recorder  "tutorial.tex"'
Running 'xelatex   -recorder  "tutorial.tex"'
Running 'xelatex   -recorder  "using.tex"'
Running 'xelatex   -recorder  "using.tex"'
Running 'xelatex   -recorder  "using.tex"'
Running 'xelatex   -recorder  "whatsnew.tex"'
Running 'xelatex   -recorder  "whatsnew.tex"'
Running 'xelatex   -recorder  "whatsnew.tex"'

shows that there wasn't an excessive number of xelatex runs indeed! I feel less guilty about sphinx-doc/sphinx@409605d and its FORCE_MAKE. Right now I don't recall the rationale of it (it may be that perhaps Latexmk did not detect always changes such as changing only an included image file, not sure about that), but maybe the fact that it was known that Latexmk would detect no extra run was needed played a role in its adoption.

hugovk

Nice find!

Doc/Makefile

jfbu · 2024-08-17T20:37:05Z

And with this PR (the Latexmk quiet option means we have to use another pattern):

$ grep -A3 'This is XeTeX' autobuild-dev.pr123113.log
<too many lines to copy paste here so I show fewer lines:>
$ grep -A3 'This is XeTeX' autobuild-dev.pr123113.log | grep Latexmk
Latexmk: Getting log file 'howto-cporting.log'
Latexmk: Getting log file 'howto-cporting.log'
Latexmk: Getting log file 'howto-clinic.log'
Latexmk: Getting log file 'howto-clinic.log'
Latexmk: Getting log file 'howto-free-threading-extensions.log'
Latexmk: Getting log file 'howto-free-threading-extensions.log'
Latexmk: Getting log file 'howto-free-threading-extensions.log'
Latexmk: Getting log file 'howto-argparse.log'
Latexmk: Getting log file 'howto-argparse.log'
Latexmk: Getting log file 'howto-argparse.log'
Latexmk: Getting log file 'howto-annotations.log'
Latexmk: Getting log file 'howto-annotations.log'
Latexmk: Getting log file 'howto-annotations.log'
Latexmk: Getting log file 'howto-curses.log'
Latexmk: Getting log file 'howto-curses.log'
Latexmk: Getting log file 'howto-curses.log'
Latexmk: Getting log file 'howto-descriptor.log'
Latexmk: Getting log file 'howto-descriptor.log'
Latexmk: Getting log file 'howto-descriptor.log'
Latexmk: Getting log file 'howto-enum.log'
Latexmk: Getting log file 'howto-enum.log'
Latexmk: Getting log file 'howto-enum.log'
Latexmk: Getting log file 'howto-functional.log'
Latexmk: Getting log file 'howto-functional.log'
Latexmk: Getting log file 'howto-functional.log'
Latexmk: Getting log file 'howto-gdb_helpers.log'
Latexmk: Getting log file 'howto-gdb_helpers.log'
Latexmk: Getting log file 'howto-gdb_helpers.log'
Latexmk: Getting log file 'howto-instrumentation.log'
Latexmk: Getting log file 'howto-instrumentation.log'
Latexmk: Getting log file 'howto-instrumentation.log'
Latexmk: Getting log file 'extending.log'
Latexmk: Getting log file 'extending.log'
Latexmk: Getting log file 'extending.log'
Latexmk: Getting log file 'howto-mro.log'
Latexmk: Getting log file 'howto-mro.log'
Latexmk: Getting log file 'faq.log'
Latexmk: Getting log file 'faq.log'
Latexmk: Getting log file 'faq.log'
Latexmk: Getting log file 'howto-ipaddress.log'
Latexmk: Getting log file 'howto-ipaddress.log'
Latexmk: Getting log file 'howto-ipaddress.log'
Latexmk: Getting log file 'howto-isolating-extensions.log'
Latexmk: Getting log file 'howto-isolating-extensions.log'
Latexmk: Getting log file 'howto-isolating-extensions.log'
Latexmk: Getting log file 'howto-pyporting.log'
Latexmk: Getting log file 'howto-pyporting.log'
Latexmk: Getting log file 'howto-pyporting.log'
Latexmk: Getting log file 'howto-perf_profiling.log'
Latexmk: Getting log file 'howto-perf_profiling.log'
Latexmk: Getting log file 'howto-perf_profiling.log'
Latexmk: Getting log file 'howto-timerfd.log'
Latexmk: Getting log file 'howto-timerfd.log'
Latexmk: Getting log file 'howto-logging-cookbook.log'
Latexmk: Getting log file 'howto-logging-cookbook.log'
Latexmk: Getting log file 'howto-logging-cookbook.log'
Latexmk: Getting log file 'howto-logging.log'
Latexmk: Getting log file 'howto-logging.log'
Latexmk: Getting log file 'howto-logging.log'
Latexmk: Getting log file 'howto-logging.log'
Latexmk: Getting log file 'howto-sockets.log'
Latexmk: Getting log file 'howto-sockets.log'
Latexmk: Getting log file 'howto-sockets.log'
Latexmk: Getting log file 'howto-regex.log'
Latexmk: Getting log file 'howto-regex.log'
Latexmk: Getting log file 'howto-regex.log'
Latexmk: Getting log file 'howto-unicode.log'
Latexmk: Getting log file 'howto-unicode.log'
Latexmk: Getting log file 'howto-unicode.log'
Latexmk: Getting log file 'howto-sorting.log'
Latexmk: Getting log file 'howto-sorting.log'
Latexmk: Getting log file 'howto-sorting.log'
Latexmk: Getting log file 'howto-sorting.log'
Latexmk: Getting log file 'howto-urllib2.log'
Latexmk: Getting log file 'howto-urllib2.log'
Latexmk: Getting log file 'howto-urllib2.log'
Latexmk: Getting log file 'installing.log'
Latexmk: Getting log file 'installing.log'
Latexmk: Getting log file 'installing.log'
Latexmk: Getting log file 'whatsnew.log'
Latexmk: Getting log file 'whatsnew.log'
Latexmk: Getting log file 'whatsnew.log'
Latexmk: Getting log file 'using.log'
Latexmk: Getting log file 'using.log'
Latexmk: Getting log file 'using.log'
Latexmk: Getting log file 'tutorial.log'
Latexmk: Getting log file 'tutorial.log'
Latexmk: Getting log file 'tutorial.log'
Latexmk: Getting log file 'reference.log'
Latexmk: Getting log file 'reference.log'
Latexmk: Getting log file 'reference.log'
Latexmk: Getting log file 'c-api.log'
Latexmk: Getting log file 'c-api.log'
Latexmk: Getting log file 'c-api.log'
Latexmk: Getting log file 'library.log'
Latexmk: Getting log file 'library.log'
Latexmk: Getting log file 'library.log'

AA-Turner · 2024-08-17T20:55:55Z

I'll merge this now so that we can see if this improves the LaTeX build times on the server -- we can then consider further improvements.

A

AA-Turner · 2024-08-17T20:56:10Z

Thanks all for the reviews!

miss-islington-app · 2024-08-17T20:58:10Z

Thanks @AA-Turner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

(cherry picked from commit 79c542b) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

bedevere-app · 2024-08-17T20:58:21Z

GH-123115 is a backport of this pull request to the 3.13 branch.

(cherry picked from commit 79c542b) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

bedevere-app · 2024-08-17T20:58:26Z

GH-123116 is a backport of this pull request to the 3.12 branch.

… (#123116) Docs: Run ``latexmk`` in parallel when creating PDFs (GH-123113) (cherry picked from commit 79c542b) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

… (#123115) Docs: Run ``latexmk`` in parallel when creating PDFs (GH-123113) (cherry picked from commit 79c542b) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

hugovk · 2024-08-23T18:15:47Z

No docs have built since this merged. From the logs:

2024-08-23 00:17:44,892 INFO uk/3.13: Running.
2024-08-23 00:17:44,892 DEBUG uk/3.13: Resetting dropped connection: sentry.io
2024-08-23 00:17:44,892 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/cpython show-ref -s origin/3.13'
2024-08-23 00:17:44,896 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/cpython reset --hard 6cd67e413b9a6c5c81466c4e516bca6430d7297e --'
2024-08-23 00:17:44,943 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/cpython clean -dfqx'
2024-08-23 00:17:44,966 DEBUG uk/3.13: https://sentry.io:443 "POST /api/1307574/store/ HTTP/1.1" 200 41
2024-08-23 00:17:45,001 DEBUG uk/3.13: https://sentry.io:443 "POST /api/1307574/store/ HTTP/1.1" 200 41
2024-08-23 00:17:45,028 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/uk/LC_MESSAGES fetch'
2024-08-23 00:17:45,036 DEBUG uk/3.13: https://sentry.io:443 "POST /api/1307574/envelope/ HTTP/1.1" 200 2
2024-08-23 00:17:45,283 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/uk/LC_MESSAGES branch -r'
2024-08-23 00:17:45,286 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/uk/LC_MESSAGES show-ref -s origin/3.12'
2024-08-23 00:17:45,290 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/uk/LC_MESSAGES reset --hard fdb4be30f50440b2a2fc8d200626afb7149b648c --'
2024-08-23 00:17:45,325 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/uk/LC_MESSAGES clean -dfqx'
2024-08-23 00:17:45,454 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/cpython rev-parse HEAD'
2024-08-23 00:17:45,458 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/uk/LC_MESSAGES rev-parse HEAD'
2024-08-23 00:17:45,460 DEBUG uk/3.13: Run: 'git -C /srv/docsbuild/cpython diff --name-only 50a595b37fe3201cd07701ffdda9d6baa6163562 6cd67e413b9a6c5c81466c4e516bca6430d7297e'
2024-08-23 00:17:45,465 INFO uk/3.13: Nothing changed, no rebuild needed.
2024-08-23 00:17:45,466 INFO tr/3.13: Running.
2024-08-23 00:17:45,466 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/cpython show-ref -s origin/3.13'
2024-08-23 00:17:45,469 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/cpython reset --hard 6cd67e413b9a6c5c81466c4e516bca6430d7297e --'
2024-08-23 00:17:45,493 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/cpython clean -dfqx'
2024-08-23 00:17:45,504 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/tr/LC_MESSAGES fetch'
2024-08-23 00:17:45,754 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/tr/LC_MESSAGES branch -r'
2024-08-23 00:17:45,757 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/tr/LC_MESSAGES show-ref -s origin/3.12'
2024-08-23 00:17:45,761 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/tr/LC_MESSAGES reset --hard 3a1f460265b904e8eb994522a51337afce9d5de2 --'
2024-08-23 00:17:45,792 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/tr/LC_MESSAGES clean -dfqx'
2024-08-23 00:17:45,938 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/cpython rev-parse HEAD'
2024-08-23 00:17:45,942 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/3.13/locale/tr/LC_MESSAGES rev-parse HEAD'
2024-08-23 00:17:45,945 DEBUG tr/3.13: Run: 'git -C /srv/docsbuild/cpython diff --name-only 6001deacb73e10e6f4b6fc16286cfda2babeb59e 6cd67e413b9a6c5c81466c4e516bca6430d7297e'
2024-08-23 00:17:45,951 INFO tr/3.13: Should rebuild: Doc/ has changed (from 6001deacb73e10e6f4b6fc16286cfda2babeb59e to 6cd67e413b9a6c5c81466c4e516bca6430d7297e)
2024-08-23 00:17:45,952 DEBUG tr/3.13: Run: '/srv/docsbuild/venv/bin/python -m venv /srv/docsbuild/venv-3.13'
2024-08-23 00:17:48,114 DEBUG tr/3.13: Run: '/srv/docsbuild/venv-3.13/bin/python -m pip install --upgrade python-docs-theme jieba -rrequirements.txt'
2024-08-23 00:17:49,222 DEBUG tr/3.13: Run: '/srv/docsbuild/venv-3.13/bin/python -m pip freeze --all'
2024-08-23 00:17:49,664 INFO tr/3.13: Build start.
2024-08-23 00:17:49,664 INFO tr/3.13: Running make autobuild-dev
2024-08-23 00:17:49,665 DEBUG tr/3.13: Run: "sed -i 's/ *-A switchers=1//' /srv/docsbuild/cpython/Doc/Makefile"
2024-08-23 00:17:49,670 DEBUG tr/3.13: Run: "make -C /srv/docsbuild/cpython/Doc PYTHON=/srv/docsbuild/venv-3.13/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.13/bin/blurb VENVDIR=/srv/docsbuild/venv-3.13 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=tr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-dev"
2024-08-23 00:24:42,566 ERROR tr/3.13: Run: "make -C /srv/docsbuild/cpython/Doc PYTHON=/srv/docsbuild/venv-3.13/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.13/bin/blurb VENVDIR=/srv/docsbuild/venv-3.13 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=tr -D gettext_compact=0' SPHINXERRORHANDLING= autobuild-dev" KO:
    mkdir -p build
    Building NEWS from Misc/NEWS.d with blurb
    /srv/docsbuild/cpython/Doc/build/NEWS is already up to date
    /srv/docsbuild/venv-3.13/bin/sphinx-build -b latex -d build/doctrees -j auto -D latex_elements.papersize=a4paper -D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=tr -D gettext_compact=0 -Ea -A daily=1  . build/latex 
    matplotlib is not installed, social cards will not be generated

    Build finished; the LaTeX files are in build/latex.
    Run `make all-pdf' or `make all-ps' in that directory to run these through (pdf)latex.
    make[2]: Leaving directory '/srv/docsbuild/cpython/Doc'
    # remove zip & bz2 dependency on all-pdf,
    # as otherwise the full latexmk process is run twice.
    # ($ is needed to escape the  https://www.gnu.org/software/make/manual/make.html#Basics-of-Variable-References)
    sed -i 's/: all-$(FMT)/:/' build/latex/Makefile
    (cd build/latex; make clean && make --jobs=) --output-sync LATEXMKOPTS='-quiet' all-pdf && make FMT=pdf zip bz2)
    /bin/sh: 1: Syntax error: word unexpected
    make[1]: *** [Makefile:217: dist] Error 2
    make[1]: Leaving directory '/srv/docsbuild/cpython/Doc'
    make: *** [Makefile:289: autobuild-dev] Error 2
    make: Leaving directory '/srv/docsbuild/cpython/Doc'

2024-08-23 00:24:42,570 ERROR tr/3.13: Badly handled exception, human, please help.
Traceback (most recent call last):
  File "/srv/docsbuild/scripts/build_docs.py", line 655, in run
    self.build()
  File "/srv/docsbuild/scripts/build_docs.py", line 762, in build
    run(
  File "/srv/docsbuild/scripts/build_docs.py", line 247, in run
    result.check_returncode()
  File "/usr/lib/python3.10/subprocess.py", line 457, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['make', '-C', '/srv/docsbuild/cpython/Doc', 'PYTHON=/srv/docsbuild/venv-3.13/bin/python', 'SPHINXBUILD=/srv/docsbuild/venv-3.13/bin/sphinx-build', 'BLURB=/srv/docsbuild/venv-3.13/bin/blurb', 'VENVDIR=/srv/docsbuild/venv-3.13', 'SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -q -D locale_dirs=/srv/docsbuild/3.13/locale -D language=tr -D gettext_compact=0', 'SPHINXERRORHANDLING=', 'autobuild-dev']' returned non-zero exit status 2.
2024-08-23 00:24:42,578 INFO pt-br/3.13: Running.

JulienPalard · 2024-08-24T08:30:02Z

Yes the error is:

    (cd build/latex; make clean && make --jobs=) --output-sync LATEXMKOPTS='-quiet' all-pdf && make FMT=pdf zip bz2)
    /bin/sh: 1: Syntax error: word unexpected

I agree with sh: I don't understand the line, the --output-sync right after the subshell.

hugovk · 2024-08-24T08:48:32Z

Yep, --output-sync was added to GNU Make version 4:
https://patchwork.kernel.org/project/qemu-devel/patch/20180622022035.18490-1-f4bug@amsat.org/

The docs server has 4.3:

hugovk@docs:~$ make --version
GNU Make 4.3
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
hugovk@docs:~$ make --help | grep output
  -O[TYPE], --output-sync[=TYPE]
                              Synchronize output of parallel jobs by TYPE.

Although I have 3.81 on my macOS and it's not available. If we want make dist to be run on user machines, we should remove that (conditionally on availability or completely).

But I think the docs failure here is that in this command:

(cd build/latex; $(MAKE) clean && $(MAKE) --jobs=$((nproc+1)) --output-sync LATEXMKOPTS='-quiet' all-pdf && $(MAKE) FMT=pdf zip bz2)

The --jobs=$((`nproc`+1)) bit is evaluating to nothing, and we need a number there:

(cd build/latex; make clean && make --jobs=) --output-sync LATEXMKOPTS='-quiet' all-pdf && make FMT=pdf zip bz2)

I think we can define this near the top:

NUM_CPUS := $(shell nproc)
JOBS := $(shell echo $$(($(NUM_CPUS) + 1)))

And then change the commands to use --jobs=$(JOBS)

JulienPalard · 2024-08-24T12:35:13Z

I'd say we're missing a $ in the $((`nproc`+1)), as Make uses $() as a variable substitution.

Tried:

debug:
	echo --jobs=$$((`nproc`+1))

and is gives:

$ make
echo --jobs=$((`nproc`+1))
--jobs=17

AA-Turner · 2024-08-24T12:41:58Z

I concur (and annoyingly I even left a note about this two lines earlier!). See #123287 to fix.

A

AA-Turner added 6 commits August 17, 2024 18:13

Use $(MAKE) over make

119a373

Add start/finish logging to the dist target

10e4688

Remove makeindex manipulation

9dd5490

The Makefile template switched to `latexmk` and stopped using `makeindex` on 2016-10-23, first released in Sphinx 1.6

Run make in parallel

bb3a201

Reduce latexmk verbosity

029e72b

Don't run the full latexmk process twice

701a893

AA-Turner added docs Documentation in the Doc dir skip issue skip news needs backport to 3.12 bug and security fixes needs backport to 3.13 bugs and security fixes labels Aug 17, 2024

bedevere-app bot added the awaiting core review label Aug 17, 2024

AA-Turner requested review from JulienPalard, hugovk and ezio-melotti August 17, 2024 18:07

AA-Turner mentioned this pull request Aug 17, 2024

Full build with PDF is taking more than 24h python/docsbuild-scripts#169

Closed

picnixz reviewed Aug 17, 2024

View reviewed changes

Doc/Makefile Outdated Show resolved Hide resolved

Review notes

3539828

hugovk reviewed Aug 17, 2024

View reviewed changes

Doc/Makefile Outdated Show resolved Hide resolved

Use nproc + 1

bb27b56

hugovk approved these changes Aug 17, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Aug 17, 2024

AA-Turner enabled auto-merge (squash) August 17, 2024 20:57

AA-Turner merged commit 79c542b into python:main Aug 17, 2024
25 checks passed

bedevere-app bot removed the awaiting merge label Aug 17, 2024

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 17, 2024

Docs: Run latexmk in parallel when creating PDFs (pythonGH-123113)

635cd3c

(cherry picked from commit 79c542b) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Aug 17, 2024

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 17, 2024

Docs: Run latexmk in parallel when creating PDFs (pythonGH-123113)

77854a8

(cherry picked from commit 79c542b) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

bedevere-app bot removed the needs backport to 3.12 bug and security fixes label Aug 17, 2024

AA-Turner changed the title ~~Docs: Only run latexmk once~~ Docs: latexmk in parallel when creating PDFs Aug 17, 2024

AA-Turner deleted the docs/latexmk branch August 17, 2024 21:19

AA-Turner changed the title ~~Docs: latexmk in parallel when creating PDFs~~ Docs: Run latexmk in parallel when creating PDFs Aug 17, 2024

jeremyhylton pushed a commit to jeremyhylton/cpython that referenced this pull request Aug 19, 2024

Docs: Run latexmk in parallel when creating PDFs (python#123113)

f7a7323

blhsing pushed a commit to blhsing/cpython that referenced this pull request Aug 22, 2024

Docs: Run latexmk in parallel when creating PDFs (python#123113)

db434b9

AA-Turner mentioned this pull request Aug 24, 2024

Docs: Fix Makefile syntax #123287

Merged

hugovk mentioned this pull request Aug 25, 2024

Build docs with fresh CPython commit not ~24h old one python/docsbuild-scripts#190

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs: Run `latexmk` in parallel when creating PDFs #123113

Docs: Run `latexmk` in parallel when creating PDFs #123113

AA-Turner commented Aug 17, 2024 •

edited

Loading

picnixz left a comment

jfbu commented Aug 17, 2024

AA-Turner commented Aug 17, 2024

jfbu commented Aug 17, 2024

hugovk left a comment

jfbu commented Aug 17, 2024

AA-Turner commented Aug 17, 2024

AA-Turner commented Aug 17, 2024

miss-islington-app bot commented Aug 17, 2024

bedevere-app bot commented Aug 17, 2024

bedevere-app bot commented Aug 17, 2024

hugovk commented Aug 23, 2024

JulienPalard commented Aug 24, 2024

hugovk commented Aug 24, 2024

JulienPalard commented Aug 24, 2024

AA-Turner commented Aug 24, 2024

Docs: Run latexmk in parallel when creating PDFs #123113

Docs: Run latexmk in parallel when creating PDFs #123113

Conversation

AA-Turner commented Aug 17, 2024 • edited Loading

picnixz left a comment

Choose a reason for hiding this comment

jfbu commented Aug 17, 2024

AA-Turner commented Aug 17, 2024

jfbu commented Aug 17, 2024

hugovk left a comment

Choose a reason for hiding this comment

jfbu commented Aug 17, 2024

AA-Turner commented Aug 17, 2024

AA-Turner commented Aug 17, 2024

miss-islington-app bot commented Aug 17, 2024

bedevere-app bot commented Aug 17, 2024

bedevere-app bot commented Aug 17, 2024

hugovk commented Aug 23, 2024

JulienPalard commented Aug 24, 2024

hugovk commented Aug 24, 2024

JulienPalard commented Aug 24, 2024

AA-Turner commented Aug 24, 2024

Docs: Run `latexmk` in parallel when creating PDFs #123113

Docs: Run `latexmk` in parallel when creating PDFs #123113

AA-Turner commented Aug 17, 2024 •

edited

Loading