parallel doc build is unreliable

Version: aa3b4373

Sometimes normal build sequence

./autogen.sh
./configure
make -j8

explodes with:

Making all in doc
make[1]: Entering directory '/home/pspacek/w/pkg/knot/git/doc'
  SPHINX   man/knot.conf.5in
  SPHINX   man/knotc.8in
  SPHINX   man/knotd.8in
  SPHINX   man/kcatalogprint.8in
  SPHINX   man/keymgr.8in
  SPHINX   man/kjournalprint.8in
  SPHINX   man/kzonecheck.1in
  SPHINX   man/kzonesign.1in

Exception occurred:
  File "/usr/lib/python3.12/site-packages/sphinx/environment/__init__.py", line 609, in get_doctree
    doctree = pickle.loads(serialised)
              ^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: Ran out of input
The full traceback has been saved in /tmp/sphinx-err-tu8judil.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make[1]: *** [Makefile:884: man/knot.conf.5in] Error 2
make[1]: *** Waiting for unfinished jobs....

I bet this has something to do with Sphinx being executed in parallel in the same working directory and overwriting its own files.

For inspiration you can have a look at https://gitlab.isc.org/isc-projects/bind9/-/commit/33db7ea16d630b3fcc756c704b1a91ae887019f4

added bug building labels

This is observed on the latest Arch. As you can see, it's not Knot related.

Could you try just one thread? However, parallelization is disabled for manual generation

closed with commit fa690249

mentioned in commit fa690249

I can assure you it's not a problem in Arch, it's a problem in Knot's Makefiles

Judging by doc/Makefile.am the problem is that sphinx-build -d option clashes between individual sphinx-build invocations. Beware - even if man targets are serial it still explodes because html and other targets use the same directory (and are not serialized).

Cleanest fix would be to use unique -d parameter for each build, or skip it altogether (that should create it under the build directory, but better test that.)

If you insist on using single doctree directory then you need to disable parallel build for all Sphinx targets.

Is the issue solved?

No, it explodes during parallel build. I just explained why.

When executing just make -j8?

I use environment variable instead of explicit parameter, but the end result is the same:

$ echo $MAKEFLAGS 
-j8

Versions:

• GNU Make 4.4.1
• sphinx-build 7.3.7

But again, there is a clear race condition in the Makefile.am, I doubt it's version dependent.

Please note that I cannot reproduce it.

So by default, html, singhlehtml, pdf, and epub targets are created in your case, right?

I don't understand your environment. Hopefully, febc738c fixes that.

I'm sorry, I've not noticed your previous message. I confirm that I cannot reproduce the issue anymore.

Thanks!