Speedup sphinx-build using parallelization

[weiji14]

Description of proposed changes

Distribute sphinx-build over several processes in parallel to make building on multiprocessor machines faster. Using 'auto' which equals the number of CPUs available. This should speed up running make html, make linkcheck and make doctest in the 'doc/' folder and cut down on time waiting for the Vercel Continuous Documentation and Github Actions Continuous Integration steps to run.

References:

• https://www.sphinx-doc.org/en/3.x/man/sphinx-build.html#cmdoption-sphinx-build-j.

Addresses #584.

Reminders

• Run make format and make check to make sure the code follows the style guide.
• Add tests for new features or tests that would have caught the bug that you're fixing.
• Add new public functions/methods/classes to doc/api/index.rst.
• Write detailed docstrings for all functions/methods.
• If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

• /format: automatically format and lint the code
• /test-gmt-dev: run full tests on the latest GMT development version

[weiji14]

/test-gmt-dev

[seisman]

Perhaps you have to change a python file to trigger the workflows.

[weiji14]

The parallelization doesn't seem to speed things up much on Github Actions CI. Scheduled test on master was 3m 47s while doc build on this PR was 3m 53s (a bit slower, but about the same). Maybe the Github Actions CI doesn't have that many CPU cores allocated? Vercel doc build time is still 7-8min.

Testing locally (8 CPU cores) does show a bit of speedup but not very significant, maybe a few tens of seconds.

[seisman]

The parallelization doesn't seem to speed things up much on Github Actions CI. Scheduled test on master was 3m 47s while doc build on this PR was 3m 53s (a bit slower, but about the same). Maybe the Github Actions CI doesn't have that many CPU cores allocated? Vercel doc build time is still 7-8min.

Testing locally (8 CPU cores) does show a bit of speedup but not very significant, maybe a few tens of seconds.

Most of the time is spent on running Python scripts. Maybe -j auto doesn't work for sphinx-gallery?

[weiji14]

The parallelization doesn't seem to speed things up much on Github Actions CI. Scheduled test on master was 3m 47s while doc build on this PR was 3m 53s (a bit slower, but about the same). Maybe the Github Actions CI doesn't have that many CPU cores allocated? Vercel doc build time is still 7-8min.
Testing locally (8 CPU cores) does show a bit of speedup but not very significant, maybe a few tens of seconds.

Most of the time is spent on running Python scripts. Maybe -j auto doesn't work for sphinx-gallery?

Possibly related to sphinx-doc/sphinx#4575.

P.S. Might need to run full tests to check, as I saw some issues with running parallel sphinx-build on Windows.

[weiji14]

Hmm, 5 test failures on Windows GMT dev tests again at https://github.com/GenericMappingTools/pygmt/runs/2026668071?check_suite_focus=true#step:15:1245. I think they're unrelated but it does looks scary!

Details

```python-traceback ________________________ test_gmt_compat_6_is_applied _________________________

capsys = <_pytest.capture.CaptureFixture object at 0x0000020835D65E80>

def test_gmt_compat_6_is_applied(capsys):
    """

Error: Ensure that users with old gmt.conf files won't get pygmt-session [ERROR]:

    GMT_COMPATIBILITY: Expects values from 6 to 6; reset to 6.
    """
    end()  # Kill the global session
    try:
        # Generate a gmt.conf file in the currenty directory
        # with GMT_COMPATIBILITY = 5
        with Session() as lib:
            lib.call_module("gmtset", "GMT_COMPATIBILITY 5")
        begin()
        with Session() as lib:
            lib.call_module("basemap", "-R10/70/-3/8 -JX4i/3i -Ba")
            out, err = capsys.readouterr()  # capture stdout and stderr
            assert out == ""
            assert err != (

Error: "pygmt-session [ERROR]: GMT_COMPATIBILITY:"
" Expects values from 6 to 6; reset to 6.\n"
)

          assert err == ""  # double check that there are no other errors

Error: AssertionError: assert 'end [ERROR]:...rmissions?]\n' == ''
Error: + end [ERROR]: Failed to remove gmt_201.ps-! [remove error: Permission denied]
Warning: + end [WARNING]: Unable to remove gmt_201.ps- [permissions?]
Warning: + end [WARNING]: Unable to remove directory C:/Users/runneradmin/.gmt/sessions/gmt_session.4504 [permissions?]
Warning: + gmtset [WARNING]: GMT_COMPATIBILITY: Expects values from 6 to 6; reset to 6.
Warning: + pygmt-session [WARNING]: GMT_COMPATIBILITY: Expects values from 6 to 6; reset to 6.
Error: + begin [ERROR]: Failed to remove gmt_201.ps-! [remove error: Permission denied]
Warning: + begin [WARNING]: Unable to remove gmt_201.ps- [permissions?]

C:\Miniconda3\envs\test\lib\site-packages\pygmt\tests\test_session_management.py:47: AssertionError

During handling of the above exception, another exception occurred:

capsys = <_pytest.capture.CaptureFixture object at 0x0000020835D65E80>

def test_gmt_compat_6_is_applied(capsys):
    """

Error: Ensure that users with old gmt.conf files won't get pygmt-session [ERROR]:

    GMT_COMPATIBILITY: Expects values from 6 to 6; reset to 6.
    """
    end()  # Kill the global session
    try:
        # Generate a gmt.conf file in the currenty directory
        # with GMT_COMPATIBILITY = 5
        with Session() as lib:
            lib.call_module("gmtset", "GMT_COMPATIBILITY 5")
        begin()
        with Session() as lib:
            lib.call_module("basemap", "-R10/70/-3/8 -JX4i/3i -Ba")
            out, err = capsys.readouterr()  # capture stdout and stderr
            assert out == ""
            assert err != (

Error: "pygmt-session [ERROR]: GMT_COMPATIBILITY:"
" Expects values from 6 to 6; reset to 6.\n"
)
assert err == "" # double check that there are no other errors
finally:
end()
# Clean up the global "gmt.conf" in the current directory

      assert os.path.exists("gmt.conf")

E AssertionError: assert False
E + where False = <function exists at 0x0000020827DB35E0>('gmt.conf')
E + where <function exists at 0x0000020827DB35E0> = <module 'ntpath' from 'C:\Miniconda3\envs\test\lib\ntpath.py'>.exists
E + where <module 'ntpath' from 'C:\Miniconda3\envs\test\lib\ntpath.py'> = os.path

C:\Miniconda3\envs\test\lib\site-packages\pygmt\tests\test_session_management.py:51: AssertionError

</details>

[seisman]

It seems adding -j auto doesn't speed up at all.

[seisman]

As -j auto doesn't work as we expected, I think we can just rename the issue title and only remove the linkcheck and doctest targets in this PR.

[weiji14]

As -j auto doesn't work as we expected, I think we can just rename the issue title and only remove the linkcheck and doctest targets in this PR.

There's no speedup on the Github Actions CI, but I think it helps a bit locally, shaving off a few tens of seconds on my 8-core machine when I tested, so perhaps ok to keep -j auto since it's harmless? Do you notice anything on your local machine?

[seisman]

As -j auto doesn't work as we expected, I think we can just rename the issue title and only remove the linkcheck and doctest targets in this PR.

There's no speedup on the Github Actions CI, but I think it helps a bit locally, shaving off a few tens of seconds on my 8-core machine when I tested, so perhaps ok to keep -j auto since it's harmless? Do you notice anything on your local machine?

Building the documentation takes 130 s in the master branch, and 120 s in this branch. It's even faster if you move -j auto to the first line, i.e.,

SPHINXOPTS    = -j auto

[seisman]

It's even faster if you move -j auto to the first line, i.e.,

SPHINXOPTS    = -j auto

It's not true, but still adding -j auto to SPHINXOPTS is better.