BUG: Parallel build JSON file issues

[larsoner]

We get doc build errors like this fairly frequently nowadays:

https://app.circleci.com/pipelines/github/mne-tools/mne-python/26824/workflows/2f0db808-f2fd-4931-841c-ac7c7d98da4f/jobs/71898

Traceback

    Unexpected failing examples (1):
    
        ../examples/preprocessing/otp.py failed leaving traceback:
    
        Traceback (most recent call last):
          File "/home/circleci/project/examples/preprocessing/otp.py", line 81, in <module>
            bias = compute_bias(raw)
                   ^^^^^^^^^^^^^^^^^
...
          File "/home/circleci/project/mne/parallel.py", line 95, in parallel_func
            cache_dir = get_config("MNE_CACHE_DIR", None)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/circleci/project/mne/utils/config.py", line 309, in get_config
            config = _load_config(config_path)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/circleci/project/mne/utils/config.py", line 234, in _load_config
            warn(msg)
          File "/home/circleci/project/mne/utils/_logging.py", line 416, in warn
            warnings.warn_explicit(
        RuntimeWarning: The MNE-Python config file (/home/circleci/.mne/mne-python.json) is not a valid JSON file and might be corrupted

This started happening once we enabled parallel building in docs. I suspect it could be because the JSON file is being written by one example as it's trying to be read by another (or maybe more likely, written by one Python thread while another tries to read it). There are a few potential solutions:

1. Ignore the corrupt-JSON warning in doc builds. A bit ugly and makes it so we'll miss if something more serious goes wrong with our config I/O. But I'll probably open a PR to do this for now just so that the build stops failing so often.
2. Stop building docs in parallel. Has the severe disadvantage of doubling our build time.
3. Use some file-lock mechanism to prevent reading while writing (or writing while writing).

It seems like (3) is probably the best solution. We could use filelock (it's used for example by virtualenv and maintained by tox-dev so should be well supported; license is public domain) but it would require adding another hard dependency. We could also use msvcrt and fnctl but we'll be reinventing the wheel. Another option would be to vendor some or all of their code.

[drammock]

We could use filelock [...] but it would require adding another hard dependency.

Only in the doc optional dependency list though right?

We could also use msvcrt and fnctl but we'll be reinventing the wheel

This SO answer is only ~50 LOC, so if that's representative of what it would take to reinvent the wheel, seems maybe worth it to facilitate sticking to the standard library.

[larsoner]

Only in the doc optional dependency list though right?

No we'd need to ideally use it any time we read from or write to the config JSON, and also the forward dot product cache files.

This SO answer ...

+1 for trying that. I can take a stab at it hopefully soon and we can see if it works okay in our main doc builds