Speed up caching

[connorjward]

This is a bit of a refactor of parallel_cache to try and address the performance issues reported in #3983.

Fixes #3983

TODOs

• Verify that this is parallel safe (esp. check with G-ADOPT)

The main changes are:

• API changes comm_getter to get_comm and cache_factory to make_cache.
• Do not use as_hexdigest every time we do a cache lookup. I think this was the main cause of the slowdown.
• Identify caches by a local ID attribute not type(make_cache()).__class__.__name__ which was unsafe and slow (?).

Using the example from #3983 the performance improvement takes us (with a hot cache) from 31s to 14s, so I must be doing something right.

[ksagiyam]

Otherwise, looks good to me.

[connorjward]

I have run a big G-ADOPT example on 16 cores and it ran without issue. I think that this is safe to merge.

[Ig-dolci]

Should I consider re-running my performance experiments after this PR is merged?

[connorjward]

Should I consider re-running my performance experiments after this PR is merged?

That might potentially be a good idea. Were your experiments done against release or master? (This is going into master)

[Ig-dolci]

Should I consider re-running my performance experiments after this PR is merged?

That might potentially be a good idea. Were your experiments done against release or master? (This is going into master)

Release

[connorjward]

Should I consider re-running my performance experiments after this PR is merged?

That might potentially be a good idea. Were your experiments done against release or master? (This is going into master)

Release

I guess this doesn't actually matter. You might just need to reinstall.

The changes here only really make an impact if you are running with small problems at the strong scaling limit. Does that apply?