Document the benchmarks, make them build and compare versions again, and pin Google Benchmark

[nlohmann]

What & why

tests/benchmarks/ had no documentation, no way to compare two versions, and, as it turned out, didn't build:

• The benchmark project hasn't configured since Detect used C++ standard library #4793 (May 2025). cmake/download_test_data.cmake compiles detect_libcpp_version.cpp from ${CMAKE_SOURCE_DIR}/cmake/. When tests/benchmarks is the top-level project, that path doesn't exist, so try_run fails and make run_benchmarks stops at configure time.
• Dump discarded dump()'s result, which is [[nodiscard]] by now. That warns, and it leaves the optimizer free to shorten the measured loop.
• Comparing versions meant building each version's own benchmarks from scratch. That compares different benchmark code, too.
• ci_benchmarks in cmake/ci.cmake has configured ${PROJECT_SOURCE_DIR}/benchmarks since CI targets #2561, but the benchmarks live in tests/benchmarks.
• Google Benchmark was fetched from its main branch, so builds on different days could measure with different library code. CMake 3.30 and later also warn that the single-argument FetchContent_Populate() is deprecated.

The change

• download_test_data.cmake takes the source from ${CMAKE_CURRENT_LIST_DIR}, the directory of the module itself. For the main build that is the same file as before; its configure still detects the standard library.

• Dump keeps the result with benchmark::DoNotOptimize.

• ci_benchmarks points at tests/benchmarks.

• Google Benchmark is pinned to release 1.9.5. CMake fetches the release archive, checks its SHA-256, and uses FetchContent_MakeAvailable(), which needs CMake 3.14 (Google Benchmark itself already needed 3.13). Its -Werror is off, so a newer compiler's new warnings can't break the pinned release, and its install rules are no longer added.

• A cache variable, JSON_BENCHMARK_INCLUDE_DIR, names the directory holding the nlohmann/json.hpp to benchmark. It defaults to single_include, as before, so the same benchmarks can be built against two headers.

• tests/benchmarks/README.md covers:

  • what is measured and what's needed to build it;
  • how to run everything or a selection, and the useful options;
  • how to read the output;
  • how to compare two versions with Google Benchmark's compare.py, and how to get stable numbers.

  It suggests running the comparison by hand before a release and for performance PRs, but not in CI, where shared runners are too noisy.

Verification

• The README's commands run as written, apart from a --benchmark_filter to keep the run short. Both builds configure (one downloading the test data, one reusing it), build without warnings, and run; compare.py compares the two result files.
• The current benchmarks build against 3.12.0's header. For example, on this machine develop parses twitter.json 47% faster than 3.12.0 (p = 0.0002, 10 repetitions each).
• With the pinned release, a fresh configure with CMake 4.3.4 prints no warnings (the old FetchContent_Populate deprecation warning is gone). The benchmarks build with Apple clang and GCC 16, and the compare recipe runs end to end, including v1.9.5's pinned tools/requirements.txt on Python 3.11.
• The main project still configures and detects libc++. With JSON_CI=ON, the ci_benchmarks target's command now names tests/benchmarks.

Public API impact

No breaking changes. This touches only the benchmark project, its docs, and the CMake module's path to a helper file. Building the benchmarks now needs CMake 3.14 instead of the nominal 3.11.

---

• The changes are described in detail, both the what and why.
• If applicable, an existing issue is referenced. (none)
• The Code coverage remained at 100%. A test case for every new line of code. (no library code changed)
• If applicable, the documentation is updated.
• The source code is amalgamated by running make amalgamate. (not applicable)

🤖 Generated with Claude Code