Feature/test installed package

[ClausKlein]

Description

Use ctest --build-and-test to check the installed version

Related Issues

Motivation and Context

Every exported cmake config package must be checked for its usability.

Testing

cmake -S . -B build -DCMAKE_CXX_STANDARD=20 -DCMAKE_BUILD_TYPE=Release --fresh
ninja -c build install
ninja -c build test

Meta

see https://discourse.cmake.org/t/is-beman-exemplar-beman-exemplar-version-cmake-a-vallid-package-name/14188

• If all approvals are obtained and the PR is green, any Beman member can merge the PR.

[wusatosi]

I'm very much for testing installation.
This clashes with #122 on functionality, but that PR has not been updated in a while.
@nickelpro can you check if this approach is good?

[ClausKlein]

see #172

[nickelpro]

Yep, you beat me to it, version file name is wrong

[ClausKlein]

@camio @wusatosi What is planed with this feature?

[ClausKlein]

@camio @dietmarkuehl @ednolan Is it possible to review and merge this MR?

It was ready since weeks, I do not want to rebase it again!

[dietmarkuehl]

I'm not cmake expert at all! Mostly the changes look OK to me. I believe such a test is present on beman.execution and it takes more time than all other tests together, significantly slowing down my development: I build often and something which takes multiple seconds is annoying.

I don't see a way to disabe this test without modifying the CMakeLists.txt (I haven't tried it, though).

[ClausKlein]

I don't see a way to disabe this test without modifying the CMakeLists.txt (I haven't tried it, though).

Good point!
We may disable it if BEMAN_EXEMPLAR_INSTALL_CONFIG_FILE_PACKAGE is set to FLASE.

[ednolan]

FYI, Claus: You can test cookiecutter locally by running exemplar/cookiecutter/check_cookiecutter.sh before pushing.

[ClausKlein]

@ednolan after rebase ci fails, what dit you do?

[ednolan]

I merged a refactor of the CI system.

It made various changes, but I think the change that appears to have broken things here is that, previously, CI would build with Ninja Multi-Config, and it would always build Debug before building Release. Now, it still uses Ninja Multi-Config, but it only builds either just Debug or just Release.

I don't know why that would make a difference, but I think that's the culprit because it looks like all the CI failures here are in one of two categories:

• Release builds with GCC or Clang
• Debug builds with MSVC

I'm continuing to look into this.

[ClausKlein]

It made various changes, but I think the change that appears to have broken things here is that, previously, CI would build with Ninja Multi-Config, and it would always build Debug before building Release. Now, it still uses Ninja Multi-Config, but it only builds either just Debug or just Release.

I got it.

We need
CTEST_OUTPUT_ON_FAILURE

on CI set!

[ednolan]

I've managed to reproduce thie issue locally by launching one of the new images with the following command:

docker run -it --rm -v $PWD:$PWD -u $(id -u):$(id -g) ghcr.io/bemanproject/infra-containers-gcc:15.1.0 bash

And running the following commands:

cmake -G'Ninja Multi-Config' -B build -S . -DCMAKE_CXX_STANDARD=20 -DCMAKE_TOOLCHAIN_FILE=./infra/cmake/gnu-toolchain.cmake -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES=./infra/cmake/use-fetch-content.cmake
cmake --build build --config Release --parallel --verbose
cmake --build build --config Release --target all_verify_interface_header_sets
cmake --install build --config Release --prefix /tmp/beman.package
ctest --test-dir build --build-config Release
ctest --test-dir build --build-config Release --rerun-failed --output-on-failure

This produced the following output:

Test project /home/eddie/sync/cpp/tests/ClausKlein/exemplar/build
    Start 5: install-to-stagedir
1/2 Test #5: install-to-stagedir ..............***Failed    0.01 sec
CMake Error at /home/eddie/sync/cpp/tests/ClausKlein/exemplar/build/src/beman/exemplar/cmake_install.cmake:47 (file):
  file INSTALL cannot find
  "/home/eddie/sync/cpp/tests/ClausKlein/exemplar/build/src/beman/exemplar/Debug/libbeman.exemplar.a":
  No such file or directory.
Call Stack (most recent call first):
  /home/eddie/sync/cpp/tests/ClausKlein/exemplar/build/cmake_install.cmake:47 (include)



    Start 6: find-package-test
2/2 Test #6: find-package-test ................***Failed    0.02 sec
Internal cmake changing into directory: /home/eddie/sync/cpp/tests/ClausKlein/exemplar/build/tests/beman/exemplar/find-package-test
======== CMake output     ======
CMake Error at CMakeLists.txt:21 (find_package):
  By not providing "Findbeman.exemplar.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "beman.exemplar", but CMake did not find one.

  Could not find a package configuration file provided by "beman.exemplar"
  (requested version 2.1.1) with any of the following names:

    beman.exemplarConfig.cmake
    beman.exemplar-config.cmake

  Add the installation prefix of "beman.exemplar" to CMAKE_PREFIX_PATH or set
  "beman.exemplar_DIR" to a directory containing one of the above files.  If
  "beman.exemplar" provides a separate development package or SDK, be sure it
  has been installed.


Configuring incomplete, errors occurred!
======== End CMake output ======
Error: cmake execution failed


0% tests passed, 2 tests failed out of 2

Total Test time (real) =   0.03 sec

The following tests FAILED:
          5 - install-to-stagedir (Failed)
          6 - find-package-test (Failed)
Errors while running CTest

Note specifically in:

  file INSTALL cannot find
  "/home/eddie/sync/cpp/tests/ClausKlein/exemplar/build/src/beman/exemplar/Debug/libbeman.exemplar.a":

It's looking for a directory called Debug despite the fact that I specified Release.

I'll push an update to CI that enables CTEST_OUTPUT_ON_FAILURE.

[nickelpro]

Now that the tests don't assume an installDir being set by the preset, we should not be mandating the installDir here

[nickelpro]

Ditto, mandating targets is no longer required so we shouldn't

[nickelpro]

The example code should not be used for testing. This code serves no purpose for a user who is copy-pasting the example into their own source directory. It serves only to confuse them.

The test should be a five line project that calls cmake_minimum_required() -> project() -> find_package(REQUIRED) -> if(NOT TARGET). It should not be an example. It should exist solely for the purpose of testing the imports.

[nickelpro]

These shouldn't exist outside the build directory anyway, no need to add them here

[ClausKlein]

see #217

[ClausKlein]

I am the point to stop my support of this project. It takes to much time and to much discussions for me!