Add ctest execution to coverage reporting

[mtreinish]

Summary

This commit updates the coverage CI job to also include execution of the C API tests. Right now the cext crate is showing 0 coverage because we only test it without instrumention and don't collect any coverage numbers for the C API testing. This commit adds a ctest execution to the coverage job to provide coverage numbers for our C API testing.

Details and comments

[qiskit-bot]

One or more of the following people are relevant to this code:

• @Qiskit/terra-core

[coveralls]

Pull Request Test Coverage Report for Build 14204461479

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 11 unchanged lines in 3 files lost coverage.
• Overall coverage increased (+0.3%) to 88.38%

---

Files with Coverage Reduction	New Missed Lines	%
crates/qasm2/src/expr.rs	1	94.23%
crates/qasm2/src/lex.rs	4	92.48%
crates/qasm2/src/parse.rs	6	97.61%

Totals
Change from base Build 14202044040:	0.3%
Covered Lines:	73074
Relevant Lines:	82682

---

💛 - Coveralls

[jakelishman]

The Coveralls report (clicking through to it) isn't showing an increase in the cext crate coverage - is this working the way you expect yet?

[mtreinish]

The Coveralls report (clicking through to it) isn't showing an increase in the cext crate coverage - is this working the way you expect yet?

Not yet, the makefile isn't setup to handle the environment variables I'm setting to enable coverage collection so it's not building the instrumented library. I was debating between either running the tests manually without make or editing the makefile at the end of day yesterday.

[mtreinish]

The most recent commit has the coverage collection working now: https://coveralls.io/jobs/162457981/source_files/1455686271

However the tradeoff here is I've made a bit of a mess of the cmake file and make file to support this. Basically for coverage we want to be building in debug mode but all the C test build automation is around reusing the makefile. The makefile is setup to support building a proper C dynlib for end user consumption in release mode which make sense if you do make clib. The underlying issue is for coverage, and I'd argue testing more broadly we want to run in debug mode which means we can't rely on make clib. To test that this was the last missing piece I temporarily added a debug build to make clib in addition to the release mode build, but obviously this isn't ideal. So to move forward here we need to figure out the organizational structure we want in the makefile around building the c library for testing vs end user builds to better encapsulate this difference.

[mtreinish]

This should be good to go now, I split out the build targets in the makefile so we have separate paths for debug and release, and then updated the ctest target to use debug mode.

[Cryoris]

The tests are very small right now so compiling in Debug is fine, but wouldn't it generally make more sense to run in release mode for more speed? Or do you want to avoid building twice, once for coverage and once for the tests?

[mtreinish]

I don't think it's really going to be an issue for a long time. Right now the compilation of the rust code and the test binary takes slightly more than 1.5min and the tests only take 0.02 sec. So we're a long time away from where I think we need to be worried about the test execution time. I generally think having the debug symbols for tests is a good thing too, although maybe not in CI.