Test Suite

#summary Instructions on setting up and running the test suite <wiki:toc max_depth="2" />

= Test Suite =

The test suite for Dr. Memory resides in the tests/ directory. The tests are applications that we execute under control of Dr. Memory.

== Building and Running Tests ==

The tests are built by default along with the rest of Dr. Memory.

CTest is used to run the tests. After building, you can use the test make target to run them:

{{{ make -j6 make test }}}

You can also invoke ctest directly, which gives greater control over which subsets of the tests are run. Use ctest -N to see the list of tests and which number is assigned to each. Then you can use ctest -I x,y to run all tests from number x to number y, and the -V parameter to display the command line and the output of the test. For example:

{{{ ctest -V -I 49,49 }}}

You can also specify tests using inclusion and exclusion regular expressions. For example:

{{{ ctest -R 'suppress|realloc' }}}

will run all tests with the string "suppress" or the string "realloc" in their names, while

{{{ ctest -E alloc }}}

will run all tests except those with alloc in their names.

If you are using CTest version 2.8 or later, you can run tests in parallel by passing -jN to ctest on the command line:

{{{ ctest -j5 }}}

As an alternative to using -V to display the test command line and output during executing, CTest stores that information (even when not run with -V) in the Testing/Temporary/LastTest.log file in the build directory.

== Test Output ==

Each test produces two types of output: stdout and the Dr. Memory results file. Both are checked against expected output, stdout versus a .out file and the results file versus a .res file. The checking is not a regular diff and simply looks for each line. We may want to change this to a stronger diff with regular expression support in the future.

== Pre-Commit Test Suite ==

The tests/runsuite.cmake script is used to execute a series of builds and test runs. Running make test in a single build directory is not sufficient to test all of the configurations that we support.

The runsuite.cmake script is meant to be executed from an empty directory. It creates a subdirectory for each build in the suite. Use ctest -S to execute the script. Here is an example:

{{{ mkdir ../build_suite cd ../build_suite ctest -S ../drmemory/tests/runsuite.cmake }}}

We recommend using Visual Studio 2010 and MSBuild, in which case pass the use_msbuild parameter:

{{{ mkdir ../build_suite cd ../build_suite ctest -S ../drmemory/tests/runsuite.cmake,use_msbuild }}}

You can pass -V to ctest to see the results as the test suite runs. Note that it is normal to have ctest output strings such as "No tests were found!!!" as we have builds for which we run no tests (particularly in the short suite). It is also expected to have an error at the end of the suite (if run with -V):

{{{ CMake Error: Some required settings in the configuration file were missing: CTEST_SOURCE_DIRECTORY = E:/derek/opensource/withwiki/trunk/suite/.. CTEST_BINARY_DIRECTORY = (Null) CTEST_COMMAND = c:/PROGRA2/CMAKE21.6/bin/ctest.exe }}}

This warning is an artifact of how CTest assumes we've set things up and can be ignored.

At the end of the suite a file called results.txt will be created and displayed. It shows configure failures, build failures, and test failures.

Here are sample results from running the test suite:

{{{

RESULTS

drheapstat-dbg-32: all 19 tests passed drheapstat-rel-32: build successful; no tests for this build drmemory-dbg-32: all 23 tests passed drmemory-rel-32: build successful; no tests for this build final package: build successful; no tests for this build

Error in read script: /home/bruening/work/build/drmem_suite/src/tests/runsuite.cmake }}}

The "Error in read script" is expected: ignore it.

Here is an example of running the test suite on Windows from the cmd shell on a new checkout, with the -V flag for more verbose output:

{{{ cd "c:\Program Files (x86)\Microsoft Visual Studio *" VC\vcvarsall.bat

set DDKROOT=C:/derek/ddk set FLEXROOT=C:/derek/flex_sdk_4.1

cd \derek\drmemory svn checkout http://drmemory.googlecode.com/svn/trunk/ src

cd \derek\drmemory\build_suite

"c:\Program Files (x86)\CMake 2.8\bin\ctest.exe" -V -S c:/derek/drmemory/src/tests/runsuite.cmake }}}

== Embedded Versus Separate DynamoRIO ==

DynamoRIO is included inside the Dr. Memory source tree as an svn:externals property. We support both building that embedded DR along with pointing to an already-built separate DR export tree. Whenever a new feature from DynamoRIO is used within Dr. Memory, be sure to update the svn:externals following the instructions at UpdatingDR. In addition, be sure to run the pre-commit test suite using the embedded DR, or be very careful if using an external DR that it matches the svn:externals version.