Test Suite
The test suite for Dr. Memory resides in the tests/ directory. The tests are applications that we execute under control of Dr. Memory.
The tests are built by default along with the rest of Dr. Memory.
CTest is used to run the tests. After building, you can use the test
make target to run them:
make -j6
make test
You can also invoke ctest directly, which gives greater control over
which subsets of the tests are run. Use ctest -N to see the list of
tests and which number is assigned to each. Then you can use ctest -I x,y to run all tests from number x to number y, and the -V parameter to
display the command line and the output of the test. For example:
ctest -V -I 49,49
You can also specify tests using inclusion and exclusion regular expressions. For example:
ctest -R 'suppress|realloc'
will run all tests with the string "suppress" or the string "realloc" in their names, while
ctest -E alloc
will run all tests except those with alloc in their names.
If you are using CTest version 2.8 or later, you can run tests in
parallel by passing -jN to ctest on the command line:
ctest -j5
As an alternative to using -V to display the test command line and output
during executing, CTest stores that information (even when not run with -V)
in the Testing/Temporary/LastTest.log file in the build directory.
Each test produces two types of output: stdout and the Dr. Memory results file. Both are checked against expected output, stdout versus a .out file and the results file versus a .res file. The checking is not a regular diff and simply looks for each line. We may want to change this to a stronger diff with regular expression support in the future.
The tests/runsuite.cmake script is used to execute a series of builds and
test runs. Running make test in a single build directory is not
sufficient to test all of the configurations that we support.
The runsuite.cmake script is meant to be executed from an empty directory.
It creates a subdirectory for each build in the suite. Use ctest -S to
execute the script. Here is an example:
mkdir ../build_suite
cd ../build_suite
ctest -S ../drmemory/tests/runsuite.cmake
We recommend using Visual Studio 2010 and MSBuild, in which case pass the use_msbuild parameter:
mkdir ../build_suite
cd ../build_suite
ctest -S ../drmemory/tests/runsuite.cmake,use_msbuild
You can pass -V to ctest to see the results as the test suite runs. Note that it is normal to have ctest output strings such as "No tests were found!!!" as we have builds for which we run no tests (particularly in the short suite). It is also expected to have an error at the end of the suite (if run with -V):
CMake Error: Some required settings in the configuration file were missing:
CTEST_SOURCE_DIRECTORY = E:/derek/opensource/withwiki/trunk/suite/..
CTEST_BINARY_DIRECTORY = (Null)
CTEST_COMMAND = c:/PROGRA~2/CMAKE2~1.6/bin/ctest.exe
This warning is an artifact of how CTest assumes we've set things up and can be ignored.
At the end of the suite a file called results.txt will be created and displayed. It shows configure failures, build failures, and test failures.
Here are sample results from running the test suite:
### ============================================
RESULTS
drheapstat-dbg-32: all 19 tests passed
drheapstat-rel-32: build successful; no tests for this build
drmemory-dbg-32: all 23 tests passed
drmemory-rel-32: build successful; no tests for this build
final package: build successful; no tests for this build
Error in read script: /home/bruening/work/build/drmem_suite/src/tests/runsuite.cmake
The "Error in read script" is expected: ignore it.
Here is an example of running the test suite on Windows from the cmd shell on a new checkout, with the -V flag for more verbose output:
cd "c:\Program Files (x86)\Microsoft Visual Studio *"
VC\vcvarsall.bat
set DDKROOT=C:/derek/ddk
set FLEXROOT=C:/derek/flex_sdk_4.1
cd \derek\drmemory
svn checkout http://drmemory.googlecode.com/svn/trunk/ src
cd \derek\drmemory\build_suite
"c:\Program Files (x86)\CMake 2.8\bin\ctest.exe" -V -S c:/derek/drmemory/src/tests/runsuite.cmake
DynamoRIO is included inside the Dr. Memory source tree as an svn:externals property. We support both building that embedded DR along with pointing to an already-built separate DR export tree. Whenever a new feature from DynamoRIO is used within Dr. Memory, be sure to update the svn:externals following the instructions at UpdatingDR. In addition, be sure to run the pre-commit test suite using the embedded DR, or be very careful if using an external DR that it matches the svn:externals version.
