Get started!

What are you getting yourself into?
You are entering the world of continuous testing. This is sort of your own little personal continuous integration system. Instead of building your source, looking up the tests you want to run and then run them you only press save. Saving a file will trigger a build of relevant projects and running all tests related to these projects. Which means that if you want to you can now do your development in your favorite text editor and still have builds and test runs covered. AutoTest.NET should also work with any OS supporting mono.

Initial setup
The first thing you need to do is to get AutoTest.NET. You can either just fetch the latest binaries from the download section or build the project from source. If you choose the binaries just unpack the zip file to any directory (write access is needed for AutoTest.NET within the directory). If you’re running on a 64bit windows installation with Visual Studio 2010 it’s all ready to go. If not your next step should be the AutoTest.config file (or maybe even README?). Inside the config the most important thing would be the BuildExecutable and and MSTest/MSpec runner if that’s what you’re using. NUnit and XUnit are configured out of the box with AutoTest.Net’s internal runner. First of the BuildExecutable setting supports Monos xbuild and Microsofts MsBuild. If you want to you can actually specify different versions of MsBuild for VS 2008 and VS 2010. If you leave out the BuildExecutable setting all together AutoTest.NET will stop tracking file changes and start tracking changes to assemblies so you can use Visual Studio to handle builds. Next you would want to set the test runner of your choice to wherever you have it located. That would be all changes required to get started.

Running your first build and testrun
After doing the initial configuration you can start the AutoTest.NET console or winforms application. When starting up the application it will ask you to set a watch directory. In both console and winforms app you can specify watch directory as the first command line parameter. The watch directory can be any directory you want AutoTest.Net to track or it can be a solution file. The watch directory is the top directory from where AutoTest.NET recursively looks for file/assembly changes. After setting the watch path you can go ahead and change a file in your project. As soon as you save it you should see AutoTest.NET detecting changes, building and running tests. If you have choosen assembly tracking you would not save the file but build your project in Visual Studio. If AutoTest.NET now starts looping your solution is generating files outside the typical output path. To solve this have a look at the Ignore file section belolw.

What is going on?
To get a bit of information about what is going on there are a couple of sources of information. Inside the winforms app there is a button at the top right. This button will take you to the application messages windows. This window shows you some usefull info about how AutoTest.NET is configured and will keep you informed when anomalies happen. If you really want to get under the hood of what’s happening you can turn on logging and the application will start logging to the debug.log file in the AutoTest.NET directory. Hopefully this will help you if something doesn’t behave like you would expect it to.

Notifications
AutoTest.NET is designed to stay out of your way as much as possible. To truly be out of your way it supports a couple of notification frameworks. For Windows and Mac you can use Growl (Growl or Growl for Windows) and for Linux you can use notify-send (sudo apt-get install notify-osd). notify-send doesn’t need any configuration other than making sure notify-osd is installed. It will also try it’s best to configure Growl but if it can’t find the growlnotify application you can refer to it using the config option with the same name. When set up you will get notifications on run started and run finished. Whether you want notifications on run started and run finished is also configurable. The winforms and console UI both supports notifications.

More configurable options
Ignore file
Sometimes you’ll have files generated by builds or test runs. This results in AutoTest.NET looping and becoming pretty useless. And btw, to figure out what files are causing it to loop turn on logging and check the debug.log. To avoid this you can set up a file containing patterns to ignore. This can be configured through the IgnoreFile option. You can set the IgnoreFile option to an absolute path or a path relative to the watch directory. The file is based on the .gitignore style content (yes you can use your existing .gitignore file with AutoTest.NET). The following would be a valid ignore file:

# Making sure the file IgnoreThisOne.xml is ignored
IgnoreThisOne.xml

# Should ignore any file or directory called TestResults
TestResults

# Should ignore all output xmls
*output.xml

Ignore certain test assemblies
To ignore test assemblies through naming you can use the ShouldIgnoreTestAssembly/Assembly configuration option. Specify assemblies like SomeAssembly.dll or *SystemTests.dll.

Ignore test categories in NUnit
You can set up AuotTest.NET to ignore certain named categories. Use the ShouldIgnoreTestCategories/Category configuration option to name categories.

IDE interaction
To be able to through the winforms app open your IDE on the file and line specified use the CodeEditor configuration option. Just specify your IDE and if it’s supported the argument options to open the IDE with the right file on the right line.

Project specific configurations
If you want some projects using different configuration than others you can use a local configuration. Just place an AutoTest.config file in your watch directory and when AutoTest.NET starts it will detect the file and merge it with the base configuration. Your local configuration can look something like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <!-- Exclude all build executable configuration all together -->
    <BuildExecutable override="exclude"></BuildExecutable>

    <!-- Merge these changes in with what's already in the base configuration -->
    <!-- The overide attrubte will be read for the first occurrence in the setting family --> 
    <NUnitTestRunner framework="v3.5" override="merge">NewTestRunner for fw 3.5</NUnitTestRunner>
    <NUnitTestRunner framework="v4.0">NewTestRunner for fw 4.0</NUnitTestRunner>
    
    <!-- This will just override whatever XUnit config is in the base configuration -->
    <XUnitTestRunner>C:\XUnit\xunit.console.exe</XUnitTestRunner>

    <!-- merges in some extra assemblies to ignore -->
    <ShouldIgnoreTestAssembly override="merge">
        <Assembly>*System.dll</Assembly>
    </ShouldIgnoreTestAssembly>
</configuration>

Building from source
To build the whole thing from source just clone or fork the repository. When done go in and run build and deploy scripts (biuld.sh/build.bat and deploy.sh/deploy.bat depending on whether you are on Windows(bat) or something else(sh)). After running build and deploy scripts you will find the built binaries under the ReleaseBinaries folder.