Run each test method (rather than test class) in isolation

[jose]

Hi @jon-bell,

I have been wondering whether your tool could be used to run each test method in isolation rather than each test class, as original proposed...

So, in order to reset classes at test method level (i.e., after the execution of a test method) I made the following change to vmvm:

diff --git a/vmvm-ant-junit-formatter/src/main/java/edu/columbia/cs/psl/vmvm/AntJUnitTestListener.java b/vmvm-ant-junit-formatter/src/main/java/edu/columbia/cs/psl/vmvm/AntJUnitTestListener.java
index d9f4398..208eb19 100644
--- a/vmvm-ant-junit-formatter/src/main/java/edu/columbia/cs/psl/vmvm/AntJUnitTestListener.java
+++ b/vmvm-ant-junit-formatter/src/main/java/edu/columbia/cs/psl/vmvm/AntJUnitTestListener.java
@@ -15,7 +15,7 @@ public class AntJUnitTestListener implements JUnitResultFormatter {
 
 	@Override
 	public void endTestSuite(JUnitTest arg0) throws BuildException {
-		Reinitializer.markAllClassesForReinit();
+		
 	}
 
 	@Override
@@ -55,6 +55,7 @@ public class AntJUnitTestListener implements JUnitResultFormatter {
 	@Override
 	public void endTest(Test arg0) {
 		// TODO Auto-generated method stub
+		Reinitializer.markAllClassesForReinit();
 	}
 
 	@Override

Then I ran vmvm (with the above change) on a couple of Java projects and I found a particular corner case to which the execution of the resetter at test method level makes a particular test method to unexpectedly fail. Please find below a minimal project example that triggers the unexpected behavior. (For convenience, the same minimal project can be find in here).

build.xml

<?xml version="1.0" encoding="UTF-8"?>
<project name="VMVM Example" default="rebuild">
  <property name="ant.build.javac.source" value="1.8" />
  <property name="ant.build.javac.target" value="1.8" />

  <property name="src.dir" location="src" />
  <property name="build.dir" location="target" />
  <property name="classes.dir" location="${build.dir}/classes" />

  <!-- JUnit deps -->
  <property name="junit.jar" value="lib/junit-4.12.jar"/>
  <property name="hamcrest.jar" value="lib/hamcrest-core-1.1.jar"/>

  <!-- VMVM deps -->
  <property name="vmvm.jar" value="${vmvm}" />
  <property name="ant-mvn-formatter.jar" value="${vmvm_formatter}" />

  <target name="clean">
    <delete dir="${build.dir}" />
  </target>

  <target name="compile">
    <mkdir dir="${classes.dir}" />
    <javac srcdir="${src.dir}"
           destdir="${classes.dir}"
           debug="true"
           includeantruntime="false"
           deprecation="false"
           optimize="false">
      <classpath>
        <pathelement path="${junit.jar}" />
      </classpath>
    </javac>
  </target>

  <target name="test" depends="compile">
    <junit printsummary="yes" haltonfailure="yes" haltonerror="yes" fork="true" forkmode="once" showOutput="true">
      <jvmarg value="-Xbootclasspath/a:${vmvm.jar}" />
      <jvmarg value="-javaagent:${vmvm.jar}" />
      <classpath>
        <pathelement location="${classes.dir}" />
        <pathelement path="${junit.jar}" />
        <pathelement path="${hamcrest.jar}" />
        <pathelement path="${ant-mvn-formatter.jar}" />
        <pathelement location="${vmvm.jar}" />
      </classpath>

      <formatter type="plain" usefile="false" />
      <formatter classname="edu.columbia.cs.psl.vmvm.AntJUnitTestListener" usefile="false" />
      <test name="uw.edu.TestEnumXY" />
    </junit>
  </target>

  <target name="rebuild" depends="clean,compile,test" />
</project>

EnumXY.java

package uw.edu;

public enum EnumXY {

  X,

  Y

}

TestEnumXY.java

package uw.edu;

import java.util.Arrays;
import java.util.Collection;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(value = Parameterized.class)
public class TestEnumXY {

  private EnumXY xy;

  public TestEnumXY(EnumXY xy) {
    this.xy = xy;
  }

  @Parameters
  public static Collection<Object[]> data() {
    final EnumXY[] enums = EnumXY.values();
    final Object[][] data = new EnumXY[enums.length][1];
    for (int i = 0; i < enums.length; i++) {
      data[i][0] = enums[i];
    }
    return Arrays.asList(data);
  }

  @Test
  public void test() {
    if (this.xy == EnumXY.X) {
      System.out.println("x == x");
    } else if (this.xy == EnumXY.Y) {
      System.out.println("y == y");
    } else {
      throw new RuntimeException("");
    }
  }
}

Basically, this minimal project has an enum class EnumXY with two values (X and Y), and a parameterized JUnit test class TestEnumXY which should test the == of enum values. When the test class TestEnumXY is executed, it first collects all values required to execute each test method. In this case it collects the values of EnumXY.X and EnumXY.Y. The execution of the test method test checks whether the parameter value is == to any value in the enum class.

On the first execution of the test method test, the value of xy is == to the value of EnumXY.X and therefore the test finishes successfully. Once the first execution of the test method test finishes, vmvm marks EnumXY class to be resetted. On the second execution of the test method test, EnumXY class is resetted (as it has been marked to be and because there is a getstatic call in the test method to the enum class) and unexpectedly else if (this.xy == EnumXY.Y) { is no longer true and a runtime exception is thrown by the else block. The problem in here is that EnumXY.X and EnumXY.Y get new values when EnumXY class is resetted. Thus, on the second execution of the test method test, this.xy (which was initialized with the initial value of EnumXY.Y) is no longer == to the new EnumXY.Y value, which makes total sense, however it is not the expected behavior.

@jon-bell, although this issue report is not a true issue/problem of vmvm (as it has been proposed to run test classes in isolation and not test methods in isolation), I would like to have your opinion on this.

--
Best,
Jose

[jon-bell]

This is very tricky. The @parameters are almost acting as an @before, since they are setting up state. This is solved in the normal (non-parameterized) case, since a new instance of the test class will be instantiated for each test method (by JUnit, under the covers).

I can think of only very ugly solutions to this (e.g. when you reset things, first walk the entire heap to find references to the static fields you are blowing away, then correct those references to point to the right thing). This would be easier to pull off using CROCHET instead of VmVm, effectively doing a checkpoint before any tests run, then rolling back to that point. You could change the rollback code to also migrate old references to new ones.