Nopol patch for Math-76 (Defects4J) is not test-adequate (plausible)

[rjust]

The Nopol patch for Math-76 (result.json) is not test-adequate, but it should be according the experiment's methodology.

Is this the patch that Nopol generated for Math-76? If not, where can we find the actual patch?

Thanks,
René

[fermadeiral]

Hi @rjust,

Thank you for reporting this issue. We are going to investigate what happened and if this is an issue in RepairThemAll or Nopol (note that RepairThemAll just calls repair tools as they are). In any case, we will come back here when we find the origin of the issue and the fix for it.

Fernanda

[rjust]

Thanks @fermadeiral!

Can you please clarify whether the result.json file is generated by Nopol or by RepairThemAll? Likewise, is the detailed-result.json file generated by Nopol or by RepairThemAll?

Thanks,
René

[fermadeiral]

• result.json is generated by RepairThemAll, it's a normalized repair tools' output (but the patch inside it is returned by the repair tool, i.e. Nopol in this case).

• detailed-result.json contains a json outputted by the repair tool, i.e. Nopol.

Just to be clear on my previous comment, we need to think on the current issue because we use Nopol as a test-suite-based repair tool, so we expect that the patches generated by Nopol are test-suite adequate ones. Maybe we are using a buggy version of Nopol. Maybe it's also a problem on our side. We need to investigate it to be sure.

Fernanda

[tdurieux]

@rjust has explained in the email, it is the textual representation of the patch that is incorrect.
Sometimes, Nopol is failing to generate the diff (btw APR have the same issue for different reasons).
If you find a case like this I invite you to look at the end of the repair.log file and you will see the actual generated patch (not presented as a diff).

However, this case is extremely strange. The generated patch is indeed not patching adequate but I have no idea how it is possible since Nopol should run all the tests at the end of the execution to check the validity of the patch.

It can also be related to a test that output: This test can fail randomly due to sampling error with probability 0.001. Flaky tests are really messing up the APR.

[fermadeiral]

@rjust has explained in the email

Which email?

However, this case is extremely strange. The generated patch is indeed not patching adequate but I have no idea how it is possible since Nopol should run all the tests at the end of the execution to check the validity of the patch.

Like I said in this discussion thread, we might be using a very buggy version of Nopol. Did we use a release version or not?

[tdurieux]

Like I said in this discussion thread, we might be using a very buggy version of Nopol. Did we use a release version or not?

Nopol does not have development based on releases.

[rjust]

Hi @tdurieux,

Thanks for following up.

it is the textual representation of the patch that is incorrect.
Sometimes, Nopol is failing to generate the diff (btw APR have the same issue for different reasons).
If you find a case like this I invite you to look at the end of the repair.log file and you will see the actual generated patch (not presented as a diff).

Can you please let me know what the correct textual representation for the generated patch should be? Regarding your suggestion about the actual patch described in repair.log, I am assuming you are referring to the following output:

----PATCH FOUND----
 - mean <= 0
 - Nb test that executes the patch: 6
 - org.apache.commons.math.random.RandomDataImpl:339: CONDITIONAL

My understanding is that Nopol generated a patch by changing the conditional in line 339 to mean <= 0 -- is this the correct interpretation of this output?

Thanks,
René