Test via iojs throws but python/make do not report error

[Trott]

out/Release/iojs test/parallel/test-http-get-pipeline-problem.js throws:

Error: ENOENT: no such file or directory, open '/Users/trott/io.js/test/tmp/0.jpg

But python tools/test.py -v --mode=release parallel/test-http-get-pipeline-problem completes successfully.

Is this a bug or do I misunderstand how the tests are supposed to work?

[Trott]

out/Release/iojs test/parallel/test-http-get-pipeline-problem.js returns error code 1, so it's not just printing something but really honestly returning an error code. Checked with:

out/Release/iojs test/parallel/test-http-get-pipeline-problem.js || echo $?

[brendanashworth]

I've come across this before. I think that the python test suite creates the temporary directory (found in test/common.js?), but the individual test suites won't create it.

[Trott]

Yes, that's exactly right. The python test harness (or whatever the right word is) is creating the tmp dir and the command line is failing if I don't create the tmp dir it expects. Should a "create tmp dir if it doesn't exist" bit be added to the test file? Or is this just sort of a known gotcha and it should be left as is?

[Trott]

Not seeing any code anywhere in any of the other tests to create the temp dir if it doesn't exist, so I'm guess this is just something you just sort of have to know.

[Trott]

Oh, wait, that code could be stuck inside of common.js...

[brendanashworth]

Well, putting it in common.js would be best, but we don't want to remove it and add it every time a test is run (x1000 in the real test suite). I'm not really sure the best way to approach that, while maintaining a clean test run each time.

[Trott]

Even easier might be to check in test/tmp (since that is what the directory will always be called outside of the python multiple threads thing). Git won't allow empty directories, but we could add a .gitignore file. Not sure if there's a test that will end up deleting such a file, though...

I wouldn't want common.js to add it and remove it after every test, but just add it if it's not already there. But still, all those existsSync() calls can add up. I'll try it to see how much of a cost there is, and then probably abandon the idea once I see...

[brendanashworth]

Perhaps use a Getter for the fixtures directory, that's been done in the past for other things there.

[jbergstroem]

At the moment we kind of expect tests to be run through test.py. It creates one temp dir per parallel runner and does some other stuff. I'm not really against adding more to common.js, but I hope somewhere down the line we can move to a pure javascript runner and improve test information by switching fully to tap(3).

[Trott]

Adding this to common.js does not seem to noticeably impact the run time of make test-parallel:

if (! fs.existsSync(exports.tmpDir)) {
  fs.mkdirSync(exports.tmpDir);
}

Can someone do more rigorous benchmarks (or point me to docs or something on how to do it)? I'll stick it in a PR for now.

[chrisdickinson]

Alternatively, we could have the python test runner set an environment variable that common.js checks for the absence of to see if it needs to be responsible for creating the temporary directory.

[bnoordhuis]

@Trott if (!fs.existsSync(exports.tmpDir)) fs.mkdirSync(exports.tmpDir); is race-y when you run tests in parallel. It's better to just call fs.mkdirSync() and handle the EEXIST error.

I don't think that's enough, though. Tests don't just expect that the temp directory is there but that it's empty as well. In other words, if the directory exists, it needs to be cleaned out. And because it can contain subdirectories, you're going to have to do a recursive walk of the directory tree.

The test runner already handles all of that so it's basically duplicating existing functionality.

[silverwind]

I'm -1 on changing tests specifically to run standalone. It is documented that single tests should run through test.py. As @jbergstroem outlined, I think these kind of changes are best done in batch if we are to switch to a new test runner.