random number seed not honored for network weights?

[arnocandel]

This test fails: https://github.com/h2oai/deepwater/blob/99a61564ec6252770077c4f6367079cf18f4135b/backends/mxnet/test/seed_test.cxx

arno@mr-dl1:~/deepwater/backends/mxnet$ ./seed_test

lenet : FAIL
bb2daeba0973fdbe49579e4e16d57e4f /tmp/model
3e02d147aac07a152668cff3f805c489 /tmp/params

bb2daeba0973fdbe49579e4e16d57e4f /tmp/model
d9196629b49e53a10857fa5ef005deda /tmp/params

alexnet : FAIL
fb97eb805ea380cf6c1e4dba0d7a0d70 /tmp/model
f5f6e0a32fab82f72ce7ff830252cf85 /tmp/params

fb97eb805ea380cf6c1e4dba0d7a0d70 /tmp/model
c6a39ecc8e7e4f9594aca36f90c5eb5c /tmp/params

googlenet : FAIL
ba468ed48bd3c25ca875b26f7e9fb28a /tmp/model
e2abebb4e643fce688bf97474dd549d5 /tmp/params

ba468ed48bd3c25ca875b26f7e9fb28a /tmp/model
e3cc4a9b99851233a84d3357f8921dfb /tmp/params

inception_bn : FAIL
880152c49b87aeb350cacd3c1ca35dd9 /tmp/model
b90ee40eb3f7ad9120c7245206e7fa1b /tmp/params

880152c49b87aeb350cacd3c1ca35dd9 /tmp/model
9eed077ebf81cebf26405bbc72cf2134 /tmp/params

resnet : FAIL
60a3e94334228df1735f9b13152fcadb /tmp/model
7f6b9800775d51883dd68fff574fa0b4 /tmp/params

60a3e94334228df1735f9b13152fcadb /tmp/model
3edc67c4b11ea6102cf7d3c40b688404 /tmp/params

vgg : FAIL
137925f6cc31061ee87b54a93c7bc645 /tmp/model
731a90eea09d75ea2f9dfa094a13f81a /tmp/params

137925f6cc31061ee87b54a93c7bc645 /tmp/model
d3ad3de6ce32f95d16178c8977ecf75c /tmp/params

@thirdwing Any ideas/suggestions?

[arnocandel]

Looks like it's only happening on GPUs (with and without CUDNN). Reproducible results on the CPU. Note that this happens without actual training, even when commenting out the Xavier initialization in image_train.cxx.

This issue said that CUDNN was an issue: apache/mxnet#1594, not sure why non-CUDNN CUDA is also not reproducible...

[arnocandel]

The sequence on non-reproducibility is at least reproducible - so we need to restart the seeding on the GPU side of things somehow.

lenet : FAIL
bb2daeba0973fdbe49579e4e16d57e4f /tmp/model
3e02d147aac07a152668cff3f805c489 /tmp/params

bb2daeba0973fdbe49579e4e16d57e4f /tmp/model
d9196629b49e53a10857fa5ef005deda /tmp/params

Also, the GTX1080 and the Titan-X lead to the same initial conditions, which is nice.

[arnocandel]

Something is calling CURAND one extra time...

arno@ovaclokka:~/deepwater/backends/mxnet$ ./seed_test
Calling setSeed
Set CURAND seed to 156718
Calling setSeed
Set CURAND seed to 0
Set CURAND seed to 156718

lenet : FAIL
bb2daeba0973fdbe49579e4e16d57e4f /tmp/model
3e02d147aac07a152668cff3f805c489 /tmp/params

bb2daeba0973fdbe49579e4e16d57e4f /tmp/model
d9196629b49e53a10857fa5ef005deda /tmp/params

Modified:
+++ b/mshadow/random.h
@@ -257,6 +257,7 @@
+ std::cerr << "Set CURAND seed to " << seed << std::endl;

[arnocandel]

hardcoded all seeds to 1234 in mxnet/mshadow/mshadow/random.h and mxnet/src/resource.cc, and still get this:

arno@mr-dl1:~/deepwater/backends/mxnet$ ./seed_test
Random Seed0: 1234
Random Seed1: 1234
Random Seed1: 1234
Random Seed1: 1234
Random.h Seed2 1234
Random Seed1: Random.h Seed2 1234
1234
Random Seed1: 1234
Random.h Seed2 1234

lenet : FAIL
bb2daeba0973fdbe49579e4e16d57e4f /tmp/model
ee04f0219be950d1ab651930754d3842 /tmp/params

bb2daeba0973fdbe49579e4e16d57e4f /tmp/model
cae11f206fd40a78f3a7ba75ea8204ce /tmp/params

This means that the number of calls to RandNext() is different, or the seed is not honored by CUDA?

[danithaca]

Any update since this is still open? We got similar problem where we could not reproduce results with GPU, but was able to reproduce results with CPU.