Multi-function recurrent unit

[juesato]

Implementation of #269

A few notes:

• The model seems to not train very well using simple SGD, or at least requires lower learning rates than typical LSTM/GRU networks. Using the same hyperparameters as the LSTM recurrent language model example, the model doesn't look like it's converging and training PPL doesn't drop below 300.
• I added an Adam option to the language model script, since the original Mufuru paper also uses Adam, and this seems to address these issues. (I also checked that this preserves behavior when using SGD by running the LSTM example and making sure the stats are the same - converges to training PPL 50, valid PPL 120, and test PPL 116 after 7 epochs before and after the change.)
• Right now, I have the max and min options turned off, because until I merge Make CMaxTable and CMinTable cunn-compatible torch/nn#954, the CMaxTable and CMinTable options aren't compatible with CudaTensors. However, this makes the model considerably worse, converging to validation PPL 164 rather than 134.
• The relative performance of different models are similar to the results in the paper, but the absolute losses don't match up with the paper. I emailed Dirk Weissenborn about this, and he was kind enough to share his Tensorflow training scripts. In Tensorflow, both MFRU and GRU converged to about 144 validation PPL. In this Torch implementation, MFRU and GRU (implemented as MFRU) converge to about 134 validation PPL. GRU, however, converges to 121 on validation set, which may be due to the forget gate biases.
• I have a spreadsheet summarizing the experiments I've run so far here: https://docs.google.com/spreadsheets/d/1snsSUCqv0frfjBztkkwONfiMT5kyKjzJQr-KI1F1qvE/edit?usp=sharing

I think this should merge after torch/nn#954 since until then the performance is poor, but I wanted to get feedback on this before adding unit tests and documentation.

Let me know if you have any questions!

@nicholas-leonard

[nicholas-leonard]

@juesato @JoostvDoorn @jnhwkim can you review this? Minimum requirements: documentation (README.md), unit tests and code.

[jnhwkim]

@nicholas-leonard For me, I should have to read the paper first. I'll revisit here to catch up.

[nicholas-leonard]

@juesato I would really like to see this get merged. Any developments?

[juesato]

@nicholas-leonard Sorry, I left this hanging. I'l spend the next two hours working on this (speeding up the CMaxTable stuff, adding docs, and unit tests), and if I don't finish then, I'll continue tomorrow.

[juesato]

@nicholas-leonard I got some time to spend on this this weekend, but found a bug, and need a bit more time. I'm hoping to finish up after work tomorrow, but I think it's fairly likely I won't have time, in which case I'll finish on Tuesday.

[juesato]

@nicholas-leonard I added unit tests and documentation, and I believe this should be ready to merge.

As a future reference, I'm going to leave the training curves on PTB here.

Command (I added support for Adam, since that's what's used in the original paper):

CUDA_VISIBLE_DEVICES=0 th examples/recurrent-language-model.lua --progress --cuda --mfru --seqlen 20 --uniform 0.1 --hiddensize '{200}' --batchsize 32 --maxepoch 20 --device 1 --adam --startlr 0.01 --cutoff 5

Epoch #1 :	
 [======================================== 29048/29048 ================================>]  Tot: 2m18s | Step: 4ms       
learning rate	0.009975025	
mean gradParam norm	8.4984952035468	
Speed : 0.004788 sec/batch 	
Training PPL : 282.28724615675	
Validation PPL : 195.89575663903	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485858192:1.t7	
	
Epoch #2 :	
 [======================================== 29048/29048 ================================>]  Tot: 2m17s | Step: 4ms       
learning rate	0.00995005	
mean gradParam norm	9.6138208177836	
Speed : 0.004739 sec/batch 	
Training PPL : 138.24728917969	
Validation PPL : 152.95716046988	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485858192:1.t7	
	
Epoch #3 :	
 [======================================== 29048/29048 ================================>]  Tot: 2m17s | Step: 4ms       
learning rate	0.009925075	
mean gradParam norm	10.585762301436	
Speed : 0.004736 sec/batch 	
Training PPL : 99.841825199886	
Validation PPL : 138.14327248568	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485858192:1.t7	
	
Epoch #4 :	
 [======================================== 29048/29048 ================================>]  Tot: 2m17s | Step: 4ms       
learning rate	0.0099001	
mean gradParam norm	11.394943580532	
Speed : 0.004733 sec/batch 	
Training PPL : 78.723720426491	
Validation PPL : 133.5123597645	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485858192:1.t7	
	
Epoch #5 :	
 [======================================== 29048/29048 ================================>]  Tot: 2m17s | Step: 4ms       
learning rate	0.009875125	
mean gradParam norm	12.078977865072	
Speed : 0.004733 sec/batch 	
Training PPL : 65.066691855094	
Validation PPL : 134.65883864519	
	
Epoch #6 :	
 [======================================== 29048/29048 ================================>]  Tot: 2m17s | Step: 4ms       
learning rate	0.00985015	
mean gradParam norm	12.725478478451	
Speed : 0.004731 sec/batch 	
Training PPL : 55.300139163342	
Validation PPL : 139.33760207709	

As a baseline, if we swap out GRU for MuFuRU, we get this curve

Epoch #1 :	
 [======================================== 29048/29048 ================================>]  Tot: 1m36s | Step: 3ms       
learning rate	0.009975025	
mean gradParam norm	7.7288590271154	
Speed : 0.003351 sec/batch 	
Training PPL : 252.14280286606	
Validation PPL : 178.4052220798	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485859742:1.t7	
	
Epoch #2 :	
 [======================================== 29048/29048 ================================>]  Tot: 1m37s | Step: 3ms       
learning rate	0.00995005	
mean gradParam norm	8.4460625165352	
Speed : 0.003365 sec/batch 	
Training PPL : 127.50807904334	
Validation PPL : 141.2727270623	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485859742:1.t7	
	
Epoch #3 :	
 [======================================== 29048/29048 ================================>]  Tot: 1m37s | Step: 3ms       
learning rate	0.009925075	
mean gradParam norm	9.040807982285	
Speed : 0.003365 sec/batch 	
Training PPL : 94.050081227786	
Validation PPL : 127.12406725727	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485859742:1.t7	
	
Epoch #4 :	
 [======================================== 29048/29048 ================================>]  Tot: 1m37s | Step: 3ms       
learning rate	0.0099001	
mean gradParam norm	9.548776472761	
Speed : 0.003364 sec/batch 	
Training PPL : 75.183904110218	
Validation PPL : 121.60937118905	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485859742:1.t7	
	
Epoch #5 :	
 [======================================== 29048/29048 ================================>]  Tot: 1m37s | Step: 3ms       
learning rate	0.009875125	
mean gradParam norm	10.028425562097	
Speed : 0.003355 sec/batch 	
Training PPL : 62.757740662728	
Validation PPL : 121.03703331266	
Found new minima. Saving to /var/storage/shared/mscog/sys/jobs/application_1485367302006_0415/save/rnnlm/ptb:phlrr1015:1485859742:1.t7	

So the training loss looks similar across the two, but generalization seems better with GRU here.
Haven't done any sort of hyperparameter search here, just thought that this could be useful info for users looking to sanity check in the future.

[nicholas-leonard]

@juesato Thanks for following through on this to the end!