seq2seq model issues?

[jumerckx]

Hello @ggordonhall ,
I'm really interested in this seq2seq implementation.
When I run it there seems to be a problem with data not being defined?
I've tried hcat the batches and zip Xs and Ys together

Xs = [hcat(x...) for x in Xs]
Ys = [hcat(y...) for y in Ys]
data = zip(Xs, Ys)

but when I run the train function I get an error with the loss:
MethodError: no method matching loss(::Array{Int64,2}, ::Array{Int64,2}).

I was wondering if you perhaps have made progress model?
Thanks,
Jules

[ggordonhall]

Hi @merckxiaan,

This is unfinished and I haven't looked at it in a while. I was trying to follow the general structure of the phonemes example in the model zoo, but glancing back at my attempt I see there are several problems.

1. The embedding struct doesn't work. Perhaps it isn't needed as outlined here, but I tried one-hot encoding with a large vocabulary (10,000 words) and it was excruciatingly slow.

2. The observed data at the previous timestep (Yo) in the phonemes example isn't defined and so can't be passed to the decoder for teacher forcing.

3. My batching function appears to be incompatible as input to the LSTM and throws some MethodError. At the moment I'm not quite sure why.

I'll have a look back over the next few days and see if I can get it to work.

Cheers,
Gabriel

[jumerckx]

Thanks a lot for you response!
I myself have already tried creating a seq2seq translation model, but when I run it, the loss decreases up until a certain point but it never gives good results (sentences full of UNK tokens, translations that make no sense...).
Perhaps you could take a look at my code, a big part is data preparation, and things aren't organized as neatly as your code but here it is:
https://gist.github.com/merckxiaan/d3612c6d84e141e240b2c4e5f49a7218
Jules