Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting first and last hidden outputs for BiLSTM with variable-length sequences #363

Open
ccsasuke opened this issue Nov 30, 2016 · 2 comments

Comments

@ccsasuke
Copy link

Hi,

I have a zero-masked bidirectional RNN for mini-batched data with variable lengths. I want to extract the last hidden output of the forward pass and the first hidden output of the backward pass as the representation of the sequence.

Since each sequence in the batch has different lengths, I can think of two possible ways to implement this:
i) Prepare two versions of the data, one with zeros padded on the left while the other with zeros padded on the right. The forward pass uses the left-padded data and use SelectTable(-1). Similarly, the backward pass uses the right-padded data and use SelectTable(1).

ii) Alternatively, I can implement a custom MaskZeroSelectTable(-1) to select the last hidden output that is not 0.

I guess this is not an uncommon usage, but I could not find an existing solution. Am I missing anything?
If there is no over-the-shelf implementation, what is the best way to implement this?

@ccsasuke ccsasuke changed the title Getting first and last hidden outputs for BiLSTM with variable-length batches Getting first and last hidden outputs for BiLSTM with variable-length sequences Dec 1, 2016
@aron-bordin
Copy link

Hi @ccsasuke, I'm facing the same issue. Have you found any good solution ?

@ccsasuke
Copy link
Author

Hi @aron-bordin, it seems this didn't catch the attention of the devs.. I switched to some other framework for this function. Maybe you can try the workaround i) I mentioned?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants