Skip to content
This repository has been archived by the owner on Dec 29, 2022. It is now read-only.

Sampled softmax loss support #147

Open
okuchaiev opened this issue Apr 6, 2017 · 1 comment
Open

Sampled softmax loss support #147

okuchaiev opened this issue Apr 6, 2017 · 1 comment

Comments

@okuchaiev
Copy link

okuchaiev commented Apr 6, 2017

Please consider adding sampled softmax loss, in addition to "cross_entropy_sequence_loss".
For tasks with large target vocabularies, the speedup can be significant (with, perhaps, minor accuracy loss per step).
Even on "nmt_large" config with batch size of 128 and voc size of 32,000 I am getting about x1.224 speedup.
It is, however, a little tricky to add. I have a first draft here: https://github.com/okuchaiev/seq2seq/tree/sampled_softmax_first_try
It is not ready to be merged yet.
Let me know if this is of interest - I plan to polish my implementation and would appreciate thoughts on what is the right way to add it here.

@sefialbo
Copy link

sefialbo commented Jun 5, 2017

is it ready to use ? this feature is really helpful

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants