Improve GPU performance

[ibab]

The performance of the network on GPUs seems to be lagging behind the CPU performance.
I suspect that this is because the 2D convolution isn't designed to work efficiently if the height of the input is 1.
It shouldn't be too difficult to write some custom code to perform an efficient 1D convolution.
For example, fft could be used for this.

[bhack]

tensorflow/tensorflow#1136

[ibab]

Just compared GPU and CPU again, and this time training is significantly faster on the GPU.

[lemonzi]

Is this solved then?

[ibab]

We would probably be able to train larger networks (and with larger sample sizes) if we had a single C++ op that does causal 1d convolution, as we would be able to avoid all the extra copying that's currently being done.
But writing and maintaining the op might be a pain :/

[lemonzi]

I've done it before, but not with gpu support...

On Thu, Oct 6, 2016, 18:58 Igor Babuschkin notifications@github.com wrote:

We would probably be able to train larger networks (and with larger sample
sizes) if we had a single C++ op that does causal 1d convolution, as we
would be able to avoid all the extra copying that's currently being done.
But writing and maintaining the op might be a pain :/

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#25 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADCF5gay3mfBQ0wKyF2mamL0xFbu5kq9ks5qxX0mgaJpZM4J_k-r
.

[vinhngx]

@ibab I've made a pull request for mixed precision training
#384

and would like to encourage the authors and community to give it a try :)