Frontend `conv`/`pool` 'same' padding with striding is inconsistent

[albertz]

In the RF PyTorch backend for conv, the padding we add there seems to be off. Specifically, currently we do:

pad = s.dimension - 1 - (src_raw.shape[2 + i] - 1) % stride_
pad_left = pad // 2
pad_right = pad - pad_left

However, I noticed, I need to do this instead so that it matches when operating on a single seq:

pad = s.dimension - 1
pad_left = pad // 2
pad_right = pad - pad_left - (src_raw.shape[2 + i] - 1) % stride_

Now I'm not exactly sure how to introduce this change. That is maybe for another new behavior version? And/or also an explicit flag for it (which would mostly be just to test the direct difference of this).

Edit This is not only PyTorch. The TensorFlow backend behaves the same way. Now fixed via #1696.

[albertz]

I realize, this problem is related to #1434, where we introduced the "same_static" padding type for the TF ConvLayer/PoolLayer (cc @robin-p-schmitt). We did this with the same motivation:

to guarantee that the convolution and pooling operations lead to the same output for single seqs and batched seqs, which previously [with normal "same" padding] was not the case.

[albertz]

I think the current code is actually correct in the way that it is consistent to how "same" padding is done in TF. However, this logic, as we correctly observed here, can give inconsistent results when comparing batched seqs vs a single seq. The same problem was observed in #1434 and the "same_static" padding was added for that.

I'm not sure that I would do it this way. I think better is a dedicated option for this, something like rf_use_same_static_padding: bool or so (and have this enabled by default in a new behavior version). And then, if this is set, it would use the "same_static" padding instead of "same" padding (both for TF and PT).

[albertz]

I also wonder about the padding in our pool function. Currently we do:

pad = s - 1 - (src_raw.shape[2 + i] - 1) % strides[i]
padding.append(pad // 2)

And this padding is then passed directly to the Torch pool function. I think this corresponds to:

pad = s - 1 - (src_raw.shape[2 + i] - 1) % strides[i]
pad_left = pad // 2
pad_right = pad // 2

So it seems a bit inconsistent to how we do padding in conv?
Edit But then it uses ceil_mode=True.