Question about the correctness of lengths_to_mask function in model.py

[hueda8]

Sorry to bother you when you are busy.
I’d like to ask you a question about the code.

In model.py, we have the following functions:

def lengths_to_mask(lengths, length=None):
    if length is None:
        length = tf.reduce_max(lengths)
    mask = tf.reshape(tf.range(length), [1, length]) - np.array(0.5)
    mask = tf.cast(mask, tf.float32)
    lengths = tf.cast(tf.reshape(lengths, [-1, 1]), tf.float32)
    mask = step_function(lengths - mask)
    return mask

# heaviside step function
def step_function(inputs, dtype=None):
    dtype = dtype or inputs.dtype
    x = (tf.sign(inputs) + 1) / 2
    if x.dtype != dtype:
        x = tf.cast(x, dtype)
    return x

Are these functions correct?

When applying it as

cross_entropy = nll * lengths_to_mask(lengths + 1, tf.shape(nll)[1])

I feel that the mask does not work properly, because lengths_to_mask(lengths + 1, tf.shape(nll)[1]) outputs vectors, such as [1, 1, ..., 1, 1, 0.5, 0, ..., 0], ... .
Is my understanding wrong?

I would greatly appreciate your reply.

[hueda8]

In addition, in train.py, regarding the gradient clipping code:

# Optimizer and gradient clipping
optimizer = tf.train.AdamOptimizer(learning_rate=hparams['learning_rate'])
gvs = optimizer.compute_gradients(loss)
with tf.variable_scope('clipping'):
    gvs = [(None if grad is None else tf.clip_by_norm(grad, 0.2),
            var) for grad, var in gvs]

I believe that using tf.clip_by_norm(grad, 0.2) fixes the "clip_grad_norm" value to 0.2.
Is my understanding correct?

I apologize for asking you questions in succession.
Please feel free to reply whenever you have time.