Wrong `target_times_raw` in masking mode

[Jubeku]

What happened?

When masking mode is on, the target_times_raw attribute of the streams data class is corrupted and doesn't return the expected timestamps.

What are the steps to reproduce the bug?

Load a batch item in masking mode, assuming the ERA5 stream.
We should get int(len_hrs / step_hrs) unique timestamps which we can get as follows:

stream_data = batch[0]
i_batch = 0
i_strm = 0
fstep = 0
t_unique = np.unique(streams_data[i_batch][i_strm].target_times_raw[fstep])

See t_unique outputs for several example cases below, compared with the desired output in forecast mode.

Version

develop

Platform (OS and architecture)

Linux

Relevant log output

## len_hrs = 6, step_hrs = 6, mode = "masking"
t_unique = array(['NaT'], dtype='datetime64[us]') 
  
## len_hrs = 6, step_hrs = 6, mode = "forecasting"
t_unique  = array(['2021-10-03T00:00:00'], dtype='datetime64[s]') 

## len_hrs = 24, step_hrs = 6, mode = "masking"
t_unique  = array(['2021-11-21T04:00:00.000000', 'NaT'], dtype='datetime64[us]')

## len_hrs = 24, step_hrs = 6, mode = "forecasting"
t_unique  = array(['2021-07-01T12:00:00', '2021-07-01T18:00:00', '2021-07-02T00:00:00', '2021-07-02T06:00:00'], dtype='datetime64[s]')

Accompanying data

No response

Organisation

ECMWF

[Jubeku]

cc @MatKbauer, @shmh40, @clessig

[mlangguth89]

I think, this is a general issue - not only masking mode, cf. #543

[clessig]

I think, this is a general issue - not only masking mode, cf. #543

It worked in forecast mode for @Jubeku . We didn't check for correctness but it wasn't NaT (not a time).