Problem with `trim_raw` start time filtering

[ksterne]

Question

So, in short, when using the -sd -st options with trim_raw should the output file not include records before the time and date specified, even if the values given don't exactly match those in the file records?

Category

• Usage
• Troubleshooting
• Software Design
• Data related
• Capabilities
• Clarity
• Workflow

Details

So, I'm looking into an issue that @egthomas and I have found on the DDWG where there's a few bpk rawacf files that have an issue of the first record having the date at Jan. 1, 1970. With regular make_fit processing, this record causes the processes to stop since the time/date is before the start of the radar operations from the radar.dat file. The next record in the rawacf file has a good time that doesn't exactly match the timestamp from the filenaming convention, but it's within a minute or so. So, it seems as though if we could remove the first record, the rest of the file can be usable again.

So, I remembered there's a few trimming binaries with rst and looked into the options for trim_raw. See the example below for how I'm using it. So, my first question is: should I be expecting the first record from 1970 to be skipped over and then the data would start at 20240325.2342.02 (the first good record in the file)?

Using the example below on the release/rst-5.1 branch, I'm still seeing the first erroneous record in the output file.

I've looked more into this and maybe have some places where things aren't working, but I want to make sure I'm using the trim_raw program correctly and it's not a case of user error. I can see about providing the bpk rawacf file that is used in the example if someone needs it.

Example

trim_raw  -sd 20240325 -st 20:40 -ed 20240326 -st 0000 20240325.2341.15.bpk.a.rawacf > 20240325.2341.15.bpk.a.rawacf.trimmed

Code

Installation Process

Using the release/rst-5.1 branch to start on an Ubuntu 22 virtual machine

Documentation

[egthomas]

@ksterne there is a related DDWG email thread from ~6-7 years ago where Simon and I raised the issue of ~25,000 dat files with incorrect / invalid / non-sequential time stamps. It was assigned as "Issue 35" but I don't know if that tracking system still has any meaning, and the issue itself was certainly never addressed.

[ksterne]

Sorry for any confusion here...I think I've been looking at this for too long today. This is a question about the trim_raw binary and not the trim_grid binary.

Thanks for the reminder about the non-sequential records in those files that were found a number of years ago. While problems with the data that are something for the DDWG, my question is more about the usage and expected results from the trim_raw binary. Was there a solution to the dat files issue that included the use of trim_raw?