Adding FoV detection, improved groundscatter flag

[aburrell]

I would like to add my automated field of view (FoV) detection and improved groundscatter flagging to RST. This won’t be simple, since we’ll need an “after fitacf, before grd” file as output that the make_grd routines can read. Would anyone be interested in helping this development and/or determining what these new files should look like and be called?

references: http://dx.doi.org/10.1002/2015RS005808, and another RS paper that's still under review or any of my conference presentations in the last year.

[kkotyk]

Can this be prototyped in python first? The python tools I started would make this way easier to start with since they can basically write any python dictionary to dmap.

[kkotyk]

If we have something finalized it would make it much more easy to translate this into RST.

[aburrell]

I have the fov code in davitpy already. The Ribiero-like reflagging doesn't work well in python (thanks to the depth-first search), but I have C code that works and has run on about 4 years of data so far. Right now this all works by outputting ascii files.

[kkotyk]

Sounds good. Well I definitely think we should figure out the scope of work and decide on the best of course of action from there! @egthomas will definitely know more about judging the scope of work of adding a feature to make_grid.

[egthomas]

@aburrell - so these will basically be files that contain all of the same information as a normal fit/fitacf file, but also some additional information about front/rear FOV and a more accurate ground scatter determination?

From working with the map files, we found it is not very difficult to add new data fields to dmap-format files. Maybe we could do something similar by adding additional data fields to the fit/fitacf format files? For example, upon creation with make_fit those fields would be empty, but after running your post-processing software those fields would be populated? There is already some precedent for this with the xtd flag in grid files which indicates whether power and spectral width information is stored in addition to the gridded velocity values.

An alternative would be to take the cfit approach where you create an entirely new file format and introduce another if/else branch within each binary for loading data from a particular file format. I would prefer to avoid this option as it would significantly increase the amount of new code to be written.

[egthomas]

Sorry, we wouldn't be adding new fields to fit-format files, only the fitacf (dmap) format files.

[aburrell]

I prefer the first option as well. From what I remember we'd need to add a FoV flag and decide whether or not to keep the old elevation angle. The groundscatter flag would have a new possible value, but the old values would mean the same thing and existing code should not be affected.

[aburrell]

Another thing to keep in mind is whether we want this code to be operational in real-time or not. I know how I'd like to structure the processing, but haven't needed it and so haven't mocked it up.

[egthomas]

When you say operational in real-time, do you mean at the radar site? Or something that can be located off-site and takes the real-time fitacf-type data stream as input (ie http://superdarn.ca/real-time-display)?