Two `action` keys in one "Perform Action“ obs filter?

[guoqing-noaa]

RDASApp/rrfs-test/validated_yamls/templates/obtype_config/aircar_airTemperature_133.yaml

Line 60 in 6f5ac85

action:

Here we have two actions:

For YAML itself, the second one will overwrite the first one. I don't know how JEDI handle this situation.

[guoqing-noaa]

@SamuelDegelia-NOAA @spanNOAA @delippi

[delippi]

I will double check how this is handled in JEDI

[SamuelDegelia-NOAA]

It would probably be cleaner to keep both actions under the same key, but I think this style is still working correctly. I tested a few different configurations just now with the above QualityMarker filter for aircar_airTemperature_133.yaml.

action:
  name: reject
Result nobs = 35767

action:
  name: reject
action:
  name: reduce obs space
Result nobs = 34019

action:
  name: reject
  name: reduce obs space
Result nobs = 34019

Note that listing reduce obs space first does not work since the reject action hasn't occurred yet:

action:
  name: reduce obs space
  name: reject
Result nobs = 35767

[delippi]

Thanks for testing this @SamuelDegelia-NOAA. I agree that keeping them under the same action with reduce obs space last looks the cleanest.

[guoqing-noaa]

@delippi @SamuelDegelia-NOAA Could you check with @danholdaway about this?

I might be wrong, but to my knowledge, PyYAML (I think JCB uses this) cannot handle duplicate keys (the last one will overwrite the previous one). But YAML itself does allow duplicate keys.

I am not familiar with the reduce obs space, but do we need to use it for lots of observation filters?
Current NCAR practice is to use one reduce obs space based on a lat/lon box boundary check covering the whole domain and other filters run as usual.

[SamuelDegelia-NOAA]

If having the duplicate keys is a problem with PyYAML then we should just go ahead and change it (though there is not a rush since we are not ready for JCB implementation).

Regarding reduce obs space, it is beneficial to include for any pre-filters since it will remove that observation from memory and save time during the hofx calculations (especially for GETKF where hofx is computed for ~10 modulated members per actual member). The action is especially beneficial for the timeOffset Domain check filter since our time window for some obs is very small and removing obs outside of the time window saves a lot of resources.

If we really wanted, there might be a way to add a final Perform Action filter where we check if QCflagsData (this is an internal variable that is updated for each QC check) is equal to zero and then perform reduce obs space just in that filter. That way we would not need it for each individual pre-filter. But I am not sure if this would work.

[delippi]

I believe we could remove the reject action since I think reject is what is done by default here. That would match some of the other filters better since they all apparently reject by default.

We also might've gone a bit heavy handed on applying the reduce obs space to every pre-filter. We might want to go back and fine tune that to see where it is needed performance wise. Not that important right now though.

[SamuelDegelia-NOAA]

Is there a downside to including reduce obs space on all pre-filters? I understand that this removes some of the info from the standard output. But that can be solved by removing the filter action temporarily for diagnostic purposes.

[delippi]

The only downside I can think of is being able to better track what is happening to those obs. Functionally, it seems better to use it as much as possible. I was just thinking we could investigate the tradeoffs in the future. I'm fine with how it is now.

[guoqing-noaa]

If having the duplicate keys is a problem with PyYAML then we should just go ahead and change it (though there is not a rush since we are not ready for JCB implementation).

It is not easy to change PyYAML as we are aware the Python dictionary does not allow duplicate keys. But we have other alternate options. I just bring this to attention since you guys want to transition to JCB which is based on PyYAML.

Thanks all for the good discussion! I need to learn more about reduce obs space. It is great to have a clean obs space as much as possible. But at the same time, it may also be fine if there are some extra "waste" obs in the obs space which does not cause memory issues. NCAR mostly tested radiance data and their simple reduce obs space worked well so far. But they did not tested such as dense mesonet observations.

[SamuelDegelia-NOAA]

It would be nice if there were a way to at least print out the number of obs removed by a filter using reduce obs space. Something worth investigating in the future.

[chunhuazhou]

This may be related to the discussion here. I was doing some single case tests to find the cause of the huge increments when assimilating T133 AIRCAR data, in which I found the passed observations along with the change in the yaml file:

This might not be very reliable comparison as I suspect the qc filter didn't work as expected due to the huge increments as shown below

[guoqing-noaa]

Ah, @chunhuazhou Thanks for the information.
Could you try to remove the second action key to see whether it help?

[chunhuazhou]

@guoqing-noaa The last row in the above screenshot has only one action key. Is that what you were asking?

[guoqing-noaa]

@guoqing-noaa The last row in the above screenshot has only one action key. Is that what you were asking?

Thanks for the information. I did not notice that. So I think the two "action" keys should be fine in you tests.