Allow filtering via masking

[nevencaplar]

I wanted to reproduce the following pandas workflow (all_forced_sources_w19 is flat pandas df with sources):

'''Identify all columns that contain 'flag' '''
flag_cols = [col for col in all_forced_sources_w19.columns if 'flag' in col.lower()]

'''Exclude rows where any flag column is True'''
flag_mask = ~(all_forced_sources_w19[flag_cols].any(axis=1)) # True where all flags are False
all_forced_sources_w19_clean = all_forced_sources_w19[flag_mask]

The solution we found via .query was something like

'''Identify flag columns'''
flag_cols = [col for col in dia_object_lc_computed.diaObjectForcedSource.nest.fields if 'flag' in col.lower()]
'''Build the condition string, e.g., "flag1 == False & flag2 == False & ..." '''
query_str = " & ".join([f"diaObjectForcedSource.{col} == False" for col in flag_cols])
'''Apply query directly to the nested column'''
dia_object_lc_computed_filtered = dia_object_lc_computed.query(query_str)

I wish I was able to more directly mask, something like

flag_mask = ~(dia_object_lc_computed.diaObjectForcedSource.nest.fields[flag_cols].any(axis=1))
dia_object_lc_computed.diaObjectForcedSource = dia_object_lc_computed.diaObjectForcedSource[flag_mask]

[hombit]

I believe you meant no ".fields" here?

dia_object_lc_computed.diaObjectForcedSource.nest.fields[flag_cols].any(axis=1) -> dia_object_lc_computed.diaObjectForcedSource.nest[flag_cols].any(axis=1)

Note on the current implementation.
Both nf.lc.nest[["t", "m"]] and nf[["lc.t", "lc.m"]] return a new pd.Series with NestedDtype dtype, with the given fields selected—not a flat dataframe with those fields.

Support of boolean nested masks
dia_object_lc_computed.diaObjectForcedSource[flag_mask]
We absolutely should support this, I'm opening a sub-issue. We can implement it so that if the indexer is a Series with the same index as the "flatten" representation, then it filters that representation.

Options for "flatten" mask creation with .any() / .all()

1. We can change the current behavior so that nf.lc.nest[["t", "m"]] and/or nf[["lc.t", "lc.m"]] return a flattened dataframe, not a nested Series. Then we can do .any(axis=1) on the result dataframe.

2. We can recommend using nf.lc.nest.to_flat(["t", "m"]).any(axis=1), but the .nest interface is currently considered an expert interface.

3. We may support nf.lc.any() by implementing .any() for NestedExtensionArray, which would try to cast each sub-array to boolean and raise an exception if it couldn't. Currently, neither we nor pandas’ ArrowExtensionArray implement this.

I'd prefer the third option, so you can do nf.lc[['flag1', 'flag2']].any() / nf[['lc.flag1', 'lc.flag2']].any(). I'm opening a sub-issue for that, but feel free to disagree.

Getting a list of the subcolumns
I'm not sure if dia_object_lc_computed.diaObjectForcedSource.nest.fields should be a recommended way, again, .nest is an expert interface. I think we should recommend using nf.all_columns (a subject to rename because of lsdb.Catalog.all_columns) or nf.lc.dtype.fields for that.