Support for observationUnitLevelName for GET /variables API

[abatac]

Is it possible to filter the variables associated to study by observationUnitLevelName in GET /variables?

In the Breeding Management System, a study may comprise multiple datasets used to group observations/sub-observations based on their levels. For instance, a dataset named "plot" includes observations for plot-level units, while a dataset named "plant" contains sub-observation units within each plot.

The challenge is that, despite belonging to the same study, each dataset in BMS can have a different set of observation variables.

Therefore, if I wish to retrieve variables specific to the "plant" dataset (level), it is not feasible to utilize the GET /variables?studyDbId=xxx endpoint, as it retrieves all variables within a study without providing a way to filter variables based on the observationUnitLevelName.

I know that this is a limitation in the Brapi spec, but this blocks the interoperability of Field-book with BMS BrAPI 2.0 implementation.

For example, If I import a Field via BrAPI using "plot" observation level, the traits retrieved from GET /variables?studyDbId endpoint will return all the variables in a study for all datasets (levels). Ideally, it should only return the variables based on observationUnitLevelName.

cc @BrapiCoordinatorSelby @mcrimi

[BrapiCoordinatorSelby]

How many variables are there typically per data set or obs unit level?
If its typically 20 or less, then there are some shorter term, more manual solutions we can try while waiting for changes in the BrAPI spec.

Manual work around: have the user import all the variables with the study, then go to Field Book >> Traits and manually deselect the traits not relevant to that observation level.

Short term FB update: add better search filters to the Trait importer in FB. When the user goes to FB >> Traits >> menu dots >> Import, provide some filters just like the Field importer to list relevant variables. Then a user could select the study and manually select which vairiables they are interested in importing, instead of FB importing everything automatically.
This is similar to the manual work around, except it is additive instead of reductive. Depends on what the typical use case is, are there usually more variable to add manually, or remove manually. I don't know which would be generally more helpful, perhaps both options

In the long term, we can discuss adding observation level to the variables in BrAPI. If we want to filter variables by obs level, then the variable data model should also contain that metadata, like a targetObservationLevel or something like that. The potential down side to this would be another division in the definition of a variable. For example, one study is measuring Plant Height at the plot level, and another study is measuring Plant Height at the plant level, there would now need to be two different variables, both defining Plant Height. this could make things difficult if you try to compare the data between the two studies later. Or maybe perhaps these two different methods for measuring Plant Height should not be compared directly, and having separate variables is ok. Again I don't know, just thinking about possibilities.

@abatac Please open an issue in the BrAPI Github for this so it is not forgotten. We can discuss with the wider community and see if others would find this functionality useful in the standard.

I assume this is a duplicate to discussion #685. If I'm wrong, let me know and I can make a separate comment over there.

[abatac]

In BMS, a data set typically contains approximately 10 trait variables, but there is no strict limit on the number of trait variables that a data set can contain. Additionally, a study can consist of multiple data sets, each with different trait variables, resulting in the possibility of returning more than 20 variables from the GET /variables?studyDbId={} query.

A short-term solution I can think of could involve leveraging the additionalInfo property in ObservationVariable. By introducing an observationLevelNames field in additionalInfo (e.g., {"observationLevelNames": ["plot", "plant"]}), we can identify the level(s) at which a variable is utilized within a study. This information can be used to facilitate variable filtering based on the selected Observation Level during Field Import by FB. I'm not sure if this is a good idea because this approach would only be applicable using BMS implementation of BrAPI.

I created this issue in BrAPI Github 575.

Yup this is a duplicate of #685. I've already closed it.

[BrapiCoordinatorSelby]

Instead of additionalInfo I would suggest that you add a observationLevelName parameter to the GET /variables request in FB. It would be a minimal change in the FB code and all other servers should ignore the extra parameter. With that you could filter the variables response on the BMS side.
If you submitted a pull request with this update to Field Book, and asked @trife very nicely, he might allow this BMS specific update. Officially, this is beyond the BrAPI standard, so it is up to the BMS and FB teams to negotiate the details of this code change. I'm not going to advocate one way or the other.

EDIT
After writing all that out I realized the request parameters are actually buried in the brapi-java-client package, managed by the BI group, and pulled into FB as an external library. So, additionalInfo and filtering on the FB side might be the easiest way to go after all

[abatac]

Hi @trife, is it possible to include this short-term solution in field-book?

[trife]

@abatac I'm fine supporting that as a custom BMS feature. I don't have the developer throughput to support this work right now, so if your team can make those changes and heavily comment where they're BMS-specific so that they don't get undone, we can get them integrated into v5.5.