Plugin API contract

[betolink]

We have been talking about a plugin API that let earthaccess use services like NASA Harmony for a long time see #328; Today at the bi-weekly meeting engineers from the transformation train joined to talk about how this may look like and we agreed on start working on an API contract that defines what a plugin will need from earthaccess and what earthaccess should expect from such plugins.

A good way of generalizing would be to search in earthaccess and then provide a list of the results to a plugin, the plugins will use this list of results to process them returning a job-id that can be used by earthaccess to query the state of the request and when done, do the same thing earthaccess does, download or open those results to stream the data in xarray.

There are some details that need to be decided, when we search we get a list of UMM records from CMR, each one of these result items is a big json dictionary, are plugins going to use any of this metadata or can we pass just the concept-id or the link to the data files listed under "GET DATA". Currently, Harmony supports a list of concept-id for granules, so we could start there.

Usually these services are async, so earthaccess will have to implement a job class to track status and completion, this is actually the case for Harmony. I think we are in good shape to implement the folloing:

import earthaccess as ea

ea.login()

results = ea.search_data(**params)
job_id = ea.services.harmony.request(results, **kwargs) # internally the API will use only the concept-id and pass the kwargs
status= ea.services.harmony.status(job_id)

if status is "COMPLETED": # or an enum of it
    files  = ea.services.harmony.results(job_id)

ds = xr.open_mfdataset(files)

From this pseudocode the initial proposal is that a service plugin to earthaccess will at least implement:

• request(results, kwargs) -> job_id: earthaccess doesn't know any particular thing about the internals just passes a list of results and kwargs and gets an id to later identify progress and how to retrieve the results.
• status(job_id) -> ENUM: this will tell us if the request is being processed, is ready, it failed, is done etc.
• results(job_id)-> list[URI]: this should be a list of URLs that can be opened with earthaccess.open or earthaccess.download once the service is done processing the granules. if HTTPS, it should be publicly accessible (with or without NASA EDL bearer token) and if it's S3, the service should provide an endpoint to get access keys... perhaps an internal interface.

Things to think about

• How will earthaccess register the plugins? same as xarray with accessors when they are imported?
• Should we use earthaccess.services.<PLUGIN> as the canonical namespace for async processing plugins?
• search results (input of the plugins): should the plugins get the whole UMM granule metadata and know how to get the things they need? will results be URIs? Harmony accepts concept-ids, so maybe the API is one of the 2?

I'm probably forgetting many nuances here but in th meantime this would be a good place to keep working on it.

cc @owenlittlejohns @danielfromearth @JessicaS11 @jhkennedy @asteiker @mfisher87 @chuckwondo @itcarroll @flamingbear

[JessicaS11]

Thanks for kicking this off, @betolink! My highest level comment is that this psuedo implementation as written is very focused on async processing plugins:

Usually these services are async, so earthaccess will have to implement a job class to track status and completion

I would be keen to think about how to make it more general so that it works for async processing tools as well as plugins that enable reading in of datasets that are not currently supported by the current ea.open functionality and/or do not rely on the request-->status-->results series of steps. Why can't the plugins handle this job tracking?

To address your specific questions:

• How will earthaccess register the plugins? same as xarray with accessors when they are imported?

I would say "yes". One of the things I really like about xarray's approach is that all users need to do is install the appropriate library (here, harmony-py?) and then xarray can find/use the library (and also tell users that it's an option).

• Should we use earthaccess.services. as the canonical namespace for async processing plugins?

Why not just earthaccess.<PLUGIN> to keep the namespace shorter? To me, by virtue of the user needing to make sure the external library is installed they are aware that they've added some extra/specialized functionality.

• search results (input of the plugins): should the plugins get the whole UMM granule metadata and know how to get the things they need? will results be URIs? Harmony accepts concept-ids, so maybe the API is one of the 2?

I think this question is getting at another question I had, which is: what are "results" here? In the current version of earthaccess, this would be a list of granule IDs. In a coming soon version, this will be an object that includes that list and some other stuff. In my view, the job of the plugin is to take an earthaccess.search_data results object and use it to provide something beyond what earthaccess can do (in many cases, data transformation services). The role of earthaccess here is then to (1) make the results object light enough to pass around (in which case the question of whether that object includes the UMM granule metadata becomes a critical part of the results class object conversation); and (2) determine which of the earthaccess functions, if any, plugins MUST implement. If I haven't gotten too mixed up again, in the case of harmony this consideration of what is passed in has implications for repeated querying of CMR. If so, is this double-query operation expensive enough that we should design our plugin architecture around not repeating it?

Excited to continue working on this with everyone!

[owenlittlejohns]

@betolink - thanks for getting this started! Apologies in advance - this is me dumping a bunch of thoughts from reading the comments from you and @JessicaS11!

How will earthaccess register the plugins? same as xarray with accessors when they are imported?

I think this sounds like a good model, too. I like the approach generally, as well as the idea of consistency with something like xarray that folks are likely familiar with.

Should we use earthaccess.services. as the canonical namespace for async processing plugins?

I'm not sure if I have strong opinions here. I suppose the nice thing about having a singular namespace for them is that it makes the grouping of "these things are external services" explicit. On the other hand... it does maybe imply a little more consistency in arguments that would be passed to the underlying packages than there actually is. Sorry, I'm waffling back and forth there!

should the plugins get the whole UMM granule metadata and know how to get the things they need?

That seems like the most intuitive thing to do: (as a user) I search for stuff and get results, I pass those results to the next thing without having to think about the pieces I have to pull out from the results - particularly if there are multiple plugins that require different pieces of a UMM-G record. I think I'm agreeing with @JessicaS11 above:

In my view, the job of the plugin is to take an earthaccess.search_data results object and use it to provide something beyond what earthaccess can do

That sounds right to me, too.

will results be URIs?

Good question. Maybe it would be nice if there was a scaled back version of the DataGranule class that could represent things unrelated to CMR, but provide the non-CMR information for something else in the Earthdata ecosystem? I'm thinking of something that is consistent-ish with the output of earthaccess.search_data and can be easily used with earthaccess.open or earthaccess.download. That said, URIs (and lists of them) would also work in those functions, I think, so maybe that's getting too complicated. (Sorry, I'm rambling at this point!)

Harmony accepts concept-ids, so maybe the API is one of the 2?

Harmony can accept either a granule concept ID or a readable granule name (including possible wildcards) to identify granules. It can also accept either a collection short name or a collection concept ID to identify the collection being used. My vague preference - if we have the search results, and therefore know the concept ID, that feels the least ambiguous thing to send to Harmony (pulling it out within the plugin itself) - but that might not be consistent with other potential plugins. Do we have any likely candidates for a second or third plugin that might help decide?

If so, is this double-query operation expensive enough that we should design our plugin architecture around not repeating it?

I don't think this is particularly expensive, so we shouldn't design the architecture around this.

Different question (expanding the scope a bit): Harmony has a /capabilities endpoint. Would there be value in exposing that via something like earthaccess.services.harmony.capabilities(<collection>)/earthaccess.harmony.capabilities(<collection>)? Or would this start to present problems for consistency with other future plugins that may not have this feature?

(Sorry lots of thoughts. Mostly only half formed!)