Adding an Adaptive Caching Layer?

[nickrsan]

Hi there,

Thanks for the work on this addition to the STAC ecosystem. I'm wondering about the possibility of adding an adaptive caching mechanism either within stac-asset or possibly as another stac-util repository - the goal being that I can have a persistent cache of downloaded assets where the least used items get evicted to make space for new downloads, ideally setting the hrefs on the Item classes back to the original paths so they can continue to be used as before (and downloaded again, if accessed).

I can see arguments for both enhancements within stac-asset and also a separate library that gets initialized with a collection from pystac-client and with downloading functionality from stac-asset, and then has a transparent layer that manages data access so that when you go to access the asset file locally, it downloads it if needed and then returns the local filepath, evicting anything needed from your local cache in the process.

Do you know if any discussions along these lines have been had already by STAC contributors? We have development time to devote to this and are happy to do the bulk of the work, but I'd ideally like to integrate it into the existing STAC ecosystem in the best way possible, so wanted to start with a discussion item rather than a formal feature request/issue, since I'm not yet sure if this is the repository where it should be integrated. Thanks!

[gadomski]

Cool idea, thanks for asking! At the most basic level, stac-asset does "caching" by skipping already-existent local paths:

stac-asset/src/stac_asset/_functions.py

Line 50 in ec56b86

if not os.path.exists(self.path) or self.config.overwrite:

It wouldn't be hard to implement a "no newer than" feature to compliment the "overwrite" behavior -- i.e. "Is this asset old? If so, re-download. If not, don't". That would be cool.

A system-level asset cache is an interesting idea, and could be within scope. Uniquely identifying the assets shouldn't be too hard (server url + collection id + item id + asset key should do the trick). "Clearing out" the cache when it gets full feels annoying, but possible. So yeah, I'd be interested in looking at a POC, whether that's here or in a new demonstration repo.

Tl;dr: cool idea, PRs welcome ✌🏼.

[nickrsan]

Great! Thank you - yes, I was looking at some of the code yesterday, but only just enough to see if this seemed like the repository to begin the discussion on - not enough to feel comfortable yet understanding just how much caching it does.

And thanks for openness - I'll talk with the person I was hoping would spend time on this and see if we could plan out a bit more to bring forward to you all before we start writing code.

[matthewhanson]

I like this idea too @nickrsan. Some ?s:

• How do you determine the least used assets? Is it calls to download?
• How could this persist across sessions?
• Do you see this being used by multiple users sharing a single shared drive, or single user?
• Would you set a hard storage cap that can't be exceeded?

I'd think that the cache could be rebuilt on initialization from an existing local path and an ItemCollection, but then how is usage determined?

[nickrsan]

With the caveat that I'm open to adjustments on all of this, here are some thoughts:

"How do you determine the least used assets? Is it calls to download?"

For this one, I'd been thinking of introducing some new functionality - something like get_local_path() or whatever - where it automatically calls the code to download the asset if it's not already on disk and then regardless, adds a usage record to the data store (more in a moment) with the asset ID and datetime, and then returns the local path on disk. Then least used could be some usage count over a rolling window with occasional prunes of older usage records to keep the table clean. I'd be aiming for something like ZFS' ARC mechanism so that it minimizes redownloading the same dataset whenever possible (rather than just evicting the one used longest ago from the cache, regardless of if it has lots of use but some more recent ones were only needed once - e.g. if I do a lot of processing with a suite of Landsat scenes, but needed some other dataset for some one-off exploration, I don't want to evict landsat even if they haven't been used in a few weeks).

How could this persist across sessions?

I was thinking of a simple SQLite DB (or optionally DuckDB) stored in the same folder as the cache.

Do you see this being used by multiple users sharing a single shared drive, or single user?

I'm open on this front. My own use case is running on a server where multi-user could be nice, but in practice it's single user. I'd want to scope out how much more complexity multi-user management would involve and haven't fully thought that through yet.

Would you set a hard storage cap that can't be exceeded?

I was debating on this too. The simplest is just making sure there's space on whatever drive/storage pool, but part of my goal is to leave space for other uses of the same storage pool by setting a cap. I'm thinking maybe something configurable - for single user, the cap could be specified per-run, but for multi-user, we'd probably want it persisted in the database. I think this would add some need to track asset size so we're not constantly stating files, but it could be a nice feature to be able to specify that it can keep up to X TB of images and then not need to think about it more.