probe: how to handle older measurements inside the in-app DB

[hellais]

It has been reported in #1950 & #1927 that it's a problem for users that run OONI Probe unattended for a long time that the database size just keeps growing.

Currently whenever a measurement is uploaded we delete the raw JSON on disk and store in the local database the metadata about the measurement so that we can display it in the result listing. If the measurement is not uploaded, in addition to the metadata in the DB, we also retain the raw JSON.

What is being proposed here is that, in addition to deleting the raw JSON once it’s uploaded (which is already done), we also delete the metadata from the database after some amount of time. If we do this, those older measurements will effectively disappear from the users app and will not be retrievable in any other way anymore.

We should have an option in the settings that will automatically delete measurements from the DB that are older than some predefined date interval.

This is also related to: #1031

[hellais]

We are probably going to do something similar now that autorun is shipped in multiplatform, though since we fetch them remotely probably we don't have to delete everything.

Adding some notes about this from slack:

For doing something automatic, that's why I suggested clearing automatically results older than 1 year. We can let let users know in the onboarding on in the settings. And if we get complains, we can turn it into a Setting in the future.

Maybe this is something worth adding to some user survey we could do in the future. Given that uploaded data is in explorer having older than even 1 month data on your device is probably not so helpful.

With the probe id work we could even have pages in explorer where users could lookup all their measurements there directly and we don’t even have to implement all these UI elements in the app

Maybe one way to look at it could be to say we always delete by default unless:

• measurement hasn’t been uploaded
• the user explicitly marked a measurement as “keep”
  For all other cases we provide an in app view to explorer to see all their measurements

It’s anyways going to be more effective to look at a bunch of measurements from your computer with explorer where you can make charts using mat and have more screen real-estate

[hellais]

Also useful metrics for multiplatform are:

Each new full OONI run is increasing the size of the database by around 12 KB. It gets slightly smaller as URLs get reused, but it's a negligible reduction.
Assuming auto-run is enabled and every run is successful, that's 288 KB per day, 8.6 MB per month, 105 MB per year.

• Does this size increase warrant any additional action?
• Users right now can only clear everything, there's no selective deletion.
  Should we automatically delete data older than 1 year? And/Or add that as a setting?
  I don't think it's something we need to address right away. But since we're now storing auto-runs, it's something we should keep in the back of our minds.

[hellais]

See also: https://docs.google.com/document/d/1bsDAdA80i4z3Prs-P-88IoNYqb7LIL9TkYlgLQgDG0M/edit?tab=t.0#heading=h.y0bgdagcsvoc