Store RFC: Improving Sync #697

matt-ramotar · 2025-02-23T23:55:09Z

Signed-off-by: Matt Ramotar <[email protected]>

codecov · 2025-02-23T23:57:09Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.43%. Comparing base (2aa3213) to head (8c5c75c).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #697   +/-   ##
=======================================
  Coverage   75.43%   75.43%           
=======================================
  Files          38       38           
  Lines         920      920           
  Branches      168      168           
=======================================
  Hits          694      694           
  Misses        142      142           
  Partials       84       84

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

blakelee

I took a look at this. I had a couple questions:

Is Meeseeks one of the StoreX extensions coming in the future? There were a lot of references and I inferred that it would be one of them.
Regarding partial failure handling: This is a new concept for me. Is the idea that some items may fail to update and we need to gracefully handle that case, whereas others may succeed so you can handle those as usual?
I imagine a one case could look like this: You have a fetcher that's a stream to the server, new items are constantly coming in, based on any updates that may or may not have happened, the conflict resolution strategy will take care of how to handle the updates.

Thanks a lot for writing this up. The timeline at the bottom is promising too. I think this piece of the puzzle will help mobile developers solve a problem that many of them have but don't have the resources to create for themselves.

What would you like from us the consumers of Store?

matt-ramotar · 2025-02-26T12:56:48Z

@blakelee Hey Blake, thanks for taking a look.

Yes, Meeseeks is a library I'm working on now. We need a KMP scheduling solution to help Store automatically retry unsynced operations in the background.
Yes, exactly. "Partial failure handling" becomes important in scenarios where multiple items are batched together in a single operation. For instance, you might request deleteMany for five items, but only three succeed while two fail. Store will need to provide a clear way for developers to detect and handle that situation - whether that means retrying just the failed items, rolling back changes, or something else defined by the developer.
That’s definitely a key scenario we’re targeting. When Fetcher returns new or updated items in real time (or near real time), Store’s conflict resolution policy will determine how to merge those updates with what is currently in the local cache or SOT. By providing hooks for custom conflict resolution, developers can decide how to handle (e.g., last write wins, field by field merge).
Right now, we're hoping the community can:
- Share real-world scenarios, especially complex/unique conflict resolution needs (understanding this will help us refine designs)
- Provide feedback on the RFC itself
- Contribute to any of the sub-projects (e.g., conflict resolution, Meeseeks, partial failure handling)

squarejesse · 2025-02-26T16:38:30Z

docs/rfc/improving-sync.md

+
+The overarching goal of this RFC is to enhance Store5’s synchronization features without introducing breaking changes. In particular:
+
+1. **Synchronization Resilience**: Automated retries and background scheduling using the `Meeseeks` library (a KMP task scheduling library influenced by [WorkManager](https://developer.android.com/reference/androidx/work/WorkManager) and [BGTaskScheduler](https://developer.apple.com/documentation/backgroundtasks/bgtaskscheduler)).


I couldn’t find the meeseeks library

Ahhh, it’s described below

Expanded on Meeseeks with links to the repo and docs 👍🏽

squarejesse · 2025-02-26T16:43:04Z

docs/rfc/improving-sync.md

+- **Reviewers**: _TBD_
+- **Started**: February 22, 2025
+
+## Introduction


I was expecting to see a discussion of CRDTs
https://crdt.tech/

Or perhaps Automerge which is a pretty rad implementation of CRDT ideas

Requiring Store5 to store CRDTs would limit the types that users can leverage. It seems like the best place for them would be in the user's own conflict resolution wiring.

tyvsmith · 2025-02-26T17:45:47Z

docs/rfc/improving-sync.md

+
+Store5 specifically introduced the `MutableStore` API to unify local and remote data synchronization. It provides mechanisms for create, update, and delete operations. However, we’ve identified critical gaps around resilience (automated retry, robust background sync) and deletion handling (both local and server-initiated removals). Along with these, we also need customizable conflict resolution strategies to better address real-world complexities.
+
+The overarching goal of this RFC is to enhance Store5’s synchronization features without introducing breaking changes. In particular:


This implies there would be some changes proposed if we were interested in breaking changes, that aren't called out here. Is there anything else in the sync category, or something we would have proposed differently here, if that was on the table and we'd want to discuss those with a major version bump at some point in the future?

tyvsmith · 2025-02-26T17:50:09Z

docs/rfc/improving-sync.md

+
+5. **Backward Compatibility**
+
+   Not willing to break existing usage. Not all developers need advanced sync.


What does advanced sync entail that we're intentionally leaving out?

Signed-off-by: Matt Ramotar <[email protected]>

rocketraman · 2025-04-17T23:21:56Z

docs/rfc/improving-sync.md

+
+The overarching goal of this RFC is to enhance Store5’s synchronization features without introducing breaking changes. In particular:
+
+1. **Synchronization Resilience**: Automated retries and background scheduling using the [Meeseeks](https://docs.meeseeks.mattramotar.dev/) library (a KMP task scheduling library influenced by [WorkManager](https://developer.android.com/reference/androidx/work/WorkManager) and [BGTaskScheduler](https://developer.apple.com/documentation/backgroundtasks/bgtaskscheduler)).


There should also be a way to trigger it manually. On iOS in aprticular BGTaskScheduler is not very configurable -- it is basically up to the OS when to call the task, and iOS provides very few if any guarantees on timing.

The system decides the best time to launch your background task, and provides your app up to 30 seconds of background runtime.

(reference)

Alternative strategies include things like waking the app with a background push and triggering sync from there, but this requires the manual API.

rocketraman · 2025-04-17T23:27:05Z

docs/rfc/improving-sync.md

+- **Reviewers**: _TBD_
+- **Started**: February 22, 2025
+
+## Introduction


Requiring Store5 to store CRDTs would limit the types that users can leverage. It seems like the best place for them would be in the user's own conflict resolution wiring.

rocketraman · 2025-04-17T23:33:41Z

docs/rfc/improving-sync.md

+
+We have the building blocks for robust sync ([MutableStore](https://store.mobilenativefoundation.org/docs/concepts/store5/mutable-store), [Bookkeeper](https://store.mobilenativefoundation.org/docs/concepts/store5/bookkeeper), [Updater](https://store.mobilenativefoundation.org/docs/concepts/store5/updater), [SourceOfTruth](https://store.mobilenativefoundation.org/docs/concepts/store5/source-of-truth)). But we need to tighten up deletion support, add background sync scheduling, and clarify conflict resolution flows.
+
+## Considerations


Should there be a consideration here for synchronization failure handling? Say an item is added to the SoT. When this item syncs to the server, there are two possibilities: 1) the item is invalid in some way, and rejected by the server (think 400 Bad Request), or 2) the item fails to sync due to some temporary problem e.g. network error, backend component down, etc. (think 502 Bad Gateway).

This also implies some kind of maximum retries counter -- sometimes the server does not correctly identify a bad request and instead returns a 500 Server Error. But the error is due to a problem in the request, so we can't keep retrying it forever.

These failed items should be stored in a way that the problem can optionally be surfaced to a user for correction, and updating the item in the local SoT.

I wrote this in #677 :

Lastly, some values may never sync to the server because it is malformed or violates server preconditions in some way. Part of the API should allow inspecting the failures for items that have failed to sync and invalidating them i.e. clearing or setting some relevant state in bookkeeper and moving them to some kind of "failed" status in the source of truth.

rocketraman · 2025-04-17T23:36:29Z

docs/rfc/improving-sync.md

+
+We have the building blocks for robust sync ([MutableStore](https://store.mobilenativefoundation.org/docs/concepts/store5/mutable-store), [Bookkeeper](https://store.mobilenativefoundation.org/docs/concepts/store5/bookkeeper), [Updater](https://store.mobilenativefoundation.org/docs/concepts/store5/updater), [SourceOfTruth](https://store.mobilenativefoundation.org/docs/concepts/store5/source-of-truth)). But we need to tighten up deletion support, add background sync scheduling, and clarify conflict resolution flows.
+
+## Considerations


Another consideration is the ability to retrieve the sync state of items. I wrote this in #677 :

Related to this, I would like a nice API to retrieve information about failed syncs. This could be used to display information to the user like "5 items have not yet synced to the server"

I'm not sure if Bookkeeper already provides this capability.

Store RFC: Improving Sync

5409607

Signed-off-by: Matt Ramotar <[email protected]>

matt-ramotar assigned yigit, psteiger, tyvsmith and digitalbuddha Feb 23, 2025

blakelee reviewed Feb 25, 2025

View reviewed changes

squarejesse reviewed Feb 26, 2025

View reviewed changes

tyvsmith reviewed Feb 26, 2025

View reviewed changes

Expand on Meeseeks

8c5c75c

Signed-off-by: Matt Ramotar <[email protected]>

matt-ramotar mentioned this pull request Mar 14, 2025

Version 1.0.0 release timeline matt-ramotar/meeseeks#7

Open

rocketraman reviewed Apr 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store RFC: Improving Sync #697

Store RFC: Improving Sync #697

matt-ramotar commented Feb 23, 2025

codecov bot commented Feb 23, 2025 •

edited

Loading

blakelee left a comment •

edited

Loading

matt-ramotar commented Feb 26, 2025

squarejesse Feb 26, 2025

squarejesse Feb 26, 2025

matt-ramotar Mar 10, 2025

squarejesse Feb 26, 2025

squarejesse Feb 26, 2025

rocketraman Apr 17, 2025

tyvsmith Feb 26, 2025

tyvsmith Feb 26, 2025

rocketraman Apr 17, 2025

rocketraman Apr 17, 2025

rocketraman Apr 17, 2025

rocketraman Apr 17, 2025


		The overarching goal of this RFC is to enhance Store5’s synchronization features without introducing breaking changes. In particular:

		1. Synchronization Resilience: Automated retries and background scheduling using the `Meeseeks` library (a KMP task scheduling library influenced by [WorkManager](https://developer.android.com/reference/androidx/work/WorkManager) and [BGTaskScheduler](https://developer.apple.com/documentation/backgroundtasks/bgtaskscheduler)).


		Store5 specifically introduced the `MutableStore` API to unify local and remote data synchronization. It provides mechanisms for create, update, and delete operations. However, we’ve identified critical gaps around resilience (automated retry, robust background sync) and deletion handling (both local and server-initiated removals). Along with these, we also need customizable conflict resolution strategies to better address real-world complexities.

		The overarching goal of this RFC is to enhance Store5’s synchronization features without introducing breaking changes. In particular:


		5. Backward Compatibility

		Not willing to break existing usage. Not all developers need advanced sync.


		We have the building blocks for robust sync ([MutableStore](https://store.mobilenativefoundation.org/docs/concepts/store5/mutable-store), [Bookkeeper](https://store.mobilenativefoundation.org/docs/concepts/store5/bookkeeper), [Updater](https://store.mobilenativefoundation.org/docs/concepts/store5/updater), [SourceOfTruth](https://store.mobilenativefoundation.org/docs/concepts/store5/source-of-truth)). But we need to tighten up deletion support, add background sync scheduling, and clarify conflict resolution flows.

		## Considerations

Store RFC: Improving Sync #697

Are you sure you want to change the base?

Store RFC: Improving Sync #697

Conversation

matt-ramotar commented Feb 23, 2025

codecov bot commented Feb 23, 2025 • edited Loading

Codecov Report

blakelee left a comment • edited Loading

Choose a reason for hiding this comment

matt-ramotar commented Feb 26, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 23, 2025 •

edited

Loading

blakelee left a comment •

edited

Loading