Migrations

[janl]

How should we handle the case when Hoodie-internal data structures change?

@gr2m should have discussion notes for this.

[gr2m]

not directly related, just wanted to put it somewhere:
http://couchmigrations.org/

[gr2m]

I did some thoughts on migrations today. It's not 100% thought through, but maybe a good base for discussion.

Migrating data in Hoodie

Data needs to be migrated at two places

1. Locally in the client
2. In the respective database(s) on the server

Migrating data locally

Hoodie by design can be used without sync. Without an account, hoodie.store.add() will save data locally in the browser and it won't be synced anywhere until the user signs up for an account.

Because of that, it wouldn't be sufficient to migrate data in the databases on server only, as there are no databases for what we call "anonymous users".

Migrating data locally would mean to pause any sync and then silently migrate all data locally.

If the app is offline-first (e.g. with an appcache manifest), we need to prevent an inconsistency between the data and the app assets.

Migrating data on the server

When migrating data on the server, hoodie server should

• stop access from outside for the time the migrations are running
• pause all continuous replications
• stop all workers
• run the migrations on all matching databases (most often the user/... databases)
• start workers
• resume continuous replications
• enable access from outside

In some cases, multiple migrations need to be run at the same time. Namely I had the case that both the user/... databases as well as the _users database needed to be migrated simultaneously.

General notes on migrations

From my experience (I did 7 such migrations in a production app), in the most cases it's sufficient to migrate each object atomically. But in some cases, there is a dependency between the docs, and sometimes also on the _users/... doc, too. We should take that into account when creating a generic migration solution for Hoodie.

Idea(s) for hoodie-plugin-migrations

This is for the simplest (and most relevant) use case, when a migration means that all user/... databases have been migrated.

1. Hoodie will store the current time as UTC in localStorage (say as _last_migration), unless it is already set. The setting persists sign ins / sign outs.
2. All existing migrations can be loaded via <script src="/_api/plugins/migrations/"><script>. The loaded script would internally compare the migrations with _last_migration and would run the ones that are newer, then update _last_migration. The script could also be loaded with a ?since=20140608200000 parameter, that would leave out all migrations older than that. As JS scripts get loaded synchronously, it would block the page and assure that all migrations are run before the actual app gets initialized.
3. For offline-first apps, both hoodie.js and /_api/plugins/migrations should be part of the appcache manifest. If the app is on a different domain, it should be part of the app's build process to download and concatenate them with the app's other JS files. We might provide helpers like grunt tasks for that in the future
4. For accounts that are syncing data, Hoodie listens to new $migration objects being pushed from remote. The $migration objects would have a timestamp property, and they would be added before migrating existing objects to make sure that it gets pushed first through the /_changes feeds. If a $migration object is encountered, the sync will be stopped immediately and a flag gets set that will run the migrations on next page load.
5. For offline-first apps, encountering a $migration object should also start an appcache update to assure that the latest app assets will be available on next page reload.

Note that this approach requires the exclusive usage of the /_changes feed. Sending manual GET / PUT/ DELETE requests would bypass the migrations and could result in an inconsistent state. An alternative to the $migration objects could be a HTTP header that would be set on every response coming from the Hoodie server, like X-LastMigration: 20140608200000. The header would be compared with the local _last_migration and if it's older than the HTTP response header, all requests to remote would be cancelled, appcache update initiated (if applicable), and the migration would be run on next page reload.

Every migration would set _last_migration to its own timestamp

Ideas for hoodie-plugin-migrations admin UI

• Ace Editor to write the migration script(s). Every script would have a pattern that matches all databases the migration is supposed to be run in. A migration can have multiple scripts.
• Preview: The migration can be run virtually on user-specified databases (or random database of the ones matching the pattern) that wouldn't persist the changes, but provide a nice diff view for before/after
• A custom data editor for every script, that would work as test cases. Left side a list of current objects, right side the expected outcome.

Thinking further

I could well imagine that Hoodie will require migrations itself at some point, for major or minor version updates. I don't think it needs to be part of Hoodie core, but for apps that are using Hoodie today, we should provide a simple way to have existing data being migrated when required by a Hoodie version update.