feat(store): improve SQL migrations by juan518munoz · Pull Request #2358 · 0xMiden/rust-sdk

juan518munoz · 2026-08-04T18:50:33Z

Froze store.sql as migrations/0001_init.sql. Schema changes are appended as new files, no longer patched into an existing one.
Pinned each migration's fingerprint in PINNED_SCHEMA_HASHES and made it the runtime check. The previous check is now test only.
Rewrote normalize_sql as a quote-aware scanner. Inline column comments are now free to edit, and three cases where different schemas shared a digest no longer do.
apply_migrations handles every SchemaVersion arm. A store from a newer client reports SchemaTooNew rather than an opaque DatabaseTooFarAhead.
A failed migration no longer prints the whole schema to the terminal.
Added scripts/check-migrations.sh and a CI job that rejects any edit to a released migration.

Notes

#2304 already moved the hash from the script's bytes to the materialized schema.

igamigo

Looks good so far. One thing we should do before closing the issue is this point from the issue:

Back up the database before migration.

We should look into backing up the store, migrating it, and once we know it has passed (by checking the hashes), delete the backup. If anything errors in the middle, we roll back the upgrade and then report the error. We could decide to do this in a separate PR.

Another thing that would be good to validate is whether the first migration is applied on a fresh file and that there is no way for a sqlite database to be applied the 0001 migration

igamigo · 2026-08-05T22:21:52Z

+/// Renders a migration failure without reproducing the migration script.
+pub fn describe_migration_error(err: &MigrationError) -> String {
+    match err {
+        MigrationError::RusqliteError { err, .. } => describe_sqlite_error(err),
+        MigrationError::ForeignKeyCheck(violations) => {
+            format!("{} foreign key violation(s) after applying the migration", violations.len())
+        },
+        other => other.to_string(),
+    }
+}


nit: Let's just inline this function where it's called (it's only one place AFAICT)

Done in 0e10af0

igamigo · 2026-08-05T22:39:35Z

+/// Rewrites the SQL text stored for a schema object into a form that ignores differences `SQLite`
+/// itself ignores, so cosmetic edits do not change the fingerprint.
 fn normalize_sql(sql: &str) -> String {


Let's revert these changes. I don't think it's code worth maintaining as it becomes instantly quite more complicated. A solution could be to introduce a SQL linter so the normalization is even more trivial.

Reverted

0e10af0

igamigo · 2026-08-05T23:49:55Z

One thing I think we can do is verify that the hash and check that it's indeed as we expect

We now follow node's procedure of checking migrations. At startup, the MigrationBuilder derives hashes by replaying the migrations, and checks them.

0e10af0

juan518munoz · 2026-08-06T20:50:35Z

We should look into backing up the store, migrating it, and once we know it has passed (by checking the hashes), delete the backup. If anything errors in the middle, we roll back the upgrade and then report the error. We could decide to do this in a separate PR.

Another thing that would be good to validate is whether the first migration is applied on a fresh file and that there is no way for a sqlite database to be applied the 0001 migration

This is now handled as of 0e10af0

gabrielbosio · 2026-08-06T22:42:09Z

+    exit 0
+fi
+
+>&2 echo "The following released migrations were modified, renamed or deleted:"


If the diff is compared against the base branch then a migration the script is comparing to might not be released yet:

Suggested change

>&2 echo "The following released migrations were modified, renamed or deleted:"

>&2 echo "The following merged migrations were modified, renamed or deleted:"

Added suggestion in 4f4dd5d

gabrielbosio · 2026-08-06T23:19:27Z

If I am correct, merging this PR would cause any future change in SQL store database to force the addition of a migration script. Do we want this once we merge it to next, should we wait after the 0.16 release or should we disable the migration freeze for now? If we go with the latter approach, maybe it would not be even necessary to disable the entire migration freeze logic but to only disable the migration check job for now.

gabrielbosio · 2026-08-07T14:35:46Z

If I am correct, merging this PR would cause any future change in SQL store database to force the addition of a migration script. Do we want this once we merge it to next, should we wait after the 0.16 release or should we disable the migration freeze for now? If we go with the latter approach, maybe it would not be even necessary to disable the entire migration freeze logic but to only disable the migration check job for now.

Let's leave this as it is right now and skip the migration check job in another PR with a new label (analogous to what no changelog is being used for). We can handle changes in the SQL store schema by changing the migration hash too for now.

juan518munoz · 2026-08-07T19:39:48Z

If I am correct, merging this PR would cause any future change in SQL store database to force the addition of a migration script. Do we want this once we merge it to next, should we wait after the 0.16 release or should we disable the migration freeze for now? If we go with the latter approach, maybe it would not be even necessary to disable the entire migration freeze logic but to only disable the migration check job for now.

Let's leave this as it is right now and skip the migration check job in another PR with a new label (analogous to what no changelog is being used for). We can handle changes in the SQL store schema by changing the migration hash too for now.

#2368

igamigo · 2026-08-10T18:23:46Z

Overall the PR looks fine, but it relies on a lot of loose functionality. I think the PR would benefit from having some of that logic wrapped in structs (e.g., SqliteMigration, SqliteBackup) that makes this easier to reason about

juan518munoz · 2026-08-11T15:59:07Z

@igamigo abstracted the new functionality of this PR into specific structs

SqliteMigration: owns the migration scripts together with the fingerprint each version builds. It checks, applies and verifies migrations.

SchemaHash: owns the schema fingerprint, how it is derived, manages storing SQL in a normalised form before hashing, and how it renders.

SqliteBackup: owns the store's path, its copy's path, and the copy's lifecycle. It is only produced by taking a copy and only consumed by restoring or discarding one, so restoring a store that was never copied is no longer expressible.

5f0fabb

igamigo

LGTM! I left one comment which I believe will be important, but it doesn't have to be done in this PR necessarily

igamigo · 2026-08-20T21:27:20Z

+    fn migrations_from(scripts: &[&'static str]) -> Migrations<'static> {
+        Migrations::new(scripts.iter().map(|&script| M::up(script).foreign_key_check()).collect())
+    }


This works, but we cannot easily do Rust hooks with this. We need to generalize this a bit more.
Basically I was expecting to have a single Migration (or SqliteMigration as this PR named it) that would have an associated .sql file and some optional Rust code (that takes a transaction and returns a Result<> or maybe just a bool). Then we could call M::up_with_hook and call the migration within the SQL transaction that would also run some Rust code. I think this would give us the most flexibility and is something that we will need.

igamigo

LGTM! Thanks

juan518munoz added 2 commits August 4, 2026 10:10

chore: improve sql migrations

8c5b813

Merge branch 'next' into jmunoz-sql-migrations

e563024

igamigo changed the title ~~chore: improve sql migrations~~ feat(store): improve SQL migrations Aug 5, 2026

igamigo marked this pull request as ready for review August 5, 2026 22:18

igamigo reviewed Aug 6, 2026

View reviewed changes

igamigo requested a review from gabrielbosio August 6, 2026 03:28

chore: address comments

0e10af0

Merge branch 'next' into jmunoz-sql-migrations

170dec7

gabrielbosio reviewed Aug 6, 2026

View reviewed changes

Comment thread crates/sqlite-store/src/db_management/backup.rs Outdated

juan518munoz added 2 commits August 7, 2026 16:20

chore: address PR comments

4f4dd5d

Merge branch 'next' into jmunoz-sql-migrations

4897f1d

juan518munoz mentioned this pull request Aug 7, 2026

CI: add label to omit SQL migration check #2368

Open

refactor: abstract migrations into a struct

5f0fabb

Merge branch 'next' into jmunoz-sql-migrations

1a883d0

igamigo approved these changes Aug 20, 2026

View reviewed changes

juan518munoz added 2 commits August 21, 2026 13:29

chore: address comments

3612523

Merge branch 'next' into jmunoz-sql-migrations

85902c7

igamigo approved these changes Aug 21, 2026

View reviewed changes

igamigo merged commit c9ccc8b into next Aug 21, 2026
23 checks passed

igamigo deleted the jmunoz-sql-migrations branch August 21, 2026 19:48

juan518munoz mentioned this pull request Aug 21, 2026

test: add rust-hook migration test for tags #2365

Closed

	>&2 echo "The following released migrations were modified, renamed or deleted:"
	>&2 echo "The following merged migrations were modified, renamed or deleted:"

Uh oh!

Conversation

juan518munoz commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Uh oh!

igamigo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juan518munoz commented Aug 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juan518munoz Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gabrielbosio commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gabrielbosio commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

juan518munoz commented Aug 7, 2026

Uh oh!

igamigo commented Aug 10, 2026

Uh oh!

juan518munoz commented Aug 11, 2026

Uh oh!

igamigo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

igamigo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

juan518munoz commented Aug 4, 2026 •

edited

Loading

igamigo left a comment •

edited

Loading

juan518munoz Aug 7, 2026 •

edited

Loading

gabrielbosio commented Aug 6, 2026 •

edited

Loading

gabrielbosio commented Aug 7, 2026 •

edited

Loading