Skip to content

perf: selective snapshot hydration - #6016

Open
fresioAS wants to merge 3 commits into
SQLMesh:mainfrom
fresioAS:selective-snapshot-hydration
Open

perf: selective snapshot hydration#6016
fresioAS wants to merge 3 commits into
SQLMesh:mainfrom
fresioAS:selective-snapshot-hydration

Conversation

@fresioAS

@fresioAS fresioAS commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Description

Avoids hydrating production snapshot payloads for models and standalone audits already defined in the local project.

Previously, Context.load() hydrated every snapshot in the production environment to determine whether each node belonged to the current project or another project. The environment's compact snapshot metadata already contains the snapshot names needed to identify locally defined nodes.

This change:

  • Marks locally defined production snapshot names as uncached using compact environment metadata.
  • Hydrates only snapshot names absent from the local project.
  • Continues hydrating missing names so deleted local nodes and cross-project nodes can be distinguished using the snapshot's project metadata.
  • Preserves cross-project state merging and existing schema reconstruction behavior.
  • Applies independently of project-index loading.

This primarily reduces Python-side snapshot deserialization and object hydration for projects with large production environments.

Test Plan

Live env (400 models)
plan1 40.17s <--> 33.16s 7.00s faster, 17.4%
plan2 41.41s <--> 33.36s 8.05s faster, 19.4%
plan3 38.41s <--> 32.48s 5.93s faster, 15.4%
average 40.00s <--> 33.00s 7.00s faster, 17.5%

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Signed-off-by: Andreas Fredhøi <andreas.fredhoi@fresio.no>
Signed-off-by: Andreas Fredhøi <andreas.fredhoi@fresio.no>
@fresioAS
fresioAS marked this pull request as ready for review September 11, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant