Agent runtime data volumes: declared data paths with snapshot/restore and portable export

[vybe]

Summary

Agents accumulate runtime data — SQLite databases, large incremental files, working datasets — that cannot live in the git-synced template repo (it bloats fast and doesn't belong in version control), yet must survive container recreate, template re-pulls, and subscription auto-switch, and must move with the agent when it migrates to another Trinity instance.

Proposed model: agent = code (git) + credentials (.credentials.enc) + data (declared volume). Runtime data becomes a third declared artifact class, parallel to the existing credential pattern.

Context

Today the only options are committing data to the template repo (bloat, churn, merge conflicts) or leaving it in the container filesystem (lost on recreate — exactly the events that already destroy runtime state today: template re-pull, subscription auto-switch, image upgrade). Neither is acceptable for database-backed or data-heavy agents.

The platform already has the adjacent primitives:

• .trinity/persistent-state.yaml (S4, Persistent state allowlist (S4) #383) — the "paths that survive reset" allowlist
• Per-agent named volumes (agent-{name}-public, FILES-001; shared-folder volumes)
• Backend-side Docker SDK file extraction (get_archive) that never mounts the agent workspace (src/backend/services/agent_shared_files_service.py)
• The ~/.trinity/pre-check hook pattern for agent-side lifecycle hooks
• Credential export/import as the portability surface

Design Sketch

1. Declare — data_paths: section in template.yaml (or an extension of the S4 persistent-state allowlist) listing runtime-data paths, e.g. ~/data/, ~/db/agent.sqlite. Gitignored by convention.
2. Mount — Trinity mounts declared paths from a named per-agent volume (agent-{name}-data), same pattern as agent-{name}-public. Survives recreate/re-pull/auto-switch.
3. Durability — a backend snapshot job tars the volume on a schedule into the existing trinity-data bind mount (e.g. /data/agent-data-snapshots/) with retention. For SQLite, snapshot via a quiesce hook (~/.trinity/pre-snapshot, shebang-selected like pre-check) or sqlite3 .backup to avoid mid-write corruption.
4. Portability — "move agent to another Trinity" = template URL + .credentials.enc + data snapshot tar. Extends the existing export/import surface.

Acceptance Criteria

• template.yaml (or persistent-state allowlist) supports declaring runtime data paths
• Declared paths are backed by a named per-agent volume that survives container recreate, template re-pull, and subscription auto-switch
• Backend snapshot job with configurable schedule + retention; snapshots stored under the existing trinity-data bind mount
• SQLite-safe snapshotting (quiesce hook or .backup-based copy), documented for template authors
• Export/import flow covers the data volume alongside credentials (agent fully reconstructable on another instance without git carrying data)
• Agent delete / rename cascades handle the data volume and snapshots (manual cascade per platform convention)
• No new agent-side privileges: snapshot path is backend-side Docker SDK, agents stay off the platform network

Technical Notes

• Replica-group interaction (Replica groups: horizontal scaling for single agents #927): a declared mutable data volume is precisely what makes an agent NOT replica-safe. data_paths should feed the template.yaml replica-safety declaration in docs/planning/TARGET_ARCHITECTURE.md — one declaration serving both features.
• Aligns with target-architecture principle security: Fix token logging and add HTML reports to gitignore #7 (data exchange over conversation) and the sovereign-infrastructure principle — explicitly avoid per-agent S3/MinIO (new service), git-LFS (still bloats, wrong tool), or storing blobs in the platform DB (agents must not reach it; blobs don't belong there).
• Volume ownership must be fixed to UID 1000 (developer), same as shared-folder volumes (Invariant Client/Viewer User Role (AUTH-002) #17, non-root containers).
• Mount spec should live in src/backend/services/agent_service/capabilities.py alongside the existing tmpfs/TMPDIR spec so create/recreate/system-agent can't drift (fix: agent /tmp is a 100MB noexec tmpfs — starves pip/npm/build scratch; default TMPDIR to home volume #1098 precedent).

Related Work & Crosslinks

Building blocks (precedents this design reuses)

• Persistent state allowlist (S4) #383 (closed) — S4 persistent-state allowlist (.trinity/persistent-state.yaml): the existing "paths that survive reset" primitive; data_paths either extends it or must stay consistent with it. Flow doc: docs/memory/feature-flows/persistent-state-allowlist.md
• Reset-preserve-state operation (S3) #384 (closed) — reset-to-main-preserve-state (S3): the recovery reset that snapshots/overlays allowlisted paths — must treat declared data paths correctly (they live on the volume, so reset should not touch them at all)
• feat: Platform file storage service with secure one-time download links (FILES-001) #295 (closed) — FILES-001: per-agent named volume (agent-{name}-public) + backend-side get_archive extraction without mounting the agent workspace — the exact mount + extraction pattern this feature generalizes
• fix: agent /tmp is a 100MB noexec tmpfs — starves pip/npm/build scratch; default TMPDIR to home volume #1098 — mount/TMPDIR spec centralized in src/backend/services/agent_service/capabilities.py so create/recreate/system-agent can't drift — the data-volume mount spec must live there too

Interacting / gated work

• Replica groups: horizontal scaling for single agents #927 — Replica groups: a declared mutable data volume is the canonical replica-UNSAFE marker. data_paths should feed the template.yaml replica-safety declaration (docs/planning/TARGET_ARCHITECTURE.md §Replica Groups) — one declaration serving both features
• feat: transactional agent executions — discard workspace changes unless validated as success (research-gated) #1095 — Transactional executions (discard workspace unless validated): declared data paths must be explicitly in or out of the discard scope — a DB the agent writes mid-turn cannot be silently rolled back with the workspace
• feat(mcp): introspect agent pipelines via standardized ~/.trinity/pipelines/ + ~/.trinity/pipeline-state/ #919 — Agent pipelines (~/.trinity/pipelines/, ~/.trinity/pipeline-state/): pipeline state is exactly the kind of runtime data that should ride the data volume so it survives recreate
• feat: integrate MemPalace persistent memory for agents #276 — MemPalace persistent memory: a natural consumer of durable agent-local storage
• Spike: evaluate Mesa filesystem for agent artifact storage #583 — Mesa filesystem spike: overlapping problem space (versioned non-code artifact storage). This issue is the boring-volume baseline; the spike's "replaces/complements/overlaps" assessment should evaluate against it

Bug classes to design against

• P1: Duplicate mount point error on agent restart with custom_mounts #689 — duplicate mount point on container recreate when mounts are both inherited from old container metadata and re-added from config. Mount assembly for agent-{name}-data must be idempotent per destination path on the recreate path

Documentation that must follow (tiered-docs rule: new capability)

• docs/memory/requirements.md — new capability entry before implementation
• docs/memory/architecture.md + a docs/memory/feature-flows/ slice
• docs/agent-validation-spec.md (implemented by feat: Agent deployment compatibility validation — server-side checks with auto-fix offers #668) — new checks once data_paths exists, e.g.:
  • STATIC/HARD: every data_paths entry is excluded in .gitignore (the whole point is keeping it out of git)
  • STATIC/SOFT: data_paths entries don't overlap Trinity-managed paths (.trinity/, .claude/, .env) or git.commit_paths
  • STATIC/SOFT: data_paths present ⇒ replica-safety declaration consistent (not marked replica-safe)
  • STATIC/INFO: ~/.trinity/pre-snapshot (if present) is executable with a valid shebang — analog of A-004 for pre-check
• docs/TRINITY_COMPATIBLE_AGENT_GUIDE.md — template-author documentation for data_paths + snapshot hooks

[AndriiPasternak31]

/claim

[github-actions]

Claimed by @AndriiPasternak31. Label status-in-progress added.

Next steps:

1. Create branch: feature/1169-<short-slug>
2. Implement the feature
3. Open a PR with Fixes #1169