feat: Platform file storage service with secure one-time download links (FILES-001)

[vybe]

Summary

Add a platform-level file storage service that agents can upload files to and receive secure one-time download URLs. This enables agents to share files with users via Slack, email, or any channel without requiring direct container access.

Motivation

Agents generate files (reports, images, exports, code artifacts) but have no way to share them externally. Current workarounds (base64 in chat, Slack file upload) are fragile and channel-specific. A platform file storage service provides a universal mechanism: upload a file, get a URL, share it anywhere.

Design

Architecture

Agent → Upload file via MCP tool → Platform storage → Returns one-time URL
User clicks URL → File downloaded → Link expires/invalidated

Storage

• Platform-level storage at /data/files/ (bind mount, survives restarts)
• Files stored with UUID filenames (no path traversal risk)
• Metadata in SQLite: original filename, uploader agent, content type, size, expiration, download status

MCP Tool

upload_file(
  file_path: string,        # Path in agent container to upload
  filename: string,          # Display filename for download
  content_type?: string,     # MIME type (auto-detected if omitted)
  expires_in?: number,       # Seconds until link expires (default: 24h)
  one_time?: boolean         # Invalidate after first download (default: true)
) → { url: string, file_id: string, expires_at: string }

REST Endpoints

Method	Path	Auth	Description
POST	/api/files/upload	JWT/MCP	Upload file (multipart or from agent path)
GET	/api/files/{file_id}/{token}	None (token)	Download file (public, one-time)
GET	/api/files	JWT	List uploaded files (admin/owner)
DELETE	/api/files/{file_id}	JWT	Delete file

Download URL Format

https://{domain}/api/files/{file_id}/{download_token}

• download_token = HMAC-SHA256(file_id + expiration, secret)
• No authentication required — the token IS the auth
• One-time: marked as downloaded after first successful GET
• Expired files return 410 Gone

Database

CREATE TABLE platform_files (
    id TEXT PRIMARY KEY,
    agent_name TEXT NOT NULL,
    original_filename TEXT NOT NULL,
    stored_filename TEXT NOT NULL,      -- UUID on disk
    content_type TEXT,
    size_bytes INTEGER,
    download_token TEXT NOT NULL,
    one_time INTEGER DEFAULT 1,
    downloaded INTEGER DEFAULT 0,
    downloaded_at TEXT,
    expires_at TEXT,
    created_at TEXT NOT NULL,
    created_by TEXT                     -- user or agent
);

Cleanup

• Background task purges expired files (both DB record and disk file)
• Configurable max storage size per agent and platform-wide
• Files older than 7 days auto-expire regardless of download status

Security

• HMAC token prevents URL guessing
• One-time download prevents link sharing/reuse
• File size limit (default: 100MB per file)
• Content-type validation (block executables by default)
• Files stored outside web root, served via streaming endpoint
• No directory listing or traversal possible

Acceptance Criteria

• Agents can upload files via MCP tool and receive a download URL
• One-time download links work without authentication
• Links expire after configured duration
• Downloaded/expired files return appropriate HTTP status
• Cleanup task removes expired files from disk and DB
• File list visible in UI for admins/owners
• File size limits enforced
• MCP tool available for agent-to-platform file upload

[pavshulin]

Implemented as FILES-001 on branch feature/295-files-outbound-sharing.

Scope shipped (Steps 1–6)

• Per-agent opt-in toggle + Docker publish volume (agent-{name}-public mounted at /home/developer/public/)
• share_file MCP tool (agent-scoped) — 73 tools total in MCP server after this addition
• Internal endpoint POST /api/internal/agent-files/share (agent-server path, X-Internal-Secret auth)
• MCP-path endpoint POST /api/agents/{name}/shared-files (owner/admin + same-agent defense)
• Public download endpoint GET /api/files/{file_id}?sig={token} with token auth, constant-time compare, streaming, attachment disposition, HEAD handler, dedicated rate-limit bucket, audit event file_share_download
• List / revoke endpoints (owner + admin only)
• UI panel in Agent Detail → Sharing tab (toggle, quota, table, copy URL, revoke)
• Cleanup sweep in cleanup_service.py for expired + old-revoked shares

Divergences from original spec

• URL format: /api/files/{id}?sig={token} (not /api/files/{id}/{token}) — chosen to traverse the existing /api/* proxy rules on Vite dev + prod nginx without new proxy config. Query param is sig (not download_token) to avoid the credential sanitizer redacting the token in agent transcripts.
• Storage layout: per-agent publish volume + flat /data/agent-files/{uuid} on backend side (rather than single shared /data/files/). Cleaner tenancy; FK cascade on agent delete.
• One-time download links deferred — schema columns retained (one_time, consumed_at) for future re-enablement.

Docs / tests

• Requirements: §13.10 (docs/memory/requirements.md)
• Architecture: schema + endpoints + services + MCP tool tables updated (docs/memory/architecture.md)
• Feature flow: docs/memory/feature-flows/file-sharing-outbound.md
• Design doc: docs/drafts/amazing-file-outbound.md
• Production-readiness plan: docs/drafts/amazing-file-outbound-production-readiness.md
• Unit tests: 33 green (migration, DB mixin, mount-match helper)
• Live-verified end-to-end via real Slack round-trip 2026-04-24

Follow-ups

Filed as separate issues (see production-readiness plan §4):
G1 stream extraction, G2 directory sharding, G3 platform-wide quota, G4 audit file_id column, G5 endpoint pytest coverage, G6 URL sanitization in transcripts, G7 one-time + unfurl defense, G8 remove ?download_token= alias, G9 UI polish, G10 OTel spans, G11 PRAGMA foreign_keys (platform-wide), G12 unify request models.

Closing as implemented.

[pavshulin]

Implementation notes (additive — not replacing the original)

Collecting how we actually built this, known flaws/concerns, and deployment considerations so reviewers and future maintainers have the full picture alongside the close summary above.

How we built it

Delivered across 6 scoped steps rather than a single big bang, each verified end-to-end before the next one landed. This let us catch two real bugs (agent-delete cascade missing; URL format incompatible with proxy + sanitizer) during build rather than after merge.

Step	What landed	Commit reference
1	agent_shared_files table + migration + file_sharing_enabled column. FK declared ON DELETE/UPDATE CASCADE (aspirational — see §Concerns).	e47c688
2	Per-agent opt-in toggle (PUT /api/agents/{name}/file-sharing) + Docker volume agent-{name}-public mounted at /home/developer/public/. Mirrors the existing shared-folders expose pattern.	e47c688
3	POST /api/internal/agent-files/share (agent-server-path, X-Internal-Secret auth) + extraction via Docker SDK get_archive (backend never mounts agent workspace).	e47c688
4	GET /api/files/{id}?sig={token} — streaming response, constant-time token compare, revoked/expired gates, Content-Disposition: attachment, nosniff, audit event.	e47c688
5	share_file MCP tool (agent-scoped) + backend endpoint POST /api/agents/{name}/shared-files (owner-only + same-agent defense for agent-scoped keys).	e47c688
6	UI panel: toggle, quota, file table, Copy URL, Revoke. List + Delete endpoints.	e47c688
Phase 1 hardening	Docs pass, filename 255-char cap, shutil.disk_usage pre-check, cleanup sweep for expired/old-revoked rows, dedicated rate-limit bucket, HEAD handler, tightened list to owner+admin, system-wide prompt nudge for agents.	1e0567b → 07ca083 (9 commits)

Key architectural choices

• Storage split: agent writes to /home/developer/public/ (Docker volume, only mounted into the agent); backend extracts on demand into /data/agent-files/{uuid}. Backend filesystem never touches the agent workspace — path-traversal escapes can't reach .env / .credentials.enc etc.
• URL format /api/files/{id}?sig=... not /files/{id}/{token} as the original spec called for. Two reasons discovered during testing:
  1. Neither the Vite dev proxy nor the prod nginx config proxies /files/* → backend; they only proxy /api/*. Using /api/files/... means zero new proxy rules.
  2. The platform credential sanitizer redacts any .*TOKEN.*=value query pair from agent-response transcripts. We renamed the param from download_token → sig to escape the pattern.
• One-time mode deferred: schema columns (one_time, consumed_at) retained but the MCP tool + request/response shape doesn't expose them yet. Explicitly filed as a discussion item in FILES-001 follow-ups — needs refinement before pickup (12 items) #490 (G7) — re-enabling is ~40 lines because the columns are already there.
• Agent identity on MCP path: POST /api/agents/{name}/shared-files enforces current_user.agent_name == path agent_name for agent-scoped MCP keys so Agent A's key can't share files from Agent B's publish volume, even when both are owned by the same user.

Problems we hit during implementation

1. FK cascade declared but not enforced

Discovered during live test: after DELETE /api/agents/{name}, agent_shared_files rows for that agent remained in the DB, disk files stayed on /data/agent-files/, volume stayed on the host. Root cause: Trinity's db/connection.py doesn't set PRAGMA foreign_keys = ON per connection — so every ON DELETE/UPDATE CASCADE across the schema is aspirational and rename_agent() manually updates all 16 child tables.

Fix: added explicit cleanup in the agent delete handler (DB rows → disk unlinks → volume remove) matching the platform-wide manual-cascade pattern. Platform-wide fix filed as G11 in #490.

2. URL format broken through the frontend proxy

Initial URL was {host}/files/{id}?t={token}. That returned 200 from localhost:8000 but 404 when hit through frontend:80 (Vite dev proxy) or the prod nginx. Plus even when the path was fixed, the credential sanitizer was stripping ?download_token= values (pattern .*TOKEN.*).

Fix: changed build_download_url() to /api/files/{id}?sig={token}. Accept legacy ?download_token= as an alias on the endpoint for backward compat with any URLs minted during the brief window before the fix.

3. /home/developer/test.pdf ≠ publish dir

An agent produced a file at /home/developer/test.pdf and said "done, here's the link" — without ever calling share_file. Obvious in hindsight: agents don't know to write to /home/developer/public/ unless told.

Fix: C8 added a section to the system-wide Trinity prompt (platform_prompt_service.py:PLATFORM_INSTRUCTIONS) instructing every agent on the publish dir + share_file tool. Applies immediately to every agent — no rebuild needed.

Known flaws and concerns

Ranked by severity. All deferred items are tracked in #490.

MEDIUM

• URL tokens end up in stored transcripts. Agents include the returned URL (which contains ?sig=...) in their response text, which gets persisted in chat_messages / schedule_executions rows for up to 7 days. Anyone with DB read access can reuse the URLs until expiration. ?sig= specifically avoids the credential sanitizer pattern, so the token is NOT stripped. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G6.
• Shared /data mount. The backend's SQLite DB, Vector logs, and agent-files storage all sit on the same /data mount. Even with C3's shutil.disk_usage pre-check (blocks writes when free space < 500 MB), a sufficiently fast fill could starve the DB. → mitigated but not eliminated. A platform-wide storage cap (G3) would close this further.
• Memory per share is ~100 MB peak. get_archive reads the tar into a bytearray (up to 50 MB), then tarfile.extractfile().read() holds another copy. Under concurrent shares this stacks. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G1 (streaming extraction).
• Pre-existing platform issue surfaced: PRAGMA foreign_keys isn't enabled, so our FK (and every other FK in the schema) is decorative. Not introduced by this feature. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G11.

LOW

• Flat /data/agent-files/ directory. ext4 directory lookups degrade past ~10k entries. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G2 (UUID-prefix sharding).
• Audit rows bury file_id in JSON details. Admin lookup-by-file-id needs json_extract rather than an indexed column. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G4.
• No pytest for share / download / list / revoke endpoints — unit coverage is only schema/migration + DB mixin + mount-match helper (33 tests). Live-verified manually. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G5.
• Shared rate-limit bucket with public chat previously — fixed in C5 (dedicated file_downloads:{ip} bucket). Not a deployment concern.
• One-time download links deferred — schema kept, but the link-previewer UA defense is not implemented. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G7.
• ?download_token= legacy alias still accepted for backward compat. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G8 to remove.
• UI polish: no pagination (table grows ugly past ~50 rows), no download-trend chart, no clipboard fallback for non-HTTPS contexts. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G9.

INFO

• OTel tracing: SEC-001 Phase 3 shipped on main and wraps all MCP tools with withAudit(), so share_file is audit-logged for free. We did not add an explicit span around create_share() service logic. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G10.
• Request model duplication: ShareFileRequest (internal) and ShareFileMcpRequest (MCP) are near-identical. Intentional for type-clarity but could be consolidated. → FILES-001 follow-ups — needs refinement before pickup (12 items) #490 G12.

Deployment notes

What changes in prod

• DB migration agent_shared_files runs on backend startup (idempotent, name-keyed). Adds 1 table, 3 indexes, 1 column on agent_ownership.
• No docker-compose changes in dev or prod (docker-compose.yml, docker-compose.prod.yml both untouched). The trinity-data mount already exists in both; our /data/agent-files/ subdirectory is created on first share.
• Backend image rebuild required (new routers, new service, new model imports).
• MCP server image rebuild required (new share_file tool + createFileTools registration).
• Frontend image rebuild required (new FileSharingPanel.vue component + store methods + embed in SharingPanel.vue).
• Agent base image unchanged — agents inherit the Docker volume automatically once toggled on.

Rollout order (recommended)

1. Merge to main. CI builds all three images.
2. Deploy backend first (picks up migration + endpoints + cleanup sweep).
3. Deploy MCP server second (registers share_file; agents reconnecting to MCP pick up the new tool).
4. Deploy frontend last (panel appears in the Sharing tab; toggle becomes usable).
5. Optional: flip the toggle ON for any agents that should be able to share files. The system-wide prompt update (C8) means every new agent discovers the tool automatically from their next execution onward — no per-agent CLAUDE.md editing required.

Risk profile

• Backwards compatibility: feature is opt-in per agent (file_sharing_enabled defaults to 0). Deployment cannot regress existing agents — worst case is the new UI panel shows up and the agent owner doesn't touch the toggle.
• Resource cost: 50 MB per share + per-agent 500 MB quota. On a 100-agent deployment that's a theoretical 50 GB ceiling; in practice most agents won't enable it. C3's disk-space pre-check protects against runaway growth.
• Blast radius of bugs: an /api/files/{id} endpoint failure would take out file downloads but leave every other agent interaction intact. An MCP tool registration failure would gracefully degrade — agents would get "tool not found" when they try to call share_file, but chat/tasks/collaboration all keep working.
• Rollback: revert the merge and redeploy. Schema changes stay (idempotent migration). Existing rows + on-disk files remain but become unreachable without the download endpoint; delete_expired_and_revoked sweep will eventually clean them up regardless of rollback. Disk storage is within /data so no host cleanup needed.

What to watch post-deploy

• /data/agent-files/ total size — confirm cleanup sweep is running (log line every cycle if anything was purged).
• 429 rate on /api/files/{id} — if the dedicated bucket hits limits, it's either legitimate traffic or the per-IP threshold needs tuning.
• Audit log file_share_download events — confirm auditing is picking up downloads.
• DB row count for agent_shared_files — should stabilize at "active shares + last 24h revoked" thanks to the cleanup sweep.
• MCP server tool count log line — should say "Registered 73 tools with audit wrapping" (was 72).

Things that WILL surface on prod that we haven't hit in dev

• Real Slack unfurl hitting /api/files/{id} from public internet — we only tested via Tailscale. Check for 403/404/500s in logs immediately after first deploy.
• High-concurrency shares from fan-out or parallel scheduled tasks — the 100 MB peak memory per share isn't an issue under one-at-a-time testing but could matter under load. G1 mitigates.
• Real users hitting the UI panel — clipboard copy may fail on non-HTTPS front-end contexts. G9.
• Agents that don't pick up the new system-wide prompt immediately — it applies on the next invocation, so there's no window of broken behavior, just a gap where an agent might still not know about share_file until it next executes a task.