Sync object metadata

[peterjan]

Description added by @chris124567:
Close #133

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	go.sia.tech/​siastorage@​v0.0.4-0.20260410121042-fa1e8e354ff0 ⏵ v0.0.4-0.20260428134956-8039d0326806	+1

View full report

[Copilot]

Pull request overview

This PR replaces the per-request object metadata refresh/cache approach with a background metadata sync loop driven by indexer object events, persisting a cursor in SQLite to resume syncing across restarts.

Changes:

• Replace SDK.Object with SDK.ObjectEvents and add a periodic syncMetadataLoop that applies updates via Store.UpdateSiaObject.
• Refactor persisted Sia metadata to use a new objects.SiaObject struct and add cursor persistence (objects_cursor_at/key) to SQLite global_settings.
• Update tests to validate cursor persistence and metadata update behavior.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
sia/sia.go	Introduces metadata sync loop and updates SDK/Store interfaces to be event/cursor-based.
sia/sdk.go	Updates indexd SDK wrapper to expose ObjectEvents and to seal/unseal via objects.SiaObject.
sia/objects.go	Removes on-demand metadata refresh; downloads now rely on locally stored sealed metadata.
sia/multipart.go	Updates part-copy logic to unseal from stored metadata instead of fetching by object ID.
sia/objects/objects.go	Replaces Object.ID/CachedAt with Object.SiaObject *SiaObject and defines SiaObject struct.
sia/sia_test.go	Updates in-memory SDK test double to serve ObjectEvents and to use new SiaObject types.
sia/objects_test.go	Replaces metadata-cache test with TestSyncMetadata validating cursor advancement + resealing.
sia/export_test.go	Exposes sync iteration method for tests.
sia/persist/sqlite/objects.go	Adds cursor persistence, refactors object read/write paths, updates MarkObjectUploaded/UpdateSiaObject.
sia/persist/sqlite/multipart.go	Updates inserts/updates to match new objects schema (no cached_at).
sia/persist/sqlite/init.sql	Removes cached_at; adds objects_cursor_at/key columns.
sia/persist/sqlite/migrations_test.go	Updates “v1” schema fixture to match new schema.
sia/persist/sqlite/objects_test.go	Updates tests for new SiaObject storage and adds cursor/update tests.
go.mod / go.sum	Bumps go.sia.tech/siastorage to a newer pseudo-version.

Comments suppressed due to low confidence (1)

sia/persist/sqlite/init.sql:26

• This schema change removes cached_at and adds new global_settings columns, but the SQLite schema version is still len(migrations)+1 (currently 1) and migrations is empty. Existing databases created with the previous v1 schema (where objects.cached_at was NOT NULL) will fail on inserts/updates that no longer provide cached_at. Add a migration (and thus bump the schema version) that updates existing DBs to this new schema (dropping/rewriting the objects table to remove cached_at and adding/initializing the new cursor columns).

CREATE TABLE objects (
    bucket_id INTEGER REFERENCES buckets(id) NOT NULL,
    name TEXT NOT NULL,
    object_id BLOB,
    content_md5 BLOB NOT NULL,
    metadata TEXT NOT NULL,
    size INTEGER NOT NULL,
    updated_at INTEGER NOT NULL,
    filename TEXT, -- name of file for regular uploads or dir for multipart uploads
    sia_object BLOB,
    -- file is either stored on disk, on Sia or empty.
    CHECK ((sia_object IS NULL AND object_id IS NULL AND filename IS NOT NULL) OR (sia_object IS NOT NULL AND object_id IS NOT NULL AND filename IS NULL) OR (object_id IS NULL AND filename IS NULL AND size = 0)),
    PRIMARY KEY (bucket_id, name)
) WITHOUT ROWID;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chris124567]

LGTM but there's a boatload of conflicts now due to upload packing 😅