Skip to content

mem migrate silently drops rows with nested idField (e.g. Attio companies id.record_id) #132

@moekatib

Description

@moekatib

Summary

`mem migrate` treated `profile.idField` as a flat property name (`row[idField]`) instead of a dotted path. On Attio companies (`idField: "id.record_id"`) where the legacy SQLite stores `id` as a JSON-stringified `{workspace_id, object_id, record_id}`, every row failed to resolve and was silently counted as "skipped" — including on an empty memory store, where skipped is never a legitimate outcome.

Repro

rm -rf ~/.one/mem.pglite
one --agent mem migrate -y
→ {\"platform\":\"attio\",\"model\":\"attioCompanies\",\"rowsSeen\":2024,\"inserted\":0,\"updated\":0,\"skipped\":2024}

Root cause

  1. `migrate.ts` used `row[idField]` (flat lookup) instead of `getByDotPath(row, idField)` (same mechanism sync uses).
  2. Legacy SQLite JSON-stringifies nested objects before INSERT, so `row.id` was a string like `'{"record_id":"..."}'` — a dotted path wouldn't walk through it anyway without hydration.
  3. The report exposed a single `skipped` counter, so a profile misconfiguration and a real key conflict looked identical.

Fix

Shipped in `feat/unified-memory` (next push):

  • `reviveStringifiedJson()` rehydrates top-level JSON-stringified columns before id resolution, matching what sync sees live.
  • Migrate now calls `getByDotPath(row, idField)` with the rehydrated row.
  • Hard reject when the resolved id is a nested object (same guard sync test/mem-writer apply) — prevents silent `[object Object]` collapse.
  • Report splits `skipped` into `skippedUnresolvedId` + `skippedError`.
  • Human output prints a warning when every row is skipped ("profile missing or idField doesn't resolve").

Verified

Unit tests for rehydration + dotted-path resolution on the exact shape that broke (Attio `id.record_id`).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions