SimplyPlural importer: real-world export robustness + chat import#134
Merged
Conversation
Cross-referenced with prism's sp_parser.dart. Handles the SP export variants that tidy fixtures miss but live (Mongo/Firebase) exports hit: collections as array OR map-keyed-by-id; a full timestamp normaliser (int/float millis, numeric string, zone-less ISO, Firebase {_seconds,_nanoseconds}) replacing the int-only converter that silently dropped or crashed on the other shapes; variant collection keys (frontStatuses||customFronts, frontHistory->fronters fallback, settings||users[0] system profile); avatar construction from avatarUuid + owner id; ARGB (8-hex, alpha-first) colours. Also folds in the defensive string coercion, the aggregated bad-timestamp warning, and the parse-stage input-counts event from the earlier logging pass. Adds tests for each shape; all SP/TB runner, import-API, and SP-gap-bundle tests pass.
Opt-in SP chat import (off by default - chat can be large). Collapses SP's multi-channel chat onto the Sheaf system board with a channel-name prefix when multi-channel, resolves authors to imported members, rebuilds reply threads, and rewrites SP mention tokens to @name. Reads both export shapes (messages channel-map and the flat chatMessages array) and the sender/timestamp/body field aliases. Legacy message bodies encrypted in SP's undocumented format are DETECTED (16-byte base64 iv + non-plaintext base64 content, heuristic ported from prism) and skipped with a warning - we don't decrypt (no published format, no client does), and no chat content is quoted into the log. New messages option + messages_imported/skipped/encrypted_skipped counters. CHANGELOG covers the full Tier 1 + Tier 2 SP work. Tests for both shapes, encryption detection, reply chains, mentions, and channel prefixing; all SP/TB, import-API, SP-gap, and parity suites pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the SimplyPlural importer against the real-world export variants tidy test fixtures never produce (the shapes that were failing or silently dropping data on live imports), and adds opt-in SP chat-message import. Cross-referenced throughout against the Prism app's SP importer.
Changes
Robustness:
{_seconds, _nanoseconds}objects. The old int-only converter silently dropped (or crashed on) the other shapes, which lost entire front histories.frontStatuses/customFronts,frontHistorywith afrontersfallback, system profile undersettingsorusers[0].avatarUuid+ owner id (still policy-gated through the existing avatar sanitiser), and 8-hex ARGB colours.Chat import (opt-in via a new
messagesoption, off by default since chat can be large):<###@id###>mention tokens are rewritten to@name.messageschannel map and the flatchatMessagesarray) and the sender/timestamp/body field aliases.Testing
ruff check sheaf/passescd web && npm run lint && npx tsc --noEmitpasses (no frontend changes)Security / privacy impact
Article 9 data. The chat-message encryption handling is detect-and-skip only: no decryption, and no message content is ever quoted into the import event log (which admins can view). Imported avatar URLs route through the existing
sanitize_external_avatar_urlpolicy gate, so theavatarUuidconstruction can't bypass the hotlink/scheme rules. Wrong-typed values are dropped rather than interpolated into error text, so a malformed field can't leak member content into a log.Screenshots
No UI changes.