S3: 104 objects are stored latin-1-expanded — migration must precede the read_bytes fix on any env

[kwit75]

Warning

#2114 is not safe to apply to an environment until that environment's bucket is migrated. It repairs binary reads that never worked and simultaneously breaks binary objects that did work. Staging is already migrated. Production is not.

What is in the buckets

Before #2114, Store.write_bytes had no S3 implementation, so it fell back to data.decode('latin-1') → write_file → .encode('utf-8'). Every byte above 0x7F became two. Objects written that way are stored expanded.

feature-1.png   stored     231241 bytes   \xc2\x89PNG
                unwrapped  153258 bytes   \x89PNG

\xc2\x89 is the UTF-8 encoding of 0x89. Inflation ~1.5x.

The old, apparently-broken read path handled these correctly. Expanded data is valid UTF-8, so read_file decoded it and .encode('latin-1') returned the original bytes. The round trip was self-consistent.

After #2114, read_bytes returns the stored bytes verbatim — the expanded ones. They are served through shell.py:366 and app_handler.py:150, so browsers receive corrupt fonts and images.

Measured, staging

104 of 778 objects, all under the system org, in the built-in apps:

app	objects
rocketride.pipeBuilder	80
rocketride.home	12
rocketride.benchmark	8
rocketride.aparavi	4

By type: 43 woff2, 41 woff, 19 png, 1 gif.

Staging migrated and verified — full rescan, 778 objects, zero remaining expanded.

Detecting these is harder than it looks — two traps

Magic bytes do not work for fonts. wOF2 and wOFF are ASCII, so they survive the expansion unchanged. A classifier keyed on the signature reports an expanded font as healthy. My first pass found 18 of 104 for exactly this reason.

Use the length field instead: WOFF and WOFF2 both carry total file size as a big-endian uint32 at offset 8. Expanded ⇒ declared ≠ actual.

Extensions lie. install.png is a JPEG. Sniff content, not the name.

Safe procedure per environment

1. Enumerate objects; skip pure-ASCII (latin-1 and UTF-8 agree — indifferent)
2. Keep only those that decode as UTF-8 with all codepoints < 256
3. Unwrap: data.decode('utf-8').encode('latin-1')
4. Write back only if the unwrapped bytes validate in the format's own terms — WOFF length field, PNG 8-byte signature, JPEG SOI+EOI. Never on a guess.
5. Rescan to confirm zero remain

Step 4 is what makes this safe to re-run: an already-migrated object fails the unwrap check and is left alone.

Origin

I asserted in #2114 that no existing object changes meaning, on the evidence of a single sampled remoteEntry.js that happened to be pure ASCII. That was a sample reported as a population.

🤖 Generated with Claude Code

https://claude.ai/code/session_015nTVr6jfSFYm1GppxbjghP

[github-actions]

🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

[kwit75]

One more reason migration must come first: copy_file propagates the damage, and it did not before.

store.py implements copy_file as read_bytes then write_bytes. Before #2114 those two were accidentally symmetric on S3 — the read reversed the expansion, the write re-applied it — so a copy of an expanded object produced another correct-on-read expanded object. Self-consistent, if by accident.

After #2114 the read returns expanded bytes and the write stores them verbatim. A copy now produces an object that is genuinely corrupt rather than one that merely looks it.

Worse across providers: copying to a destination whose write_bytes still routes through latin-1 — azure and memory both still do — expands an already-expanded object a second time.

So on any environment that has not been migrated, do not run copies, moves, or backups over affected prefixes. Moves implemented as copy-then-delete are the dangerous shape, because the correct original is deleted at the end.

Migration is idempotent and safe to re-run: an already-raw object fails the format validation in step 4 and is left alone.

Credit where due — I found the stored-expansion problem myself, but this consequence came out of an adversarial review pass, not from me.

[kwit75]

Correction: production has nothing to migrate

I wrote above that production is unmigrated and that applying #2114 there would break fonts and images for customers. That was wrong, and wrong the same way the original mistake was — I measured staging and extrapolated to prod without enumerating it.

Enumerated both stores in full:

rocketride-saas-dev-storage       (prod)      1609 objects   orgs/ = 0   .deployments/ = 0
rocketride-saas-staging-storage   (staging)    780 objects   orgs/ = 725 .deployments/ = 725

Production holds no deployed-app content at all. 3.6 MB total, entirely users/… prefixes of .json, .pipe and .chat files. No fonts, no images, no bundles, no zips. The app-deploy feature has never been used there.

Expanded objects only exist where write_bytes wrote binary content, and on prod nothing ever did.

What actually applies

• fix(store): give the S3 provider real binary I/O #2114 is safe to apply to production as-is. There is nothing to migrate first.
• The migration requirement is conditional on an environment having deployment objects, not a blanket precondition. Staging had 104 and is done.
• The copy_file hazard in the comment above is likewise conditional — it needs expanded objects to propagate, and prod has none.

The procedure in the issue body stays correct and is worth keeping: it is idempotent, and any future environment restored from a pre-fix bucket will need it.

Why prod looks different

Prod's built-in apps are served from the CloudFront UI bucket (cloud.rocketride.ai), populated by aws s3 sync from a build, not through the store's write_bytes. Those objects are raw by construction and were never at risk. Staging's shell serves them out of the store, which is why staging had them and prod does not.

The pattern, since it is the second time in an hour

The first error was one sampled object standing in for a bucket. The second was one measured bucket standing in for an environment. Both were reasoning that would have been right if the input had held, and in both cases the input was never checked — only assumed to generalise.