Skip to content

Add text-encoding-safety skill#382

Open
gandzekas wants to merge 1 commit into
addyosmani:mainfrom
gandzekas:feat/text-encoding-safety-skill
Open

Add text-encoding-safety skill#382
gandzekas wants to merge 1 commit into
addyosmani:mainfrom
gandzekas:feat/text-encoding-safety-skill

Conversation

@gandzekas

Copy link
Copy Markdown

Summary

Adds a new text-encoding-safety skill that guides agents through detecting,
diagnosing, recovering, and preventing text-encoding bugs — mojibake, charset
mismatches, BOM mishandling, and Unicode normalization — at I/O boundaries.

Gap justification (per CONTRIBUTING)

  • No existing skill covers text encoding / Unicode safety (verified against the 24 skills in skills/).
  • No open PR proposes one (checked the open-PR list).
  • This is a distinct, common, silent data-corruption class not addressed by security-and-hardening, debugging-and-error-recovery, or api-and-interface-design.

What the skill provides

  • A byte-level reproduce step (capture raw bytes, not rendered glyphs)
  • A classification table mapping garbled patterns (é, €, �) to their root cause
  • Recovery one-liners for the common cases (Latin-1/UTF-8, Windows-1252, double-encode)
  • Boundary-fix guidance (files, HTTP, DB, CSV/JSON, terminals)
  • Unicode normalization (NFC/NFKC) for comparison and search
  • Common rationalizations, red flags, and a verification checklist with runnable regression tests

Verification

  • Follows the anatomy in docs/skill-anatomy.md (frontmatter + Overview / When to Use / Process / Rationalizations / Red Flags / Verification).
  • Added a catalog row in the Build table of README.md.
  • The Verification section includes copy-paste regression tests (test_mojibake_recovery, test_normalization_equality) that fail on the corrupt form and pass once fixed.

This is a new skill; it does not close an existing issue.

Add a new skill covering detection, recovery, and prevention of text-encoding
bugs (mojibake, charset mismatch, BOM, normalization) at I/O boundaries.

Gap: no existing skill or open PR covers text-encoding/Unicode safety - a
common, silent data-corruption class. Catalog row added to README Build table.
@nucliweb

Copy link
Copy Markdown
Collaborator

First off, this is well-made: the byte-level reproduce step, the mojibake classification table, the boundary fixes, and the NFC/NFKC normalization are all solid, and encoding really is an uncovered topic. The hesitation is about fit and altitude, not quality.

Where would this sit in the lifecycle?

The pack is organized by lifecycle phase (Define / Plan / Build / Verify / Review / Ship), and each skill is a broad, phase-level discipline. This is a single-defect-class playbook, which is a different granularity. It shows up in the step mapping: the README row files it under Build, but the body is debugging-shaped (reproduce, classify, recover, guard), which is the exact shape of debugging-and-error-recovery in Verify, and Step 4 (fix at the trust boundary) overlaps security-and-hardening and api-and-interface-design. It is genuinely cross-cutting across Build, Verify, and Review, so it does not land on one step.

A skill that cannot pick a phase is usually a sign it belongs at a different altitude than this pack. Accepting it also shifts the organizing principle from "phases" to "bug topics", which is hard to bound: encoding invites date/timezone, floating-point, path handling, money/decimal, line endings, and so on.

Suggested reshape: a reference, not a lifecycle skill

The pack's model for exactly this kind of topical material is a checklist under references/, pulled in by the broad skills (the way security-checklist.md and performance-checklist.md work). Concretely: move this to references/text-encoding-checklist.md and link it from debugging-and-error-recovery (reproduce/classify), security-and-hardening (untrusted input at the boundary), and api-and-interface-design (UTF-8 everywhere). That keeps all the value without adding a phase skill that does not fit, and without opening the door to a family of per-bug skills.

Since references are principle/checklist shaped, this is also a chance to frame the guidance language-agnostically. The concepts (inspect raw bytes, re-decode with the correct charset, fix at the boundary, normalize for comparison) are universal; the current Python snippets can stay as short illustrations rather than the substance. The project is process-first and not tied to one language, so agnostic framing fits better than either Python-only or a rewrite into another single language.

@addyosmani

Copy link
Copy Markdown
Owner

Encoding bugs are genuinely nasty and underserved, so there's a real itch here. Two things hold it up though. It's a new skill, which is a catalog-growth call I've been batching rather than saying yes to one at a time, and more concretely it fails the eval suite as-is because the new description knocks something out of its own top-k routing. Fix the routing regression first (the runner names the prompt that broke), and I'll weigh it alongside the other new-skill proposals. Holding on scope, not on whether encoding safety matters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants