Address performance problems for StructuredText code blocks

[bisonbet]

Cherry-picked from gonzalezreal/textual PR gonzalezreal#39

Improves reliability of StructuredText when rendering large code blocks and makes Textual more suited to streaming content.

[bisonbet]

@codex please review this PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9bf43d8cc4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Include highlighter theme in tokenization task identity

When a code block is first rendered with .plain or another theme whose tokenProperties is empty, tokenize now intentionally skips Prism and stores only a single plain token; if the same view later receives a syntax-highlighting theme for the same code string, this .task(id: code) does not run again, so the block stays un-tokenized and cannot apply token colors. This affects apps that switch themes dynamically via highlighterTheme(_:); key the task on the tokenization inputs that now include the theme/tokenProperties (or otherwise retrigger tokenization when an empty-token theme changes to a highlighting theme).

Useful? React with 👍 / 👎.

[bisonbet]

@codex please fix the issues you noted in the comments

[chatgpt-codex-connector]

Summary

• Updated HighlightedTextFragment so tokenization is keyed by the full tokenization-relevant input: code, language hint, and highlighter token properties. This retriggers tokenization when switching from .plain to a syntax-highlighting theme for the same code string.
• Included the active highlighter theme in the highlight change observation so already-tokenized code is re-highlighted when the theme changes.
• Updated the file overview comment to reflect the new tokenization identity behavior.
• Committed the changes with commit 9164ba7 and created a follow-up PR titled “Fix highlighter tokenization identity.”

Testing

• ✅ make format
• ✅ git diff --check
• ⚠️ swift test — blocked because SwiftPM dependency fetches from GitHub failed with CONNECT tunnel failed, response 403.
• ⚠️ swift test --skip-update — blocked because SwiftPM still attempted dependency fetches from GitHub and failed with CONNECT tunnel failed, response 403.

View task →