Skip to content

parser: add promptSource to LARGE_ROOT_FIELDS and remove the >32 KB head/tail workaround #1030

Description

@iamtoruk

Follow-up to #1023 (partially addressed by #994, which added isSidechain to LARGE_ROOT_FIELDS).

parseLargeJsonl (src/parser.ts) still drops promptSource for lines > 32 KB, which is exactly where SDK-generated prompts live. #1021's recurring-context detector works around it by regex-scanning the raw line's head/tail (isMachineWrittenPrompt, src/optimize.ts). Proper fix: add promptSource to LARGE_ROOT_FIELDS (tiny scalar, add-only), then delete the head/tail workaround and rely on the parsed field. Add a > 32 KB fixture with promptSource: "sdk" to tests/parser-large-json-scanner.test.ts and assert the recurring-context detector still excludes it. Cache impact: none (promptSource is read at parse time; verify whether it needs to be persisted on CachedFile for the optimize path — if it does, that is add-only like isSidechain).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions