Skip to content
This repository was archived by the owner on Aug 3, 2026. It is now read-only.

fix: load real DCLM bundle format in CORE-22 loader - #35

Merged
bitzic merged 1 commit into
mainfrom
fix/core22-real-bundle-format
Jun 19, 2026
Merged

fix: load real DCLM bundle format in CORE-22 loader#35
bitzic merged 1 commit into
mainfrom
fix/core22-real-bundle-format

Conversation

@bitzic

@bitzic bitzic commented Jun 19, 2026

Copy link
Copy Markdown
Contributor
  • schema parser reads {context_options, continuation, gold}, broadcasting the shared continuation across options (winograd-schema scoring)
  • lm parser reads the continuation key (bundle uses it, not target)
  • both keep legacy {contexts,continuations}/{context,target} for fixtures
  • add task->file aliases: winograd->winograd_wsc, hellaswag_zeroshot->
    hellaswag, jeopardy->jeopardy_all
  • before: 2 schema + 9 lm tasks crashed the parser + 3 failed file resolution -> downstream eval was effectively mc-only (11/22)
  • now all 22 CORE-22 tasks load from the real bundle
  • tests: real-format schema/lm rows, alias resolution, skip-if-absent real-bundle smoke (loads all 22)

- schema parser reads {context_options, continuation, gold}, broadcasting
  the shared continuation across options (winograd-schema scoring)
- lm parser reads the `continuation` key (bundle uses it, not `target`)
- both keep legacy {contexts,continuations}/{context,target} for fixtures
- add task->file aliases: winograd->winograd_wsc, hellaswag_zeroshot->
  hellaswag, jeopardy->jeopardy_all
- before: 2 schema + 9 lm tasks crashed the parser + 3 failed file
  resolution -> downstream eval was effectively mc-only (11/22)
- now all 22 CORE-22 tasks load from the real bundle
- tests: real-format schema/lm rows, alias resolution, skip-if-absent
  real-bundle smoke (loads all 22)
@bitzic
bitzic merged commit bfe5a1f into main Jun 19, 2026
4 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant