docs: add DATA_SPLITS.md (scaled eval sets + experiment→split map) - #2
Open
overthelex wants to merge 7 commits into
Open
docs: add DATA_SPLITS.md (scaled eval sets + experiment→split map)#2overthelex wants to merge 7 commits into
overthelex wants to merge 7 commits into
Conversation
Canonical map of the 2K static + 2K temporal LiveHumanRightsBench eval sets (echr-livehrb-static-2k / echr-livehrb-temporal-2k) to per-experiment splits and fields: MFT baseline, Ukraine effect, article buckets, temporal drift, UA pre/post-2022, and the state-swap counterfactual protocol. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per Arian's finding, the article field is a bare number with protocol prefixes collapsed (verified on the parquet: 0 P-codes). "1" == P1-1 (property), "4" conflates Convention Art 4 and Protocol 4. Document the encoding, correct the substantive/procedural buckets to actual values, flag "4" for the state-swap, and confirm the natural label split (overall 83.2%; regular 72.5%, ukr 93.8%). violation_label is a string "violation"/"no_violation". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
article_full is now live on all four datasets. Key per-article logic on it (P1-1, P4-2, P7-4, ...); article kept as lossy legacy. Buckets, experiment map, state-swap protocol, and loading snippet updated to full codes; the Convention-Art-4 vs Protocol-4 split is now clean so no hand-dropping of "4". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified against the live sets: article_full can carry P4-3 (Protocol 4 Art 3) and the bare legacy 1 (unresolved P1-1 that fell back to article). Both are substantive; without them 9 rows in the full pool were in neither bucket. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
overthelex
added a commit
that referenced
this pull request
Jul 16, 2026
Co-authored-by: Vladimir Ovcharov <mcvovkes@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Expand the high-level state-swap block into the full generation + eval contract Arian requested: base pool (001-only substantive ex-UA), 3-4 swap arms per swap_group_id (control_original / control_neutral=Iceland / probe_ukraine / optional probe_russia), state-cue neutralization via placeholders with a legally-material-nationality guard (drop, do not corrupt), respondent passed as a dedicated prompt field, paired per-item metric per article_full bucket, output schema, and the 4 open decisions for model-side sign-off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix a mechanism error in the prior spec commit: neutralization is done by the LLM anonymization prompts (prompts.py STEP2/STEP4 -> [DEFENDANT STATE]/ [DEFENDANT STATE ADJ]/[APPLICANT NATIONALITY]), and placeholders.py:: apply_replacements FILLS those tokens per arm. The swap is anonymize-then-fill (country woven into the already-anonymized body, matching the Mursic example), not a separate prompt field; this neutralizes the leakage concern behind the old separate-field note. Canonical set is not yet anonymized and apply_replacements has no callsite, so the generator must run the anonymization pass + wire the fill. Schema updated (case_text_templated + case_text_rendered, respondent_adj, anonymization_report); no prompt-template change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the ~700-850 estimate with the exact count now that static-2k is rebuilt 001-only: 1000 regular minus 28 procedural = 972 substantive ex-UA items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
DATA_SPLITS.md— the canonical map from experiments to the scaled evaluation sets, so everyone runs against the same splits and fields.What this documents
overthelex/echr-livehrb-static-2k—group ∈ {regular, ukr}overthelex/echr-livehrb-temporal-2k—group ∈ {regular_temporal, ukr_temporal}+binverdict_free_text(input),violation_label(gold),article,respondent,group/bin.Supersedes the 141-pair pilot for the headline benchmark; the pilot stays valid for the RQ1–RQ3 methodology checks.
cc @TerryJCZhang — flagging for the data-side coordination; happy to adjust bucketing / naming to match repo conventions.
🤖 Generated with Claude Code