Skip to content

docs: add DATA_SPLITS.md (scaled eval sets + experiment→split map) - #2

Open
overthelex wants to merge 7 commits into
AI4Collaboration:mainfrom
overthelex:add-data-splits
Open

docs: add DATA_SPLITS.md (scaled eval sets + experiment→split map)#2
overthelex wants to merge 7 commits into
AI4Collaboration:mainfrom
overthelex:add-data-splits

Conversation

@overthelex

Copy link
Copy Markdown
Collaborator

Adds DATA_SPLITS.md — the canonical map from experiments to the scaled evaluation sets, so everyone runs against the same splits and fields.

What this documents

  • Datasets (public, verdict-free, HUDOC-only, seed=42, leakage-audited v1.1):
    • overthelex/echr-livehrb-static-2kgroup ∈ {regular, ukr}
    • overthelex/echr-livehrb-temporal-2kgroup ∈ {regular_temporal, ukr_temporal} + bin
  • Field referenceverdict_free_text (input), violation_label (gold), article, respondent, group/bin.
  • Article buckets — substantive vs procedural (Art. 41 + procedural reported separately, per the MFT-baseline collapse).
  • Experiment → split map — baseline, Ukraine effect, temporal drift, UA pre/post 2022-02-24, and the state-swap counterfactual protocol (hold facts fixed, swap respondent state to separate country-prior from temporal drift).

Supersedes the 141-pair pilot for the headline benchmark; the pilot stays valid for the RQ1–RQ3 methodology checks.

cc @TerryJCZhang — flagging for the data-side coordination; happy to adjust bucketing / naming to match repo conventions.

🤖 Generated with Claude Code

Canonical map of the 2K static + 2K temporal LiveHumanRightsBench eval
sets (echr-livehrb-static-2k / echr-livehrb-temporal-2k) to per-experiment
splits and fields: MFT baseline, Ukraine effect, article buckets, temporal
drift, UA pre/post-2022, and the state-swap counterfactual protocol.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
overthelex and others added 3 commits July 14, 2026 15:14
Per Arian's finding, the article field is a bare number with protocol
prefixes collapsed (verified on the parquet: 0 P-codes). "1" == P1-1
(property), "4" conflates Convention Art 4 and Protocol 4. Document the
encoding, correct the substantive/procedural buckets to actual values,
flag "4" for the state-swap, and confirm the natural label split
(overall 83.2%; regular 72.5%, ukr 93.8%). violation_label is a string
"violation"/"no_violation".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
article_full is now live on all four datasets. Key per-article logic on it
(P1-1, P4-2, P7-4, ...); article kept as lossy legacy. Buckets, experiment
map, state-swap protocol, and loading snippet updated to full codes; the
Convention-Art-4 vs Protocol-4 split is now clean so no hand-dropping of "4".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified against the live sets: article_full can carry P4-3 (Protocol 4
Art 3) and the bare legacy 1 (unresolved P1-1 that fell back to article).
Both are substantive; without them 9 rows in the full pool were in neither
bucket.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
overthelex added a commit that referenced this pull request Jul 16, 2026
Co-authored-by: Vladimir Ovcharov <mcvovkes@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
overthelex and others added 3 commits July 18, 2026 10:23
Expand the high-level state-swap block into the full generation + eval contract
Arian requested: base pool (001-only substantive ex-UA), 3-4 swap arms per
swap_group_id (control_original / control_neutral=Iceland / probe_ukraine /
optional probe_russia), state-cue neutralization via placeholders with a
legally-material-nationality guard (drop, do not corrupt), respondent passed as
a dedicated prompt field, paired per-item metric per article_full bucket, output
schema, and the 4 open decisions for model-side sign-off.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix a mechanism error in the prior spec commit: neutralization is done by the
LLM anonymization prompts (prompts.py STEP2/STEP4 -> [DEFENDANT STATE]/
[DEFENDANT STATE ADJ]/[APPLICANT NATIONALITY]), and placeholders.py::
apply_replacements FILLS those tokens per arm. The swap is anonymize-then-fill
(country woven into the already-anonymized body, matching the Mursic example),
not a separate prompt field; this neutralizes the leakage concern behind the old
separate-field note. Canonical set is not yet anonymized and apply_replacements
has no callsite, so the generator must run the anonymization pass + wire the fill.
Schema updated (case_text_templated + case_text_rendered, respondent_adj,
anonymization_report); no prompt-template change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the ~700-850 estimate with the exact count now that static-2k is
rebuilt 001-only: 1000 regular minus 28 procedural = 972 substantive ex-UA items.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant