feat: live HUDOC auto-refresh pipeline (self-refreshing, contamination-controlled)#3
Open
overthelex wants to merge 4 commits into
Open
feat: live HUDOC auto-refresh pipeline (self-refreshing, contamination-controlled)#3overthelex wants to merge 4 commits into
overthelex wants to merge 4 commits into
Conversation
Adds the "Live" data pipeline that keeps the benchmark contamination-controlled as new ECtHR judgments are published: - hudoc_live_refresh.py: cron-friendly orchestrator (detect latest date -> fetch new judgments -> verdict-leakage removal -> append -> optional HF push) - hudoc_scraper.py: HUDOC bulk ingestion (search + full texts) - verdict_leakage_removal.py + conclusion_scrub.py: 3-stage verdict removal - eval-set builders (static-2k, temporal-2k, stratified, cutoff partitions) + respondent normalization, decision-date backfill, enrich/push - configs/model_cutoffs.json for per-model pre/post-cutoff partitioning - PIPELINE.md documenting the chain; requirements: +requests, +boto3 Supersedes the closed #1 and adds the live-refresh orchestrator that #1 lacked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n parse parse_conclusion_to_pairs collapsed "Article 1 of Protocol No. 1" to "1" (the P1-1 conflation Arian flagged). New parse_article_code() extracts the protocol-aware code into a non-breaking article_full column while keeping the legacy collapsed article for continuity. Verified on representative HUDOC conclusion strings. Part of the full-article-code rebuild (LSYC-41). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… HUDOC sweep Published sets dropped the conclusion column, so article_full can't be recovered from the row. This resumable sweep re-fetches HUDOC conclusion per item_id (exact-itemid guard), parses it with the fixed parser, and writes a non-breaking article_full column. Rows that can't be resolved unambiguously fall back to the legacy article (never fabricated) and are counted. Supports --limit test batches and --push-to-hf. Verified: offline enrich() unit test + live 3-item HUDOC fetch (001-246017 article 1 -> P1-1). Part of LSYC-41. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-item exact queries (~14,449 requests, ~3.5h) replaced by a paged bulk sweep over per-year JUDGMENTS windows (~500/page, under the 10K offset cap), keeping only target ids, then a per-item fallback for the uncovered 002-* Information Notes. Validated: the 2024-2026 window alone resolves 2,280 ids with correct protocol codes (P1-1, P7-2, P4-2, ...). Full run drops from hours to ~15-20 min. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
overthelex
added a commit
that referenced
this pull request
Jul 16, 2026
Contamination MFT pipeline + adapter (interim article remap, custom_id & label fixes)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires the "Live" data pipeline into the shared repo: the self-refreshing HUDOC ingestion that keeps the benchmark contamination-controlled as new ECtHR judgments are published. This is the piece that makes it LiveHumanRightsBench rather than a static snapshot.
Auto-refresh chain
scripts/hudoc_live_refresh.py— cron-friendly orchestrator: detect latestdecision_date→ fetch new judgments from HUDOC → verdict-leakage removal → append → optional HF push.scripts/hudoc_scraper.py— HUDOC bulk ingestion (search + full texts).scripts/verdict_leakage_removal.py+scripts/conclusion_scrub.py— 3-stage verdict removal (pattern truncation → dual-model verify → repair), with the mandatory conclusion-scrub stage (rows marked+conclusion_scrub).Eval-set construction (produces the sets documented in DATA_SPLITS.md / PR #2)
build_livehrb_static.py→echr-livehrb-static-2kbuild_temporal_split.py→echr-livehrb-temporal-2kbuild_cutoff_partitions.py→ per-model matched pre/post training-cutoff partitions (contamination control), driven byconfigs/model_cutoffs.jsonbuild_stratified_sample.py,clean_respondent_names.py,backfill_decision_dates.py,enrich_and_push.py— sampling, normalization, metadata, HF push.See
PIPELINE.mdfor the full chain and cron usage.Notes
openai/boto3) are guarded, so ingestion + regex conclusion-scrub run without cloud credentials.requirements.txt: +requests, +boto3.cc @TerryJCZhang — this is the data/live-refresh piece owned end to end on my side (CRediT: Data Curation / Methodology / Software). Happy to adjust the layout to repo conventions.
🤖 Generated with Claude Code