HeuriBoost

RAG reranking that remembers its mistakes.

中文 README · Reference manual · Design specs

The problem

Your RAG system answers a personal-finance question with a passage about the wrong situation.

The retriever did not completely fail. It found the right evidence — but it also found a semantically similar hard negative (same financial topic, wrong entity/situation) and ranked that misleading passage too high. The generator then saw plausible-looking evidence that could not support the answer.

Fixing the retriever's embedding model is expensive and risks regressing everything else. So the mistake quietly comes back next week, in a slightly different query, and nobody notices until a user does.

The idea

HeuriBoost is a lightweight reranker that sits after your retriever and learns from the specific failures you have already seen — and, crucially, never forgets them.

query: "Can I deduct home-office expenses as a sole proprietor?"

retriever output:                          HeuriBoost rerank:
  #1 corporate_office_lease  ✗ hard neg      #1 home_office_deduction  ✓ direct
  #2 standard_deduction      ~ weak          #2 simplified_method      ~ partial
  #3 home_office_deduction   ✓ direct        #4 corporate_office_lease ✗ remembered

When it fixes a mistake, it writes that mistake down as a regression gate. Every future reranker must keep the misleading passage out of the protected top-k, forever. The longer you run it, the more failures it has nailed shut.

That is the whole pitch: a reranker with a memory of its own mistakes, so the same failure can't reappear twice.

How it works

Four stages feed into each other. The first three turn data into an evaluated model; the fourth closes the loop by learning from the model's own mistakes.

flowchart TD
  DATA["Prepare data<br/>retrieve + label candidates"]
  LEARN["Train model<br/>rank documents per query"]
  EVAL["Evaluate & gate<br/>metrics + known failure cases"]
  EVOLVE["Learn from failures<br/>mine similar examples, feed back"]

  DATA --> LEARN --> EVAL --> EVOLVE
  EVOLVE -.next round.-> LEARN

Loading

Stage	What happens
Prepare data	Run retrieval over the corpus (your retriever ships no labels), and grade each query-document pair.
Train model	Turn each row into features and train an XGBoost ranker, grouped so one query's candidates stay together.
Evaluate & gate	Score against retriever baselines and replay the known failure cases. A failing gate case blocks the run.
Learn from failures	For failures still open, mine similar examples and feed them back into training. A reliably fixed failure is promoted by hand into a gate.

The loop in pseudocode:

function run_round(dataset, failure_cases, history):
    train = load(dataset, split="train")

    # optionally attack open failures by mining similar examples
    for case in failure_cases.open:
        train += mine_similar(case, corpus)   # kept separate from the cases

    model = train_ranker(train)

    metrics = evaluate(model)
    results = replay(failure_cases, model)    # a frozen case failing stops here
    history.record(metrics, results)

    suggest_promote(failures_that_now_pass)   # promotion is always manual
    return ok if every frozen case still passes

Two rules are absolute, and they are what make the memory trustworthy:

1. A known failure case is an exam question, never a training row. Only mined samples — deliberately kept separate from the cases — enter training.
2. Promoting an open failure into a frozen gate is always a human decision.

Demo results

The demo uses a real slice of BEIR/FiQA-2018 (financial QA), where a passage on the same topic but the wrong entity is semantically close to the query yet cannot support the answer — exactly the failure HeuriBoost targets.

On the validation split (40 queries), the learned reranker dominates the raw retriever baselines:

Ranker	nDCG@10	MRR@10	Recall@5	Hard-neg@3
HeuriBoost	0.853	0.874	0.797	0.63
dense	0.329	0.403	0.318	2.33
sparse	0.232	0.297	0.208	0.13
RRF	0.281	0.337	0.261	0.95

It generalizes to the cold test holdout (nDCG@10 ≈ 0.83), tracking validation closely — so the gain is not just memorization. Top-3 hard-negative exposure drops from 2.33 (dense) to 0.63.

These numbers come from heuristic labels (qrel positives plus dense-rank-based hard negatives). They illustrate the loop end-to-end; for benchmark-grade labels run the builder with --label-mode llm. See DATA_CARD.

Quick start

HEURIBOOST_RAG_SKILL_DIR=plugins/heuriboost/skills/heuriboost-rag

# install runtime deps (on macOS, also: brew install libomp for xgboost)
python -m pip install -r "$HEURIBOOST_RAG_SKILL_DIR/requirements.txt"

# validate -> train -> evaluate the committed FiQA demo
python3 "$HEURIBOOST_RAG_SKILL_DIR/scripts/validate_dataset.py" examples/fiqa/query_doc_examples.csv
python3 "$HEURIBOOST_RAG_SKILL_DIR/scripts/train_reranker.py"  examples/fiqa/query_doc_examples.csv --output-dir examples/fiqa/output
python3 "$HEURIBOOST_RAG_SKILL_DIR/scripts/eval_reranker.py"   examples/fiqa/query_doc_examples.csv --output-dir examples/fiqa/output --regression-cases examples/fiqa/regression_cases.yaml

Codex plugin

This repo includes a repo-local Codex plugin at plugins/heuriboost/. It packages two skills:

Skill	Use it for
$heuriboost:heuriboost-rag	Audit a RAG repo, bootstrap templates, validate/train/evaluate query-document CSVs.
$heuriboost:reckless-input-builder	Turn messy logs, spreadsheets, tickets, labels, CSV/JSON/JSONL exports, and production feedback into base_dataset + production_cases files for reckless repair.

Install it into Codex from this checkout:

codex plugin marketplace add .
codex plugin add heuriboost@heuriboost-local

Start a new Codex thread after installing or reinstalling so the namespaced skills are injected into the prompt.

Typical prompts:

Use $heuriboost:heuriboost-rag to audit this RAG repo for HeuriBoost readiness.
Use $heuriboost:heuriboost-rag to run an experiment from examples/fiqa/query_doc_examples.csv.
Use $heuriboost:reckless-input-builder to turn these production feedback logs into reckless repair input files.

To run the lower-level reckless closed-loop variant:

HEURIBOOST_RAG_SKILL_DIR=plugins/heuriboost/skills/heuriboost-rag

python3 "$HEURIBOOST_RAG_SKILL_DIR/scripts/train_reranker.py" examples/fiqa/query_doc_examples.csv --output-dir examples/fiqa/output --reckless
python3 "$HEURIBOOST_RAG_SKILL_DIR/scripts/eval_reranker.py" examples/fiqa/query_doc_examples.csv --output-dir examples/fiqa/output --split test --reckless

Reckless mode: production-case fast repair

--reckless is the online-learning-style repair lane for production cases. The user-facing path needs only two tables:

• base_dataset.csv: your stable train / validation / test rows, with minimal columns query,text,relevance.
• production_cases.csv: online failures or feedback rows, with minimal columns query,shown_doc_text,user_verdict.

If your material starts as messy logs, spreadsheets, tickets, labels, or CSV/JSON/JSONL exports, use $heuriboost:reckless-input-builder. It tells an agent how to map raw retrieval candidates, accepted/rejected documents, ranks, scores, and user feedback into these two files, how to choose full vs weak acceptance, and how to validate the result with compile_cases.py.

The repair compiler expands those two tables into the internal query_doc_examples.csv, regression_cases.yaml, and case_sets/ artifacts under output/.heuriboost/compiled/. Those files are audit/debug output, not the user contract.

HEURIBOOST_RAG_SKILL_DIR=plugins/heuriboost/skills/heuriboost-rag

python3 "$HEURIBOOST_RAG_SKILL_DIR/scripts/compile_cases.py" \
  --base-dataset examples/fiqa/repair/base_dataset_minimal.csv \
  --production-cases examples/fiqa/repair/production_cases_full.csv \
  --output-dir examples/fiqa/output \
  --strict

python3 "$HEURIBOOST_RAG_SKILL_DIR/scripts/repair_reranker.py" \
  --base-dataset examples/fiqa/repair/base_dataset_minimal.csv \
  --production-cases examples/fiqa/repair/production_cases_full.csv \
  --output-dir examples/fiqa/output \
  --reckless

python3 "$HEURIBOOST_RAG_SKILL_DIR/scripts/promote_repair.py" \
  --output-dir examples/fiqa/output

This intentionally lets the model absorb current production cases quickly, even if the local fix is close to "overfitting" the observed production problem. The first strict repair run initializes its ledger anchor from the base dataset; later runs reuse that anchor unless --reset-anchor is explicit.

The mode is still strict at acceptance time. Full acceptance is the default: at least one good production doc must enter top-k, all bad docs must stay outside top-k, historical gates must pass, global test nDCG@10 and MRR@10 must both beat the anchor, and every touched domain must avoid regression. Bad-only cases can run with --acceptance-level weak, but weak runs are never promotion eligible.

flowchart LR
    A["base_dataset + production_cases"] --> B["compile generated artifacts"]
    B --> C["repair --reckless trains one candidate"]
    C --> D{"Cases, gates, test, domain pass?"}
    D -- yes --> E["promote explicitly"]
    D -- no --> F["Hard fail and iterate"]

Loading

The lower-level train_reranker.py --reckless / eval_reranker.py --reckless path still exists for maintainers who already work directly with regression_cases.yaml and mined case_sets.

Reports land in examples/fiqa/output/reports/ (gitignored). To bring your own data, follow the CSV contract; for the full failure-attack loop, ledger, and skill modes see the Reference manual.

Implementation checklist

Done:

• Standard query-document CSV contract + validator
• Real XGBoost ranking model, grouped by query_id
• Retriever + text-signal feature set (overlap, hard-negative, length signals)
• FeatureRecipe registry / recipe DSL (declared metadata + load-time leakage/online-safe validation)
• Metrics: nDCG, MRR, recall, hard-negative exposure vs baselines
• Reports: ranking diff, feature importance, deterministic failure analysis
• HPO adapter (Optuna backend, deterministic, case/test-blind search + post-hoc test eval)
• A/B/C/D ablation framework (candidate probe + dual val/test/gate promotion rule)
• LLM candidate discovery (one-shot JSON mode, static validation, feeds ablation)
• Regression cases as gates, with a three-state machine (gate / pending / retired)
• Per-case checks (require_rank, min_ndcg10) + overall-quality check
• Cross-round ledger with a manually-anchored baseline
• case_sets mining loop: mine similar failures, fold into training, isolated from cases
• --reckless loop: fold case_sets directly into training and require test nDCG@10 + MRR@10 to beat the anchor
• Two-table production repair flow: base_dataset + production_cases compile, repair, and explicit promote
• AI-friendly reckless input builder skill for turning raw user materials into compliant repair inputs
• End-to-end FiQA-2018 demo (committed CSV, offline builder, both label modes)
• Codex-compatible agent skill (audit / bootstrap / experiment)

Not yet:

• LLM-mode (benchmark-grade) labels for the committed demo
• Feature promotion memory (FeatureMemory; discovery + ablation are done, institutional memory of decisions pending)
• Other task profiles (classification / regression / …)
• Online serving, shadow/backtest, A/B rollout
• Stable Python package / public API (pyproject.toml)

The design behind the "Not yet" items lives in docs/specs/.

Concepts

HeuriBoost is an adaptive XGBoost framework that learns from labeled examples and historical failures. The shipped specialization is a RAG query-document reranker; the same architecture generalizes to classification, regression, and other supervised tabular tasks.

Concept	Meaning
TaskProfile	Binds a task type to its objective, metrics, gates, slices, and serving behavior. The Q-D reranker is one task profile.
LearningExample	One supervised row. For ranking, rows share a group_id (query_id).
PredictionContextSnapshot	The immutable candidate set a model is evaluated against. Comparing models requires the same snapshot.
RegressionCase	A historical failure expressed as a gate. A gate, never training material.
FeatureRecipe	A declared, versioned feature (inputs, cost, online-safety, leakage risk). Features live in a registry, not scattered code.
PromotionGate	The bar a candidate model must clear (global metric, per-case, slice, latency) before replacing the current one.
FeatureMemory	The record of which features were promoted / rejected / quarantined, and why.

Full definitions are in docs/specs/ADAPTIVE_XGBOOST_HEURISTIC_SPEC.md.

Repository layout

.
├── README.md / README.zh-CN.md      project story, concepts, demo
├── .agents/plugins/marketplace.json repo-local Codex plugin marketplace
├── docs/
│   ├── REFERENCE.md                  contracts + commands (operational reference)
│   └── specs/                        long-form design specs
├── examples/fiqa/                    the committed FiQA demo + cases + ledger
└── plugins/heuriboost/               Codex plugin packaging both skills and the runtime

There is no pyproject.toml yet — run the skill-local scripts directly.