Skip to content

program.md

Latest commit

 

History

History
169 lines (119 loc) · 5.02 KB

program.md

File metadata and controls

169 lines (119 loc) · 5.02 KB
Raw

LTR autoresearch

This project uses an agent to improve a learning-to-rank model through repeated, measured experiments.

Goal

Maximize normalized recall log-AUC across these product cutoffs:

100, 400, 1_000, 2_000, 5_000, 10_000, 20_000

Higher is better. Validation NDCG is diagnostic only. Recall log-AUC decides whether a model is kept.

Files

The only file the research agent may edit is:

  • src/ltr_training/autoresearch_train.py

The agent may read but must never edit:

  • src/ltr_training/autoresearch.py
  • src/ltr_training/autoresearch_dashboard.py
  • src/ltr_training/features.py
  • src/ltr_training/ranking.py
  • src/ltr_training/recall_evaluation.py
  • src/ltr_training/sampling.py
  • src/ltr_training/storage.py
  • src/ltr_training/vespa.py
  • program.md
  • datasets and files under output/

Do not install packages or add dependencies. Do not rebuild the dataset. Do not run SHAP during research.

Editable training surface

Experiments may change:

  • MODEL_FEATURES, using any non-empty subset of ALL_FEATURES
  • LightGBM parameters except the fixed settings below
  • boosting rounds
  • early stopping
  • training logic

Keep these settings fixed:

  • objective="lambdarank"
  • metric="ndcg"
  • seed=42

Prefer feature-selection and ranking-strategy experiments over fine-grained hyperparameter tuning. Keep current baseline values unless changing a parameter tests a clear hypothesis. Do not run grid searches or random sweeps. Change at most two related hyperparameters per experiment. Parameters not explicitly configured may use LightGBM defaults.

Keep the output contract intact. A successful trainer must write:

  • model.txt
  • training_metadata.json
  • training_metrics.json

Storage

Git is not used by the experiment loop. The controller snapshots every trainer version under:

output/<dataset-version>/autoresearch/runs/<run-id>/

It also maintains:

  • champion.json: current winning model and scores
  • results.tsv: all experiments
  • dashboard.html: auto-refreshing progress dashboard
  • harness_hashes.json: fixed evaluator hashes
  • STOPPED.json: fatal Vespa/evaluation stop marker

The controller restores the champion trainer after every discarded, crashed, or invalid experiment. It regenerates dashboard.html after every recorded result. A dashboard rendering failure is recorded separately and never stops research.

Baseline

The baseline must use all ALL_FEATURES.

Run it once:

uv run --env-file .env python src/ltr_training/autoresearch.py baseline

The baseline trains once and evaluates the first 500 Brave queries. Its 100-query score is derived from the first 100 results in that same evaluation. After the baseline finishes, open output/<dataset-version>/autoresearch/dashboard.html in a browser. It refreshes every 30 seconds and is regenerated after each recorded experiment.

Experiment loop

For each experiment:

  1. Read program.md, champion.json, results.tsv, and the current autoresearch_train.py.
  2. Choose one clear idea.
  3. Edit only autoresearch_train.py.
  4. Run exactly one controlled experiment:
uv run --env-file .env python src/ltr_training/autoresearch.py experiment \
  --description "short description of the idea"
  1. Read the printed status and saved artifacts.
  2. Continue with another idea unless the controller reports a fatal evaluation failure or the human stops the session.

The controller applies this funnel:

  1. Train model.
  2. Evaluate same first 100 Brave queries.
  3. If 100-query log-AUC does not beat champion, discard.
  4. If it beats champion, evaluate first 500 Brave queries.
  5. Promote only when 500-query log-AUC beats champion.

One main idea per experiment. Prefer simple changes. A tiny gain from deleting complexity is valuable. A tiny gain requiring fragile complexity is not.

Limits and failures

  • Training timeout: 10 minutes
  • 100-query evaluation timeout: 8 minutes per whole-evaluation attempt
  • 500-query evaluation timeout: 40 minutes per whole-evaluation attempt
  • Query concurrency: fixed at 4
  • Vespa concurrency: fixed at 5

Training crash or timeout:

  • Controller saves source and logs.
  • Controller records failure.
  • Controller restores champion.
  • Agent may inspect the failure and continue.

Invalid score:

  • Any missing cutoff, NaN, wrong query count, partial result, or changed fixed harness invalidates the experiment.
  • Controller restores champion.

Vespa/evaluation failure:

  • Existing per-query retries run first.
  • Controller retries the whole evaluation once.
  • If the second attempt fails, controller cancels work, writes STOPPED.json, restores champion, and exits with status 2.
  • Agent must stop immediately. Never call clear-stop or refresh-harness. Only a human may inspect Vespa, clear a stop marker, or approve a fixed harness change.

Never stop rule

After a human explicitly starts the autonomous experiment loop, continue until:

  • human interrupts,
  • controller creates STOPPED.json, or
  • controller exits with status 2.

Do not ask whether to continue between normal experiments.