Skip to content

README.md

Latest commit

 

History

History
181 lines (130 loc) · 5.84 KB

README.md

File metadata and controls

181 lines (130 loc) · 5.84 KB
Raw

Pioneer Adaptive Finetuning Lab

Experiment harness for using api.pioneer.ai as an adaptive finetuning system for small coder models, with evaluation hooks for:

  • LiveCodeBench style coding benchmarks
  • Aider code editing/refactor benchmark style tasks

This repo is intentionally config-driven so you can iterate quickly on:

  1. Which Pioneer base/candidate models to test
  2. Which benchmark commands to run
  3. Fine-tuning hyperparameters and stopping policy

What this gives you

  • A reusable Pioneer API client (/v1/models, /v1/chat/completions, /felix/datasets/upload, /felix/training-jobs)
  • A benchmark runner that executes arbitrary commands and parses scores
  • Weighted aggregation across benchmarks
  • Adaptive loop:
    • establish inference baselines first (seed + candidates)
    • ask Pioneer adaptive agent (/adaptive-finetuning/chat) for the next action/model
    • persist the full adaptive API response for each iteration (endpoint-first auditing)
    • evaluate the API-recommended model and promote only when score improves
  • JSON history + summary artifacts in outputs/

Quick start

1) Install

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

2) Configure API key

export PIONEER_API_KEY="your_key_here"

Optional: override API base URL (for local backend testing):

export PIONEER_API_BASE_URL="http://127.0.0.1:8000"

Keep keys in environment variables; do not commit them into files.

3) Validate config

pioneer-adaptive validate-config configs/experiment.yaml

4) Run benchmark-only evaluation

pioneer-adaptive run-benchmarks "llama-3.1-8b" configs/experiment.yaml --run-id "smoke-001"

5) Run adaptive finetuning cycle

pioneer-adaptive run-cycle configs/experiment.yaml

Results are written to:

  • outputs/<run_id>/history.json
  • outputs/<run_id>/summary.json
  • outputs/<run_id>/baselines.json

The CLI prints the generated run_id at the end of each cycle.

Repository layout

configs/experiment.yaml             # Main experiment config
data/train.jsonl                    # Example train file
data/valid.jsonl                    # Example validation file
scripts/setup_benchmarks.sh         # Clone external benchmark repos
scripts/run_livecodebench_mini.py   # LiveCodeBench mini baseline runner
scripts/run_aider_refactor_mini.py  # Aider refactor mini baseline runner
scripts/run_python_functions_benchmark.py # Deterministic 20-task function benchmark
scripts/run_*_stub.py               # Optional deterministic dry-run runners
src/pioneer_adaptive/
  config.py                         # Pydantic config schema
  pioneer_client.py                 # Pioneer API wrapper
  benchmarking.py                   # Command execution + score parsing
  adaptive_loop.py                  # Adaptive finetuning orchestration
  cli.py                            # Typer CLI
tests/

Benchmarks used by default

Default config points to real mini benchmark subsets:

  • scripts/run_livecodebench_mini.py: runs a subset of LiveCodeBench code-generation tasks via Pioneer inference and computes pass@1.
  • scripts/run_aider_refactor_mini.py: runs a subset of Aider refactor tasks and checks refactor correctness via AST rules.

To prepare benchmark assets:

  1. Clone benchmark repos:

    ./scripts/setup_benchmarks.sh

  2. Ensure each benchmark emits a metric artifact (JSON file or JSON stdout), then point parser fields:

  • parser.mode: json_file, stdout_json, or regex
  • parser.key_path: dot path to metric field, e.g. metrics.pass_at_1

CLI reference

pioneer-adaptive validate-config [CONFIG_PATH]
pioneer-adaptive list-models [CONFIG_PATH]
pioneer-adaptive run-benchmarks MODEL_ID [CONFIG_PATH]
pioneer-adaptive run-cycle [CONFIG_PATH]

Additional benchmark for adaptive sweeps

For quicker and more stable coding experiments (less sensitive to huge prompt context), use the included deterministic function benchmark:

python3 scripts/run_python_functions_benchmark.py \
  --model-id "base:Qwen/Qwen3-8B" \
  --out outputs/experiments/python_functions_base.json \
  --repeat 3 \
  --max-tokens 1800

You can run the same benchmark against a tuned decoder job ID:

python3 scripts/run_python_functions_benchmark.py \
  --model-id "<training-job-uuid>" \
  --out outputs/experiments/python_functions_tuned.json \
  --repeat 3 \
  --max-tokens 1800

An example config is provided at configs/experiment_python_functions.yaml.

Notes

  • Baseline inference uses:
    • /v1/chat/completions for standard inference model IDs
    • /inference for base:* decoder IDs
  • Tuned decoder inference uses /inference with the training job UUID as model_id.
  • Adaptation orchestration uses Pioneer adaptive endpoint (/adaptive-finetuning/chat) as the source of truth.
  • Decoder inference evaluation uses /inference (training job UUIDs and base:* IDs).
  • Fine-tuning internals are delegated to Pioneer’s adaptive system.
  • This repo no longer triggers fine-tune jobs directly inside the loop; it delegates decisions to Pioneer adaptive chat and only performs benchmark evaluation locally.
  • Benchmark scripts honor PIONEER_API_BASE_URL (default: https://api.pioneer.ai).
  • If your Pioneer tenant uses custom endpoint conventions, adjust src/pioneer_adaptive/pioneer_client.py.
  • Adaptive policy knobs live under policy in configs/experiment.yaml.

Latest experiment learnings

  • Mini LiveCodeBench/Aider runs are highly sensitive to long <think> traces and token budget.
  • Session instability (Too many active sessions) can impact both inference and training jobs.
  • On a deterministic executable benchmark (python_functions_20), we observed small but repeatable tuned improvements in some runs.
  • Committed experiment artifacts are stored in results/.