llm-confidence-scoring

Do proper scoring rules reduce LLM overconfidence? This repository studies whether stating the logarithmic scoring rule in the prompt changes how a large language model reports confidence on multiple-choice QA, and whether those confidence estimates support better abstention policies.

It extends the setup of Kalai, Nachum, Vempala & Zhang, "Evaluating large language models for accuracy incentivizes hallucinations" (Nature, 2026), which argues that accuracy-only evaluation rewards confident guessing and thereby incentivizes hallucination. Here we ask the inference-time counterpart: if the model is told it will be scored by a strictly proper rule, does it report less overconfident probabilities?

Headline finding. Stating the log rule sharply reduces overconfidence and improves the log score on multiple-choice tasks (MMLU, and more so on the harder MMLU-Pro), with negligible accuracy cost. An exploratory SimpleQA run did not show the same benefit on open-ended factual QA, so we treat that as future work with stronger models and graders. See the technical report in paper/report.pdf.

What's in here

Path	Contents
llm.py	Unified, cached, multi-provider LM wrapper (DeepSeek / OpenRouter / OpenAI / any OpenAI-compatible endpoint).
experiments/run_experiments.py	CLI for the MMLU / MMLU-Pro confidence experiments, plus SimpleQA exploratory runs.
experiments/calibration.py	Post-hoc calibration (isotonic regression, temperature scaling, ECE/MCE).
experiments/analysis.py	Paired-bootstrap deltas, reliability/risk-coverage figure generation.
experiments/make_report.py	Aggregates a full run into the Markdown/LaTeX report.
experiments/config.json	Exact commands + seeds for the headline run.
experiments/prompts.md	Prompt templates for the neutral and log-rule conditions.
figures/	Publication figures (PDF) from the headline run.
paper/report.pdf	Technical report (LaTeX source in paper/).
data/README.md	How to download the benchmarks and regenerate results.

Setup

python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt

cp .env.example .env   # then add your API key(s)

Choosing a provider

llm.py talks to any OpenAI-compatible chat-completions endpoint. Pick a provider with --provider (or LLM_PROVIDER) and a model with --model (or LLM_MODEL):

Provider	Base URL	Key env
deepseek	https://api.deepseek.com	DEEPSEEK_API_KEY
openrouter	https://openrouter.ai/api/v1	OPENROUTER_API_KEY
openai	https://api.openai.com/v1	OPENAI_API_KEY
custom	$LLM_BASE_URL	LLM_API_KEY

from llm import make_lm
lm = make_lm(provider="openrouter", model="gpt-4.1")   # short names map to OpenRouter ids
print(lm.generate("Say hi in one word.", max_tokens=8))

All calls are cached on disk under .cache/<provider>/, so reruns are cheap and reproducible.

Quickstart

# Connectivity smoke test (any provider)
python experiments/run_experiments.py smoke --provider deepseek --n 3

# MMLU: neutral vs. log-rule probability elicitation (small)
python experiments/run_experiments.py mmlu-confidence --n 20 --seed 17 --temperature 0
python experiments/run_experiments.py mmlu-confidence --n 20 --seed 17 --temperature 0 --include-log-rule

# MMLU-Pro: same prompt comparison on a harder multiple-choice benchmark
python experiments/run_experiments.py mmlu-confidence --dataset mmlu-pro --n 20 --seed 17 --temperature 0
python experiments/run_experiments.py mmlu-confidence --dataset mmlu-pro --n 20 --seed 17 --temperature 0 --include-log-rule

# Optional exploratory SimpleQA run
python experiments/run_experiments.py simpleqa-confidence --n 10 --seed 17 --temperature 0

Swap in a different model/provider with e.g. --provider openrouter --model opus-4.5.

Reproducing the headline run

The full run (deepseek-v4-flash, seed 17, n=1000 per dataset) is specified command-by-command in experiments/config.json, and the aggregated tables/figures are produced by experiments/make_report.py. Benchmark download links and the data layout are in data/README.md. Raw result files are intentionally not committed (regenerate them from the config).

Citation

This work builds on Kalai et al. (2026):

@article{kalai2026evaluating,
  author  = {Kalai, Adam Tauman and Nachum, Ofir and Vempala, Santosh S. and Zhang, Edwin},
  title   = {Evaluating large language models for accuracy incentivizes hallucinations},
  journal = {Nature},
  year    = {2026},
  doi     = {10.1038/s41586-026-10549-w},
  url     = {https://doi.org/10.1038/s41586-026-10549-w}
}

See docs/relation_to_openai.md for exactly what is reused vs. new, and NOTICE for attribution.

License

MIT — see LICENSE. Derived in part from OpenAI's MIT-licensed reference code for the paper above.