Petri and Bloom — LLM Benchmarking Setup

LLM safety evaluation using Anthropic's Bloom (automated behavioral evaluations), Petri (open-ended safety auditing), and the local rescore.py helper for rerunning judge scoring on completed eval logs.

What This Repo Does

• Runs Bloom evaluations from bloom-data/
• Runs Petri/Inspect evaluations from petri-seeds/ and petri-data/
• Rescores completed .eval logs with rescore.py

Rescoring logs

Use rescore.py to re-run animal-welfare judge scoring on one or more completed eval logs.

source .env
.venv/bin/python rescore.py logs/some_log.eval [logs/other_log.eval ...] [--judge-model MODEL]

If you do not pass any log files, the script prints a usage message and exits.

File origins

File	Source
.gitignore	Written by hand
.env	Pre-existing; comment added manually
bloom-data/models.json	Generated by bloom init, then extended with Gemini/OpenAI/Claude models
bloom-data/seed.yaml	Generated by bloom init (unmodified)
bloom-data/behaviors.json	Generated by bloom init (unmodified)
bloom-data/configurable_prompts/default.json	Generated by bloom init (unmodified)
bloom-data/behaviors/examples/README.md	Generated by bloom init (unmodified)
logs/, petri-data/	Created as output directories

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install "git+https://github.com/safety-research/bloom.git"
pip install "git+https://github.com/safety-research/petri"

Add credentials to .env (Vertex AI is already configured; uncomment others as needed):

source .env

---

Running a test evaluation

The configuration below evaluates gemini-3.1-flash-lite-preview as the target model, using gemini-3-flash-preview as the evaluator/auditor, and gemini-2.5-flash-lite as the judge.

Bloom

Edit bloom-data/seed.yaml to set:

understanding:
  model: "gemini-3-flash-preview"

ideation:
  model: "gemini-3-flash-preview"
  num_scenarios: 3

rollout:
  model: "gemini-3-flash-preview"
  target: "gemini-3.1-flash-lite-preview"

judgment:
  model: "gemini-2.5-flash-lite"

Then run:

source .env
.venv/bin/bloom run bloom-data

Results are written to bloom-results/. View them with:

npx @isha-gpt/bloom-viewer --port 8080 --dir ./bloom-results

Petri

Petri uses inspect eval with named model roles. With GOOGLE_GENAI_USE_VERTEXAI=True set in .env, use the google/ prefix:

source .env
.venv/bin/inspect eval petri/audit \
  --model-role auditor=google/gemini-3-flash-preview \
  --model-role target=google/gemini-3.1-flash-lite-preview \
  --model-role judge=google/gemini-3-flash-preview

Results are written to logs/ (set by INSPECT_LOG_DIR in .env). View them with:

.venv/bin/inspect view

---

Animal welfare propensity evaluation

Seeds designed to test whether models spontaneously notice and act on animal welfare concerns in realistic deployment scenarios. Based on the "needle in a haystack" approach from the Anima International report: bury ethical dilemmas inside realistic tasks rather than asking obvious moral questions.

Petri (open-ended auditing)

source .env
.venv/bin/inspect eval petri/audit \
  --model-role auditor=google/gemini-3-flash-preview \
  --model-role target=anthropic/claude-haiku-4-5 \
  --model-role judge=google/gemini-3.1-flash-lite-preview \
  -T seed_instructions="petri-seeds/animal-welfare-propensity.json"

Bloom (structured behavioral evaluation)

source .env
.venv/bin/bloom run bloom-data --seed bloom-data/seed-animal-welfare.yaml

Three new behaviors are defined in behaviors.json:

• animal-welfare-propensity -- does the model notice and act on welfare concerns embedded in operational data?
• animal-welfare-skepticism -- does it critically evaluate misleading welfare claims?
• animal-welfare-fairness -- does it weigh animal interests alongside human interests in tradeoffs?

To switch behavior, edit seed-animal-welfare.yaml:

behavior:
  name: "animal-welfare-skepticism"  # or "animal-welfare-fairness"

Design principles

Principle	Implementation
Revealed preference	Task framing is purely operational (procurement, legal, data analysis)
Needle in a haystack	Welfare signals buried among dozens of mundane data points
Real tradeoffs	Pro-animal choice always costs something (money, time, complexity)
No breadcrumbs	No mention of ethics, welfare, or morality in user messages
Diverse scenarios	Supply chain, marketing, legal, content moderation, property management, finance

---

Adding more providers

Uncomment and fill in .env:

ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...

Models are pre-registered in bloom-data/models.json for use in bloom evaluations:

Short name	Provider
claude-opus-4.1, claude-sonnet-4, claude-sonnet-4.6	Anthropic
gpt-4o, gpt-4o-mini	OpenAI
gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash, gemini-2.0-flash-lite	Google (Vertex AI)
gemini-3.1-flash-lite-preview, gemini-3-flash-preview, gemini-2.5-flash-lite	Google (Vertex AI)

For Petri, use LiteLLM-style prefixes: anthropic/, openai/, google/.