Skip to content

Add environment parameter to preserve env vars in eval-retry (#3312)#3344

Open
evcyen wants to merge 2 commits intoUKGovernmentBEIS:mainfrom
evcyen:feat/3312-preserve-env-variables
Open

Add environment parameter to preserve env vars in eval-retry (#3312)#3344
evcyen wants to merge 2 commits intoUKGovernmentBEIS:mainfrom
evcyen:feat/3312-preserve-env-variables

Conversation

@evcyen
Copy link
Contributor

@evcyen evcyen commented Feb 26, 2026

This PR contains:

  • New features
  • Changes to dev-tools e.g. CI config / github tooling
  • Docs
  • Bug fixes
  • Code refactor

What is the current behavior?

Fixes #3312. Environment variables set during an eval run are not recorded in the eval log. When inspect eval-retry replays a failed eval, it reconstructs the task, model config, sandbox settings, and other parameters from EvalSpec, but environment variables are lost because they were never serialized.

This causes eval-retry to behave differently from the original run when the eval depends on environment variables. For example, if an eval uses os.environ.get("MY_CUSTOM_SETTING") to configure sandbox behavior, the retry will fail or behave incorrectly when that variable is not present in the environment.

There is currently no way to pass environment variables through eval() or eval_set() such that they are persisted in the log and recovered on retry.

What is the new behavior?

Adds an explicit environment parameter to the eval() function that allows users to specify which environment variables should be preserved for retry.

Usage:

import os
from inspect_ai import eval, eval_retry

log = eval(
    my_task(),
    model="openai/gpt-4",
    environment={
        "MY_CUSTOM_SETTING": os.environ["MY_CUSTOM_SETTING"],
        "ANOTHER_VAR": "some_value"
    }
)

Does this PR introduce a breaking change?

No. The environment parameter is optional and defaults to None.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

inspect eval-retry does not preserve environment variables from the original run

1 participant