Non-agentic evals: parameterize evalchemy sbatch paths by $USER

[mrinal-bespoke]

Problem

The four evalchemy sbatch scripts hardcode /leonardo_work/AIFAC_5C0_290/bfeuer00/...
for every path — conda root, the evalchemy-marin clone, nltk data, HF cache, JIT
caches, and the output root. A second operator cannot run a non-agentic eval at all.

Change

Split paths by whether they are read-only shared runtime or per-user writable:

• Read-only runtime (conda env, evalchemy-marin clone, nltk data) — use the
  submitter's own copy if present, else fall back to the shared bfeuer00 one. A new
  operator does not have to duplicate a ~20 GB conda env to run an eval.
• Writable (HF_HOME/HF_HUB_CACHE, CACHE_ROOT, OUT_ROOT) — always per-user.
• #SBATCH --output uses SLURM's %u.

Each is ${VAR:-<default>}, so an explicit env var still wins.

This is a no-op for the original operator: with USER=bfeuer00 every path resolves
to the same byte-identical string as before.

Notable fixes

Conda fallback must validate the env, not the root. Testing only for a miniforge3
directory is not enough — an operator can have a partial install (e.g. otagent but not
evalchemy-marin) and would then fail with EnvironmentNameNotFound: evalchemy-marin
instead of falling back to the shared root. The check is now for
<root>/envs/evalchemy-marin.

--output directories must pre-exist. SLURM opens the log file before the job script
runs and does not create intermediate directories, so a new operator gets an instant
"Unable to open file" that reads like a launch bug rather than a missing mkdir. The script
cannot self-heal this — its own mkdir -p runs far too late — so the one-time setup is
documented inline next to each path.

Reviewer note

With HF_HUB_OFFLINE=1 and HF_HOME now per-user, a new operator's cache starts empty,
so models will not resolve until they pre-cache or symlink them in. That is inherent to
making the cache writable (one shared HF cache across operators is not safe), and the
symlink workaround is documented in the diff — but it is a genuine behaviour change for a
new user.

Testing

• bash -n clean on all four scripts.
• Path resolution simulated for three cases: original operator (all-own, byte-identical),
  new operator (shared fallback for runtime, per-user for writable), and partial install
  (correctly prefers shared over an incomplete own root).
• Comments-only in the final commit.

🤖 Generated with Claude Code