Skip to content

Fix swebenchmultimodal timeout for large repo images #579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
simonrosenberg wants to merge 4 commits into from
Closed

Fix swebenchmultimodal timeout for large repo images #579

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b978383
fix: copy Python 3.13 runtime into eval images for venv compatibility
Mar 26, 2026
dc5060f
fix: use LD_LIBRARY_PATH instead of ldconfig (non-root compatible)
Mar 26, 2026
5020fb3
Increase api_timeout for swebenchmultimodal to 300s
Mar 26, 2026
5158fa3
Merge remote-tracking branch 'origin/fix/copy-python-runtime-for-eval…
Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions benchmarks/swebenchmultimodal/run_infer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,7 @@ def prepare_workspace(
f"(tag prefix: {IMAGE_TAG_PREFIX}, resource_factor: {resource_factor})"
)
startup_timeout = float(os.getenv("REMOTE_RUNTIME_STARTUP_TIMEOUT", "600"))
api_timeout = float(os.getenv("REMOTE_API_TIMEOUT", "300"))
workspace = APIRemoteWorkspace(
runtime_api_url=os.getenv(
"RUNTIME_API_URL", "https://runtime.eval.all-hands.dev"
Expand All @@ -215,6 +216,7 @@ def prepare_workspace(
server_image=agent_server_image,
init_timeout=startup_timeout,
startup_wait_timeout=startup_timeout,
api_timeout=api_timeout,
target_type="source" if "source" in build_target else "binary",
forward_env=forward_env or [],
resource_factor=resource_factor,
Expand Down
10 changes: 10 additions & 0 deletions benchmarks/utils/Dockerfile.agent-layer
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,15 @@ ARG USERNAME=openhands
ARG OPENHANDS_BUILD_GIT_SHA=unknown
ENV OPENHANDS_BUILD_GIT_SHA=${OPENHANDS_BUILD_GIT_SHA}
ENV UV_PYTHON_INSTALL_DIR=/agent-server/uv-managed-python

# Copy the Python 3.13 runtime from the builder so the venv's symlinks resolve.
# Since SDK v1.15.0 the builder venv uses --python-preference only-system,
# meaning .venv/bin/python symlinks to /usr/local/bin/python3.13. The eval-base
# image doesn't have Python 3.13, so we bring the interpreter and stdlib along.
COPY --from=builder /usr/local/bin/python3.13 /usr/local/bin/python3.13
COPY --from=builder /usr/local/lib/python3.13 /usr/local/lib/python3.13
COPY --from=builder /usr/local/lib/libpython3.13* /usr/local/lib/
ENV LD_LIBRARY_PATH=/usr/local/lib

COPY --chown=${USERNAME}:${USERNAME} --from=builder /agent-server /agent-server
ENTRYPOINT ["/agent-server/.venv/bin/python", "-m", "openhands.agent_server"]
Loading