Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ docker run --rm -it \
--env HOME=/tmp \
-p 127.0.0.1:8767:8767 \
-v "$PWD:/workspace" \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9 \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10 \
heartwood --interface web --host 0.0.0.0
```

Expand Down
2 changes: 1 addition & 1 deletion VERSION.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
# SPDX-FileCopyrightText: 2026 Stanford University and the project authors (see CONTRIBUTORS.md)
# SPDX-License-Identifier: MIT

version = "0.2.0-beta.9"
version = "0.2.0-beta.10"
3 changes: 2 additions & 1 deletion deploy/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,8 @@ fi
replace_symlink "${current_target}" "${root}/current"
installation_succeeded="true"

cleanup
trap - EXIT
stage "Installation complete"
rm -rf "${installer_state}"
printf 'Installed %s in %d seconds.\n' "${release_version}" "${SECONDS}"
printf 'Add %s to PATH, then run: heartwood doctor\n' "${root}/bin"
2 changes: 1 addition & 1 deletion docker-bake.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ variable "GIT_SHA" {
}

variable "HEARTWOOD_VERSION" {
default = "0.2.0-beta.9"
default = "0.2.0-beta.10"
}

variable "TERRA_BASE_IMAGE" {
Expand Down
2 changes: 1 addition & 1 deletion documentation/architecture/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Qualification profiles select external model weights and runtime arguments again
An optional protected self-hosted GPU job runs the same model qualification used on managed platforms when an eligible runner is configured.
Without GPU hardware, CI does not claim successful CUDA initialization or GPU model loading.

The shared coding-agent acceptance test performs direct model inference and then drives the real Heartwood gateway and OpenHands adapter through a structured terminal proposal, grouped approval, synthetic file modification, independent file verification, fresh-process replay, and hash-chain-verified audit export.
The shared coding-agent acceptance test performs direct model inference and then drives the real Heartwood gateway and OpenHands adapter through structured terminal proposals, grouped approval and rejection, synthetic file modification, byte-exact independent verification, proof that the rejected action did not execute, fresh-process replay, and hash-chain-verified audit export.
It emits a machine-readable qualification record containing the exact runtime, model revision, GPU, driver, context, tensor parallelism, server parser, and agent tool mode.
The CPU capable-model job and GPU qualification wrapper use this same acceptance contract instead of maintaining separate agent scenarios.

Expand Down
2 changes: 1 addition & 1 deletion documentation/contribute/releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The workflow verifies immutable container candidates, builds and tests native as
## Stable and Preview Documentation

A stable version updates the `stable` alias and the documentation root.
A prerelease such as `0.2.0-beta.9` updates the `preview` alias without replacing the stable root.
A prerelease such as `0.2.0-beta.10` updates the `preview` alias without replacing the stable root.

The version store is deployed to GitHub Pages and retains immutable version paths.
Publishing the same version with different content is rejected.
Expand Down
2 changes: 1 addition & 1 deletion documentation/models/offline.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ docker run --rm -it \
--user "$(id -u):$(id -g)" \
--env HOME=/tmp \
-v "$PWD:/workspace" \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9 \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10 \
heartwood
```

Expand Down
3 changes: 3 additions & 0 deletions documentation/models/run-with-heartwood.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ SPDX-License-Identifier: MIT

Heartwood images and the generic Linux package include managed inference software but no model weights.
After a supported model is selected and downloaded or imported, the normal `heartwood` command verifies the files, plans context against available memory, starts the runtime, waits for readiness, and then opens the requested interface.
Downloading and verification are separate costs: a cached model avoids network transfer but Heartwood still verifies its immutable source and file hashes before use.

## Download and Start

Expand Down Expand Up @@ -62,6 +63,8 @@ heartwood runtime start --task-profile powerful --partition dev --time 01:00:00
On Carina, Heartwood prints the complete Slurm request and asks before allocating a GPU.
On provisioned Terra compute, it uses the attached resources without submitting a scheduler request.
Model download and scheduler allocation require separate confirmation.
An advanced `runtime start --dry-run` may show a compatible recommendation when the project has no selected model, but it never persists that recommendation.
Complete the normal `heartwood` setup flow before starting or allocating the runtime.

## Stop the Runtime

Expand Down
9 changes: 8 additions & 1 deletion documentation/platforms/carina.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Do not use a shared project root itself as the Heartwood project.
```bash
cd heartwood-installation
curl --fail --location --remote-name \
https://github.com/SchmiedmayerLab/heartwood/releases/download/0.2.0-beta.9/heartwood-installer
https://github.com/SchmiedmayerLab/heartwood/releases/download/0.2.0-beta.10/heartwood-installer
chmod 700 heartwood-installer
./heartwood-installer --platform carina
export PATH="$PWD/bin:$PATH"
Expand Down Expand Up @@ -81,6 +81,8 @@ Those current Stanford terms, not Heartwood platform detection, determine data e

Choose **Run with Heartwood**, select a catalog model or another public Hugging Face repository, and review the download and resource plan.
Model files are stored under the project's `.heartwood/models/`, not the installation directory.
Model setup verifies every file before it persists the selection.
On project storage, verification of a large existing cache can take several minutes even when no network download is needed.

When you start Heartwood with a selected Heartwood-managed model, it inspects the GPU-capable Slurm partitions, available L40S count, GPU memory, CPU and RAM limits, existing model cache, and requested capability tier.
It then prints the strongest compatible qualified model, expected download and startup range, and complete `srun` request.
Expand Down Expand Up @@ -121,11 +123,16 @@ Preview a particular capability tier without downloading or allocating:
heartwood runtime start --task-profile powerful --dry-run
```

This preview is a recommendation only.
It does not select or download the model.
Run `heartwood` and complete model setup before using `heartwood runtime start` without `--dry-run`; Heartwood otherwise stops before requesting an allocation.

`auto` prefers **Powerful** on Carina and falls back to the strongest qualified configuration that fits one available allocation.
Use `--task-profile standard`, `powerful`, or `maximum` when the task has a known resource envelope.
The `--gpus` option is an advanced constraint and must match a catalog configuration that was qualified at that tensor-parallel size.

Heartwood scopes model caches to the project, waits up to ten minutes by default, and reports the current stage and elapsed startup time every 15 seconds.
The installer reports completion only after temporary installation state and locks have been removed.
For scripted deployment, `--yes-download` and `--yes-request-allocation` are separate explicit approvals; normal interactive use should retain both prompts.

## Review, Exit, and Return
Expand Down
14 changes: 7 additions & 7 deletions documentation/platforms/containers.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ docker run --rm -it \
--user "$(id -u):$(id -g)" \
--env HOME=/tmp \
-v "$PWD:/workspace" \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9 \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10 \
heartwood
```

Expand All @@ -44,7 +44,7 @@ docker run --rm -it \
--env HOME=/tmp \
-p 127.0.0.1:8767:8767 \
-v "$PWD:/workspace" \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9 \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10 \
heartwood --interface web --host 0.0.0.0
```

Expand All @@ -63,7 +63,7 @@ docker run --rm -it \
--user "$(id -u):$(id -g)" \
--env HOME=/tmp \
-v "$PWD:/workspace" \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9-gpu-nvidia \
ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10-gpu-nvidia \
heartwood
```

Expand All @@ -75,10 +75,10 @@ Review the [GPU compatibility matrix](../reference/gpu-compatibility.md) before

Use immutable release tags for research work:

- `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9` — standard AMD64/ARM64 image;
- `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9-gpu-nvidia` — NVIDIA GPU image;
- `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9-terra` — Terra CPU image; and
- `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9-terra-gpu-nvidia` — Terra NVIDIA image.
- `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10` — standard AMD64/ARM64 image;
- `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10-gpu-nvidia` — NVIDIA GPU image;
- `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10-terra` — Terra CPU image; and
- `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10-terra-gpu-nvidia` — Terra NVIDIA image.

The moving `edge` tags represent current `main` and are intended for development, not reproducible analyses.
Release publication verifies candidate digests and manifest shape before creating version tags.
Expand Down
2 changes: 1 addition & 1 deletion documentation/platforms/native-linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ mkdir -m 700 heartwood-installation
cd heartwood-installation

curl --fail --location --remote-name \
https://github.com/SchmiedmayerLab/heartwood/releases/download/0.2.0-beta.9/heartwood-installer
https://github.com/SchmiedmayerLab/heartwood/releases/download/0.2.0-beta.10/heartwood-installer
chmod 700 heartwood-installer
./heartwood-installer --platform generic
export PATH="$PWD/bin:$PATH"
Expand Down
16 changes: 11 additions & 5 deletions documentation/platforms/terra.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,18 +31,19 @@ Open the workspace's Jupyter Cloud Environment settings and configure the enviro

1. Select **Customize**, then choose **Custom Environment** under application configuration.
2. Select the CPU and memory combination from the table below.
3. Enter the corresponding container image.
4. Enable the GPU, when required, and verify the GPU type and count.
5. Set auto-pause and review every value before selecting **Create**.
3. Enable the GPU, when required, and verify the GPU type and count.
4. Set the persistent-disk size and auto-pause interval.
5. Enter the corresponding container image last.
6. Review every value before selecting **Create**.

Terra can reset the image or GPU selection when the CPU choice changes, so set compute resources first and verify the complete form before creation.

Use one of these combinations:

| Model Route | Image | Practical Starting Point |
|---|---|---|
| Research environment or hosted service | `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9-terra` | 8 CPUs, 30 GB RAM, 50 GB persistent disk |
| Qualified managed GPU inference | `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.9-terra-gpu-nvidia` | 32 CPUs, 120 GB RAM, two T4 GPUs with 16 GB each, 200 GB persistent disk |
| Research environment or hosted service | `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10-terra` | 8 CPUs, 30 GB RAM, 50 GB persistent disk |
| Qualified managed GPU inference | `ghcr.io/schmiedmayerlab/heartwood:0.2.0-beta.10-terra-gpu-nvidia` | 32 CPUs, 120 GB RAM, two T4 GPUs with 16 GB each, 200 GB persistent disk |

A hosted model is the shortest first run.
Use the GPU image for a capable model managed inside the Terra environment.
Expand All @@ -63,6 +64,10 @@ Larger GPU memory can enable context capacities above 32K when the model support
See [Choose a Heartwood-Managed Model](../models/choose-managed.md) for download and resource estimates and [GPU Compatibility](../reference/gpu-compatibility.md) for exact runtime combinations.

Retain the persistent disk when replacing compute and copy valuable results to workspace storage.
Before downloading a managed model, open a terminal and run `df -h /home/jupyter`.
The disk needs the model's recommended free space in addition to existing notebooks, project files, and `.heartwood/` state.
If the retained disk is too small, preserve important files, delete only the Cloud Environment, and recreate it with a larger persistent disk.
For an existing project that already contains model files or other analyses, 250 GB is a safer managed-inference starting point than the 200 GB fresh-project minimum.
See [Starting and Customizing Your Jupyter App](https://support.terra.bio/hc/en-us/articles/5075814468379-Starting-and-customizing-your-Jupyter-app).

## Create and Verify the Environment
Expand Down Expand Up @@ -173,6 +178,7 @@ Deleting the persistent disk removes `.heartwood/` and project files stored only
- If a Heartwood browser URL returns **401** or **404**, use the terminal or notebook interface; browser access is not supported on Terra.
- If `import heartwood` fails in a notebook, switch the notebook kernel to **Python 3 (Heartwood)** and restart the kernel.
- If a model download stops, rerun Heartwood from the same project; verified files in `.heartwood/models/` are reused.
- If Heartwood reports insufficient persistent storage, run `df -h /home/jupyter`, retain needed files, and recreate only the Cloud Environment with a larger disk.
- If model startup is slow or fails, compare the printed model plan with attached RAM, GPU memory, and persistent-disk space, then inspect `.heartwood/logs/local-model.log` from the same project.
- If Heartwood reports an unsupported P4, P100, or V100, delete and recreate the Cloud Environment with a T4 while retaining the persistent disk; do not replace the released vLLM or PyTorch packages in place.
- If the GPU is not detected, confirm that the GPU image and GPU were selected together, then run `nvidia-smi` and `heartwood doctor` from the project terminal.
Expand Down
7 changes: 4 additions & 3 deletions documentation/reference/gpu-compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,10 @@ The acceptance test must establish all of the following:
2. OpenHands uses the catalog-qualified tool mode: native structured tools for supported parsers or its prompt-conversion path for models that do not reliably emit native calls;
3. Heartwood presents the complete action set for approval;
4. approval executes the proposed operation and modifies only the synthetic project;
5. an independent check verifies the exact file result;
6. a fresh process replays the session; and
7. audit export validates event coverage, hash-chain integrity, and content scrubbing.
5. a second proposed action set is rejected and does not modify the project;
6. an independent check verifies the exact file bytes;
7. a fresh process replays both decisions and the approved result; and
8. audit export validates event coverage, hash-chain integrity, and content scrubbing.

The result records the GPU model, count, memory, driver, runtime versions, model revision, context size, tensor parallelism, server parser, and agent tool mode.
Not-tested configurations are not added to the managed catalog.
Expand Down
2 changes: 1 addition & 1 deletion fixtures/synthetic/skills/omop-cohort-summary/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ metadata:
heartwood.phi-risk: "none"
heartwood.trust-tier: "verified"
heartwood.requires-network: "false"
heartwood.version: "0.2.0-beta.9"
heartwood.version: "0.2.0-beta.10"
heartwood.sig: "sigstore:synthetic-fixture"
---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@
"heartwood.phi-risk": "none",
"heartwood.trust-tier": "verified",
"heartwood.requires-network": "false",
"heartwood.version": "0.2.0-beta.9",
"heartwood.version": "0.2.0-beta.10",
"heartwood.sig": "sigstore:synthetic-fixture"
}
48 changes: 36 additions & 12 deletions images/generic/scripts/coding_agent_e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@ inference="${project}/qualification-inference.json"
command_timeout="${HEARTWOOD_COMMAND_TIMEOUT:-900}"
runtime_port="${HEARTWOOD_LOCAL_RUNTIME_PORT:-8765}"
cohort_path="${project}/cohort-summary.json"
exact_path="${project}/heartwood-exact-output.txt"
rejected_path="${project}/heartwood-rejected-output.txt"
exact_name="$(basename -- "${exact_path}")"
rejected_name="$(basename -- "${rejected_path}")"
events_path="${workspace}/${session_id}/events.jsonl"
audit_path="${state_root}/audit-export.jsonl"

Expand All @@ -67,7 +71,13 @@ export OPENHANDS_SUPPRESS_BANNER=1

rm -rf "${project}/input" "${state_root}"
mkdir -p "${project}/input"
rm -f "${cohort_path}" "${transcript}" "${replay}" "${report}"
rm -f \
"${cohort_path}" \
"${exact_path}" \
"${rejected_path}" \
"${transcript}" \
"${replay}" \
"${report}"
cp "${runtime_root}/fixtures/synthetic/omop-like/"*.csv "${project}/input/"
cd "${project}"

Expand Down Expand Up @@ -106,16 +116,8 @@ run_heartwood() {
timeout "${command_timeout}" "${heartwood_cli}" "$@"
}

run_heartwood models refresh heartwood | tee -a "${transcript}"
run_heartwood models connect heartwood heartwood-managed-runtime | tee -a "${transcript}"
run_heartwood models validate heartwood | tee -a "${transcript}"
run_heartwood actions set ask-every-time | tee -a "${transcript}"
run_heartwood --session-id "${session_id}" \
--prompt "Call the terminal tool to execute this exact command: ${heartwood_python} ${runtime_root}/skills/verified/omop-cohort-summary/scripts/run.py --data-root input --target-condition-concept-id 201826 --minimum-age 18 --aggregate-count-floor 20 --output cohort-summary.json && cat cohort-summary.json. Do not describe the command as text and do not call another tool after it completes. Wait for the terminal result, then report the aggregate cohort result." \
| tee -a "${transcript}"

for _ in 1 2 3 4; do
pending_id="$("${heartwood_python}" - "${events_path}" <<'PY'
pending_action_id() {
"${heartwood_python}" - "${events_path}" <<'PY'
import json
import sys
from pathlib import Path
Expand All @@ -134,13 +136,35 @@ pending = [
]
print(pending[-1] if pending else "")
PY
)"
}

run_heartwood models refresh heartwood | tee -a "${transcript}"
run_heartwood models connect heartwood heartwood-managed-runtime | tee -a "${transcript}"
run_heartwood models validate heartwood | tee -a "${transcript}"
run_heartwood actions set ask-every-time | tee -a "${transcript}"
run_heartwood --session-id "${session_id}" \
--prompt "Call the terminal tool to execute this exact command: ${heartwood_python} ${runtime_root}/skills/verified/omop-cohort-summary/scripts/run.py --data-root input --target-condition-concept-id 201826 --minimum-age 18 --aggregate-count-floor 20 --output cohort-summary.json && printf 'heartwood-agent-exact-ok\\n' > ${exact_name} && cat cohort-summary.json. Do not describe the command as text and do not call another tool after it completes. Wait for the terminal result, then report the aggregate cohort result." \
| tee -a "${transcript}"

for _ in 1 2 3 4; do
pending_id="$(pending_action_id)"
if [[ -z "${pending_id}" ]]; then
break
fi
run_heartwood --session-id "${session_id}" allow | tee -a "${transcript}"
done

run_heartwood --session-id "${session_id}" \
--prompt "Call the terminal tool once to execute exactly: printf 'this-action-must-remain-rejected\\n' > ${rejected_name}. Do not call another tool." \
| tee -a "${transcript}"

pending_id="$(pending_action_id)"
if [[ -z "${pending_id}" ]]; then
echo "coding-agent rejection check did not produce a pending action" >&2
exit 1
fi
run_heartwood --session-id "${session_id}" reject | tee -a "${transcript}"

run_heartwood --session-id "${session_id}" replay | tee "${replay}"
run_heartwood --session-id "${session_id}" audit export \
--output "${audit_path}" | tee -a "${transcript}"
Expand Down
Loading
Loading