Sync harbor-adapter-creator and harbor-task-creator with upstream changes (2026-04-19)#6
Open
benediktstroebl wants to merge 2 commits intomainfrom
Open
Sync harbor-adapter-creator and harbor-task-creator with upstream changes (2026-04-19)#6benediktstroebl wants to merge 2 commits intomainfrom
benediktstroebl wants to merge 2 commits intomainfrom
Conversation
…parity runs schema, new reference adapters
… guidance, Step 9 README
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncs two skills with upstream changes merged into harbor-framework/harbor in the last 48 hours, primarily from commit
9ad34d5(Split adapter tutorial to human/ai, update registry handling) and related adapter PRs.Skills updated
harbor-adapter-creator
srclayout. Adapter code now lives undersrc/<adapter_name>/withadapter.py,main.py, andtask-template/(wastemplate/). Addedpyproject.toml,.python-version,__init__.pyto the structure diagram.python3 adapters/mybenchmark/run_adapter.pytouv run python -m <adapter_name>.main, matching the new package-based layout.[task]section withnamefield. The upstream fix (commit9ad34d5) explicitly documents thatnamemust be under[task], not at the TOML top level — tasks without it cannot be registered. Added naming requirements guidance.original_trials/harbor_trialstooriginal_runs/harbor_runsthroughout. Added new required fields:adapted_benchmark_size,parity_benchmark_size,number_of_runs. Updated the field reference table and example. The validator on main now enforces*_runsnaming.added_agentsandparity_unmatching_agentsfields toharbor_adapterentries. Updatedparity_costsformat to include$prefix.cooperbench(multi-agent workflow via messaging/sidecars) andfeaturebench(GPU tasks, comprehensive Docker + Modal example), plusadebench,evoeval,bixbench,financeagent,medagentbench.featurebenchas the canonical GPU adapter example.[task]section and unstable task names.harbor-task-creator
[task]section to the minimal example withname,description, andkeywordsfields. This matches the upstream fix for the task.toml format wherenamemust be under[task], not at the top level.keywords" note — pick 3–8 lowercase tokens covering domain, verifier style, and hardware. Leaving it empty makes the task invisible to registry search.keywords = [], missing[task]section, and stub README.Skills not updated
harbor-cli: No confirmed CLI command changes in the 48-hour window that would affect the reference. Other merged PRs (llmsr-bench adapter, ScienceAgentBench adapter, RExBench Modal support) add new adapters but do not change CLI commands or flags.