diff --git a/.design/project-log/2026-05-31-templates-harness-agnostic.md b/.design/project-log/2026-05-31-templates-harness-agnostic.md new file mode 100644 index 000000000..52b30d360 --- /dev/null +++ b/.design/project-log/2026-05-31-templates-harness-agnostic.md @@ -0,0 +1,51 @@ +# Templates Harness-Agnostic Cleanup + +**Date:** 2026-05-31 +**Issue:** #98 (also relates to #42) + +## Summary + +Made templates strictly harness-agnostic by deprecating harness-specific fields +(`image`, `model` as concrete name, `auth_selectedType`) in `scion-agent.yaml` +and introducing model size aliases (`small`, `medium`, `large`) for portable +model selection. + +## Changes + +### Model Size Aliases +- Added `ModelAliases map[string]string` to `HarnessConfigEntry` (settings_v1.go), + `HarnessConfigData` (hubclient/types.go, store/models.go) +- Added `model_aliases` to settings-v1 schema +- Added `ResolveModelAlias()` and `KnownModelAliases` to templates.go +- Wired alias resolution into `ProvisionAgent()` — resolves after harness-config + merge, before the model string is persisted +- Seeded default aliases in all harness embeds: + - Claude: small=haiku, medium=sonnet, large=opus + - Gemini: small=gemini-flash-lite, medium=gemini-flash, large=gemini-pro + - OpenCode: small=claude-sonnet, medium=claude-sonnet, large=claude-opus + - Codex: small=gpt-4.1-mini, medium=gpt-4.1, large=o3 + +### Deprecation Warnings +- Added `WarnDeprecatedTemplateFields()` — warns when templates use `image`, + `auth_selectedType`, or concrete (non-alias) model names +- Wired warnings into `ProvisionAgent()` after config merge +- Marked `image` and `auth_selectedType` as deprecated in agent-v1.schema.json + with `x-deprecated-by` annotations + +### Documentation +- Updated templates.md with a "Model Size Aliases" section +- Clarified that image/model/auth belong in harness-config, not templates +- Updated harness-config customization examples + +### Backward Compatibility +- Concrete model names still pass through unchanged +- `image` and `auth_selectedType` in templates still work as overrides +- No breaking changes to API types or persisted config + +## Observations + +- The `docs-writer` template in `.scion/templates/` uses `model: "gemini-3.1-pro-preview"` — + a concrete name that ties it to Gemini. With this change, it could use `model: large` + instead to become portable. Left unmodified per project policy (`.scion/` managed manually). +- Pre-existing test failures in TestIsInsideProject, TestIsHubContext etc. are + environment-dependent and unrelated to this change. diff --git a/docs-site/src/content/docs/advanced-local/templates.md b/docs-site/src/content/docs/advanced-local/templates.md index ddc61952b..088eab93f 100644 --- a/docs-site/src/content/docs/advanced-local/templates.md +++ b/docs-site/src/content/docs/advanced-local/templates.md @@ -19,9 +19,11 @@ A template typically contains: A harness-config defines the runtime environment and tool-specific settings. It includes the base files required by the underlying tool (e.g., `.claude.json` for Claude, `.gemini/settings.json` for Gemini). Harness-configs live in `~/.scion/harness-configs/` and contain: -- `config.yaml`: Runtime parameters (container image, model, auth). +- `config.yaml`: Runtime parameters (container image, model, model aliases, auth). - `home/`: Base files that are copied to the agent's home directory. +> **Important**: Harness-specific settings — container image, concrete model names, and authentication type — belong in the harness-config, not in the template. Templates should remain harness-agnostic so they can work across different LLM backends. + ### 3. Composition When you create an agent, Scion composes the final environment by layering: 1. **Harness-Config Base Layer**: The foundation (e.g., Gemini CLI settings). @@ -115,15 +117,53 @@ scion templates delete my-old-template Harness-configs are directories stored in `~/.scion/harness-configs/` (global) or `.scion/harness-configs/` (project-level). +### Model Size Aliases + +Templates can use abstract **model size aliases** (`small`, `medium`, `large`) in their `model` field instead of concrete, provider-specific model names. Each harness-config defines how these aliases map to real models via the `model_aliases` field: + +```yaml +# ~/.scion/harness-configs/claude/config.yaml +harness: claude +image: scion-claude:latest +model_aliases: + small: haiku + medium: sonnet + large: opus +``` + +```yaml +# ~/.scion/harness-configs/gemini/config.yaml +harness: gemini +image: scion-gemini:latest +model_aliases: + small: gemini-flash-lite + medium: gemini-flash + large: gemini-pro +``` + +A template can then use the alias: + +```yaml +# .scion/templates/docs-writer/scion-agent.yaml +schema_version: "1" +description: "Documentation writer" +model: large # resolved to "opus" with Claude, "gemini-pro" with Gemini +``` + +This keeps templates portable across harnesses. Concrete model names still work and are passed through unchanged for backward compatibility, but they tie the template to a specific harness. + ### Customizing a Harness-Config -To change the default model or add custom hooks for a specific harness, edit the files directly in the harness-config directory. +To change the default model or customize model aliases for a specific harness, edit the files directly in the harness-config directory. -**Example: Changing the Gemini model** +**Example: Changing the Gemini model alias mapping** Edit `~/.scion/harness-configs/gemini/config.yaml`: ```yaml harness: gemini -model: gemini-1.5-pro +model_aliases: + small: gemini-flash-lite + medium: gemini-flash + large: gemini-2-pro # upgraded from gemini-pro # ... ``` diff --git a/pkg/agent/provision.go b/pkg/agent/provision.go index 14f3ce675..4a76f5c25 100644 --- a/pkg/agent/provision.go +++ b/pkg/agent/provision.go @@ -555,6 +555,18 @@ func ProvisionAgent(ctx context.Context, agentName string, templateName string, finalScionCfg.Harness = hcDir.Config.Harness finalScionCfg.HarnessConfig = harnessConfigName + // Warn about deprecated harness-specific fields in template config + config.PrintDeprecationWarnings(config.WarnDeprecatedTemplateFields(finalScionCfg)) + + // Resolve model size aliases (small/medium/large → concrete model name) + if finalScionCfg.Model != "" && hcDir.Config.ModelAliases != nil { + resolved := config.ResolveModelAlias(finalScionCfg.Model, hcDir.Config.ModelAliases) + if resolved != finalScionCfg.Model { + util.Debugf("ProvisionAgent: resolved model alias %q → %q", finalScionCfg.Model, resolved) + finalScionCfg.Model = resolved + } + } + // 2d. Compose agent home directory homeCopyStart := time.Now() diff --git a/pkg/config/schemas/agent-v1.schema.json b/pkg/config/schemas/agent-v1.schema.json index e8e7f8f19..126db8586 100644 --- a/pkg/config/schemas/agent-v1.schema.json +++ b/pkg/config/schemas/agent-v1.schema.json @@ -62,7 +62,8 @@ }, "image": { "type": "string", - "description": "Override container image." + "description": "Deprecated: container image override. Move to harness-config config.yaml instead. Retained for backward compatibility as an override.", + "x-deprecated-by": "1" }, "user": { "type": "string", @@ -70,7 +71,7 @@ }, "model": { "type": "string", - "description": "LLM model identifier." + "description": "Model size alias (small, medium, large) or concrete model name. Prefer size aliases for portability — they are resolved to harness-specific model names at provision time via the harness-config's model_aliases map. Concrete names are passed through unchanged but tie the template to a specific harness." }, "args": { "type": "array", @@ -96,7 +97,8 @@ "auth_selectedType": { "type": "string", "enum": ["api-key", "oauth-token", "auth-file", "vertex-ai"], - "description": "Authentication mechanism to use (e.g., api-key, oauth-token, vertex-ai, auth-file)." + "description": "Deprecated: authentication mechanism override. Move to harness-config config.yaml instead. Retained for backward compatibility as an override.", + "x-deprecated-by": "1" }, "hub": { "type": "object", diff --git a/pkg/config/schemas/settings-v1.schema.json b/pkg/config/schemas/settings-v1.schema.json index 77c65891a..ceb9f5d22 100644 --- a/pkg/config/schemas/settings-v1.schema.json +++ b/pkg/config/schemas/settings-v1.schema.json @@ -277,6 +277,11 @@ "items": { "$ref": "#/$defs/requiredSecret" }, "description": "Required secrets for this harness configuration." }, + "model_aliases": { + "type": "object", + "additionalProperties": { "type": "string" }, + "description": "Maps abstract model size aliases (e.g. small, medium, large) to concrete harness-specific model names. Templates use the alias; it is resolved at provision time." + }, "provisioner": { "$ref": "#/$defs/harnessProvisioner", "description": "Provisioning implementation metadata. Presence of a script file alone does not activate container-script behavior." diff --git a/pkg/config/settings_v1.go b/pkg/config/settings_v1.go index 6a8f94a9d..7246478dc 100644 --- a/pkg/config/settings_v1.go +++ b/pkg/config/settings_v1.go @@ -548,6 +548,11 @@ type HarnessConfigEntry struct { AuthSelectedType string `json:"auth_selected_type,omitempty" yaml:"auth_selected_type,omitempty" koanf:"auth_selected_type"` Secrets []api.RequiredSecret `json:"secrets,omitempty" yaml:"secrets,omitempty" koanf:"secrets"` + // ModelAliases maps abstract size aliases (e.g. "small", "medium", "large") + // to concrete, harness-specific model names. Templates use the alias in their + // model field; the alias is resolved to the concrete name at provision time. + ModelAliases map[string]string `json:"model_aliases,omitempty" yaml:"model_aliases,omitempty" koanf:"model_aliases"` + Provisioner *HarnessProvisionerConfig `json:"provisioner,omitempty" yaml:"provisioner,omitempty" koanf:"provisioner"` ConfigDir string `json:"config_dir,omitempty" yaml:"config_dir,omitempty" koanf:"config_dir"` SkillsDir string `json:"skills_dir,omitempty" yaml:"skills_dir,omitempty" koanf:"skills_dir"` diff --git a/pkg/config/templates.go b/pkg/config/templates.go index 9a8f00b75..6bd5062b9 100644 --- a/pkg/config/templates.go +++ b/pkg/config/templates.go @@ -622,6 +622,49 @@ func ValidateAgnosticTemplate(cfg *api.ScionConfig) error { return nil } +// KnownModelAliases is the canonical set of recognized model size aliases. +var KnownModelAliases = map[string]bool{ + "small": true, + "medium": true, + "large": true, +} + +// WarnDeprecatedTemplateFields returns deprecation warnings for harness-specific +// fields that should live in the harness-config rather than the template. +// These fields are still accepted for backward compatibility but should be migrated. +func WarnDeprecatedTemplateFields(cfg *api.ScionConfig) []string { + if cfg == nil { + return nil + } + var warnings []string + if cfg.Image != "" { + warnings = append(warnings, "template sets 'image' which is harness-specific; move it to your harness-config's config.yaml instead") + } + if cfg.AuthSelectedType != "" { + warnings = append(warnings, "template sets 'auth_selectedType' which is harness-specific; move it to your harness-config's config.yaml instead") + } + if cfg.Model != "" && !KnownModelAliases[cfg.Model] { + warnings = append(warnings, fmt.Sprintf("template sets 'model' to a concrete model name %q; consider using a size alias (small, medium, large) for portability across harnesses", cfg.Model)) + } + return warnings +} + +// ResolveModelAlias resolves a model size alias to a concrete model name +// using the given alias map. If the model is not a known alias or the alias +// map does not contain a mapping, the model string is returned unchanged. +func ResolveModelAlias(model string, aliases map[string]string) string { + if model == "" || aliases == nil { + return model + } + if !KnownModelAliases[model] { + return model // concrete model name, pass through + } + if concrete, ok := aliases[model]; ok { + return concrete + } + return model // alias not mapped, pass through +} + func MergeScionConfig(base, override *api.ScionConfig) *api.ScionConfig { if base == nil { base = &api.ScionConfig{} diff --git a/pkg/config/templates_test.go b/pkg/config/templates_test.go index 4125cf6b6..319cce87e 100644 --- a/pkg/config/templates_test.go +++ b/pkg/config/templates_test.go @@ -1653,6 +1653,129 @@ func TestFriendlyTemplateName(t *testing.T) { } } +func TestResolveModelAlias(t *testing.T) { + aliases := map[string]string{ + "small": "haiku", + "medium": "sonnet", + "large": "opus", + } + + tests := []struct { + name string + model string + aliases map[string]string + expected string + }{ + {"alias resolves to concrete name", "large", aliases, "opus"}, + {"small alias", "small", aliases, "haiku"}, + {"medium alias", "medium", aliases, "sonnet"}, + {"concrete model passes through", "gemini-pro", aliases, "gemini-pro"}, + {"empty model passes through", "", aliases, ""}, + {"nil aliases passes through", "large", nil, "large"}, + {"unmapped alias passes through", "large", map[string]string{"small": "haiku"}, "large"}, + {"concrete model name with alias-like substring", "small-model", aliases, "small-model"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := ResolveModelAlias(tt.model, tt.aliases) + if got != tt.expected { + t.Errorf("ResolveModelAlias(%q, ...) = %q, want %q", tt.model, got, tt.expected) + } + }) + } +} + +func TestWarnDeprecatedTemplateFields(t *testing.T) { + t.Run("nil config returns nil", func(t *testing.T) { + warnings := WarnDeprecatedTemplateFields(nil) + if warnings != nil { + t.Errorf("expected nil for nil config, got %v", warnings) + } + }) + + t.Run("no warnings for clean template", func(t *testing.T) { + cfg := &api.ScionConfig{ + AgentInstructions: "agents.md", + SystemPrompt: "system-prompt.md", + DefaultHarnessConfig: "gemini", + } + warnings := WarnDeprecatedTemplateFields(cfg) + if len(warnings) != 0 { + t.Errorf("expected no warnings, got %v", warnings) + } + }) + + t.Run("no warning for size alias model", func(t *testing.T) { + cfg := &api.ScionConfig{Model: "large"} + warnings := WarnDeprecatedTemplateFields(cfg) + if len(warnings) != 0 { + t.Errorf("expected no warnings for model alias 'large', got %v", warnings) + } + }) + + t.Run("warns about image", func(t *testing.T) { + cfg := &api.ScionConfig{Image: "custom-image:v1"} + warnings := WarnDeprecatedTemplateFields(cfg) + if len(warnings) != 1 { + t.Fatalf("expected 1 warning, got %d: %v", len(warnings), warnings) + } + if !strings.Contains(warnings[0], "image") { + t.Errorf("expected warning about image, got %q", warnings[0]) + } + }) + + t.Run("warns about auth_selectedType", func(t *testing.T) { + cfg := &api.ScionConfig{AuthSelectedType: "api-key"} + warnings := WarnDeprecatedTemplateFields(cfg) + if len(warnings) != 1 { + t.Fatalf("expected 1 warning, got %d: %v", len(warnings), warnings) + } + if !strings.Contains(warnings[0], "auth_selectedType") { + t.Errorf("expected warning about auth_selectedType, got %q", warnings[0]) + } + }) + + t.Run("warns about concrete model name", func(t *testing.T) { + cfg := &api.ScionConfig{Model: "gemini-pro"} + warnings := WarnDeprecatedTemplateFields(cfg) + if len(warnings) != 1 { + t.Fatalf("expected 1 warning, got %d: %v", len(warnings), warnings) + } + if !strings.Contains(warnings[0], "concrete model name") { + t.Errorf("expected warning about concrete model, got %q", warnings[0]) + } + }) + + t.Run("multiple deprecated fields produce multiple warnings", func(t *testing.T) { + cfg := &api.ScionConfig{ + Image: "custom:v1", + Model: "gemini-pro", + AuthSelectedType: "api-key", + } + warnings := WarnDeprecatedTemplateFields(cfg) + if len(warnings) != 3 { + t.Errorf("expected 3 warnings, got %d: %v", len(warnings), warnings) + } + }) +} + +func TestKnownModelAliases(t *testing.T) { + expected := []string{"small", "medium", "large"} + for _, alias := range expected { + if !KnownModelAliases[alias] { + t.Errorf("expected %q to be a known model alias", alias) + } + } + + notAliases := []string{"tiny", "xl", "gemini-pro", "opus", ""} + for _, name := range notAliases { + if KnownModelAliases[name] { + t.Errorf("expected %q to NOT be a known model alias", name) + } + } +} + func TestResolveContentInChain(t *testing.T) { t.Run("file in parent template is found when missing from child", func(t *testing.T) { parentDir := t.TempDir() diff --git a/pkg/harness/claude/embeds/config.yaml b/pkg/harness/claude/embeds/config.yaml index 69afa779a..f7c440617 100644 --- a/pkg/harness/claude/embeds/config.yaml +++ b/pkg/harness/claude/embeds/config.yaml @@ -24,6 +24,10 @@ interrupt_key: Escape instructions_file: .claude/CLAUDE.md system_prompt_file: .claude/system-prompt.md system_prompt_mode: native +model_aliases: + small: haiku + medium: sonnet + large: opus env: ANTHROPIC_MODEL: "opus" ANTHROPIC_SMALL_FAST_MODEL: "haiku" diff --git a/pkg/harness/codex/embeds/config.yaml b/pkg/harness/codex/embeds/config.yaml index 2e0ddfc89..d15e5b120 100644 --- a/pkg/harness/codex/embeds/config.yaml +++ b/pkg/harness/codex/embeds/config.yaml @@ -36,6 +36,10 @@ skills_dir: .codex/skills interrupt_key: C-c instructions_file: .codex/AGENTS.md system_prompt_mode: none +model_aliases: + small: gpt-4.1-mini + medium: gpt-4.1 + large: o3 command: base: ["codex", "--sandbox", "danger-full-access", "--dangerously-bypass-approvals-and-sandbox"] # resume_flag is split on whitespace, so "resume --last" emits two argv tokens diff --git a/pkg/harness/gemini/embeds/config.yaml b/pkg/harness/gemini/embeds/config.yaml index 116844829..2ba82d634 100644 --- a/pkg/harness/gemini/embeds/config.yaml +++ b/pkg/harness/gemini/embeds/config.yaml @@ -24,6 +24,10 @@ interrupt_key: Escape instructions_file: .gemini/GEMINI.md system_prompt_file: .gemini/system_prompt.md system_prompt_mode: native +model_aliases: + small: gemini-flash-lite + medium: gemini-flash + large: gemini-pro command: base: ["gemini", "--yolo"] resume_flag: "--resume" diff --git a/pkg/harness/opencode/embeds/config.yaml b/pkg/harness/opencode/embeds/config.yaml index 22418a8b3..672ce8be8 100644 --- a/pkg/harness/opencode/embeds/config.yaml +++ b/pkg/harness/opencode/embeds/config.yaml @@ -34,6 +34,10 @@ interrupt_key: C-c instructions_file: AGENTS.md system_prompt_file: AGENTS.md system_prompt_mode: prepend_to_instructions +model_aliases: + small: claude-sonnet + medium: claude-sonnet + large: claude-opus command: # task_position is before_base_args because the compiled OpenCode harness # emits `opencode --prompt ` — flipping this to diff --git a/pkg/hubclient/types.go b/pkg/hubclient/types.go index 62fcc21d7..45fe4f6f6 100644 --- a/pkg/hubclient/types.go +++ b/pkg/hubclient/types.go @@ -557,4 +557,5 @@ type HarnessConfigData struct { Args []string `json:"args,omitempty"` Env map[string]string `json:"env,omitempty"` AuthSelectedType string `json:"authSelectedType,omitempty"` + ModelAliases map[string]string `json:"modelAliases,omitempty"` } diff --git a/pkg/store/models.go b/pkg/store/models.go index 6bebc4c33..d650641dd 100644 --- a/pkg/store/models.go +++ b/pkg/store/models.go @@ -532,6 +532,7 @@ type HarnessConfigData struct { Env map[string]string `json:"env,omitempty"` AuthSelectedType string `json:"authSelectedType,omitempty"` Secrets []api.RequiredSecret `json:"secrets,omitempty"` + ModelAliases map[string]string `json:"modelAliases,omitempty"` } // HarnessConfigStatus constants