Skip to content

fix(skills): use utility model for skill tests instead of chat default - #5746

Merged
alteixeira20 merged 1 commit into
odysseus-dev:devfrom
holden093:fix/skills-test-utility-model-reopen
Jul 30, 2026
Merged

fix(skills): use utility model for skill tests instead of chat default#5746
alteixeira20 merged 1 commit into
odysseus-dev:devfrom
holden093:fix/skills-test-utility-model-reopen

Conversation

@holden093

Copy link
Copy Markdown
Contributor

Summary

Skill tests were using resolve_endpoint("default", ...) which returns the chat model. They should use the utility model, like auto-naming and memory audit already do (fixed in PR #4027). One-word change: "default""utility".

Note: Re-opening of #5026, which was auto-closed on 2026-07-23 when the repository was transferred to the odysseus-dev org (the dev history rewrite orphaned the old base and forced all open PRs closed). This branch is the same one-line fix cleanly rebased onto the current dev. Original review thread: #5026.

Target branch

  • This PR targets dev, not main.

Linked Issue

Fixes #5025

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I verified the change by running a skill test and confirming the logs show the utility model instead of the chat model.

How to Test

  1. Configure different models for Default Chat and Utility in Settings
  2. Go to Skills, pick a skill, click Test
  3. Check logs: the skill test agent round should show the utility model, not the chat model

Visual / UI changes

None — backend-only change.

@github-actions github-actions Bot added the ready for review Description complete — ready for maintainer review label Jul 25, 2026
@holden093

Copy link
Copy Markdown
Contributor Author

Heads-up: the failing Python tests (pytest) check is a pre-existing dev bug, not this PR

The pytest job fails at test collection, before any of this PR's changes are exercised:

NameError: name 'Any' is not defined

Root cause is on dev itself: src/agent_loop.py annotates with dict[str, Any] (lines 1503 and 1877) but only imports AsyncGenerator, List, Dict, Optional, Set — no Any, and no from __future__ import annotations. So importing the module raises NameError, which breaks collection for every test that transitively imports agent_loop (hence the repo-wide failure).

This PR doesn't touch agent_loop.py; every other check (syntax, CodeQL, lint, security scans) passes here, and this PR's own tests pass locally.

One-line fix (add Any to the typing import) is included in #5751. Once dev carries that fix, this PR's pytest will go green with no changes needed here.

Skill tests are background automation tasks (like auto-naming and
memory audit) and should use the configured utility model. Previously
they resolved via resolve_endpoint("default") which returned the
chat model, bypassing the utility model entirely.

This completes the sweep started in PR odysseus-dev#4027 which fixed auto-naming
and memory audit but missed skill tests.
@alteixeira20
alteixeira20 force-pushed the fix/skills-test-utility-model-reopen branch from 76aecae to 8b1431b Compare July 29, 2026 21:18
@alteixeira20
alteixeira20 merged commit 6a84398 into odysseus-dev:dev Jul 30, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(skills): skill tests use chat model instead of configured utility model

2 participants