MoA: ensembling that improves as mesh nodes join - #1116
Open
michaelneale wants to merge 68 commits into
Open
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughMoA now supports single-model fallback routing, structured worker-pool assembly, truncation-aware text arbitration, optional refinement, asymmetric tool handling, and expanded replay and OpenRouter evaluation coverage. ChangesMoA routing and worker-pool assembly
Mixture-of-agents orchestration
Validation and evaluation
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review. |
michaelneale
changed the title
michaelneale
force-pushed
the
feat/moa-synthesis
branch
from
August 1, 2026 01:50
43d1f05 to
44ef1d7
Compare
michaelneale
changed the title
michaelneale
changed the title
michaelneale
changed the title
michaelneale
force-pushed
the
feat/moa-synthesis
branch
from
August 4, 2026 03:42
28b94b5 to
228a1ef
Compare
michaelneale
marked this pull request as ready for review
August 4, 2026 04:14
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 19
🧹 Nitpick comments (18)
evals/moa-openrouter/probe_tools.py (1)
65-70: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueManage the output file handles in both eval scripts. Both scripts open a JSONL output file without a context manager, so no exit path guarantees a close. The shared fix is to scope each handle with
with open(...).
evals/moa-openrouter/probe_tools.py#L65-L70: move the module-levelopen("fanout.jsonl", "a")intomainand scope it withwith, so importing the module no longer creates the file.evals/moa-openrouter/record_agentic.py#L236-L293: replaceout = open("agentic.jsonl", "w")plusout.close()with awith open(...)block, so an exception during a scenario still closes the file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evals/moa-openrouter/probe_tools.py` around lines 65 - 70, Manage both output file handles with context managers: in evals/moa-openrouter/probe_tools.py lines 65-70, move the module-level open call into main and scope it with with; in evals/moa-openrouter/record_agentic.py lines 236-293, replace the out handle and explicit close with a with open block covering scenario processing. Ensure importing probe_tools.py does not create the output file and exceptions still close both handles.evals/moa-openrouter/record_agentic.py (1)
275-291: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse iterable unpacking for the message list.
Ruff reports RUF005 here. Replace the list concatenation with unpacking.
♻️ Proposed refactor
- messages = messages + [ + messages = [ + *messages, { "role": "assistant",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evals/moa-openrouter/record_agentic.py` around lines 275 - 291, Update the message accumulation expression in the agentic recording flow to use iterable unpacking for the existing messages and the two new message dictionaries, replacing list concatenation while preserving their order and contents.Source: Linters/SAST tools
evals/moa-openrouter/make_fixture.py (2)
23-23: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the redundant
intcall.
round()with a single argument already returns anint. Ruff reports this as RUF046.♻️ Proposed change
- "elapsed_ms": int(round((w["elapsed"] or 0) * 1000)), + "elapsed_ms": round((w["elapsed"] or 0) * 1000),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evals/moa-openrouter/make_fixture.py` at line 23, Update the elapsed_ms expression in the fixture-building logic to remove the redundant int wrapper and rely on round((w["elapsed"] or 0) * 1000) returning an integer, preserving the existing elapsed fallback and rounding behavior.Source: Linters/SAST tools
14-16: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winResolve the input and output paths relative to the script.
OUTand bothopencalls use paths relative to the current working directory. The script writes the fixture to the wrong location, or fails, when it runs from the repository root. The docstring does not state the required working directory. The two input handles also stay open until interpreter exit.Anchor the paths to
__file__and use context managers.♻️ Proposed change
import json +from pathlib import Path -OUT = "../../crates/mesh-mixture-of-agents/tests/fixtures/real_traces.json" +HERE = Path(__file__).resolve().parent +OUT = HERE.parents[1] / "crates/mesh-mixture-of-agents/tests/fixtures/real_traces.json"-for line in open("agentic.jsonl"): - line = line.strip() - if not line: - continue - r = json.loads(line) - cases.append( +with open(HERE / "agentic.jsonl") as fh: + agentic_lines = fh.read().splitlines() +for line in agentic_lines: + line = line.strip() + if not line: + continue + r = json.loads(line) + cases.append(Apply the same change to the
corpus.jsonlloop at line 53.Also applies to: 34-34, 53-53, 71-72
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evals/moa-openrouter/make_fixture.py` around lines 14 - 16, Update the fixture path setup in the script to resolve OUT and both input paths relative to the script’s __file__, so execution is independent of the current working directory. Wrap every input and output open call, including the corpus.jsonl loop, in context managers to close handles promptly.crates/mesh-mixture-of-agents/tests/eval_openrouter.rs (1)
1453-1459: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winArm C can get one candidate while arm B gets none.
diverse.len().max(1)forces at least one homogeneous candidate. If every diverse peer call fails, arm B runs with zero candidates and is identical to arm A, while arm C still receives one candidate. The reportedB-Cdifferential then measures candidate presence, not family diversity. Match the counts exactly.♻️ Proposed change
- for _ in 0..diverse.len().max(1) { + for _ in 0..diverse.len() { if let Some(c) = structured_proposal(&backend, &finalizer, task).await { homo.push(c); } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/tests/eval_openrouter.rs` around lines 1453 - 1459, Update the homogeneous candidate loop in the arm C setup to generate exactly diverse.len() candidates, removing the max(1) fallback. Preserve the existing structured_proposal failure handling so zero diverse candidates produces zero homogeneous candidates and the B-C comparison measures family diversity.crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs (1)
213-217: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUpdate the stale comment on
enable_thinking.The comment says
try_handle_moaoverrides this value "when the caller has expressed a preference". That is no longer true.effective_enable_thinking_for_moanow always returnsSome(false)(lines 17-46), andtry_handle_moaassigns it unconditionally. A reader following this comment would look for caller-driven behavior that no longer exists.📝 Proposed wording
- // Defaults to leaving each model's thinking behavior alone. - // `try_handle_moa` overrides this from the inbound request body - // when the caller has expressed a preference - // (`reasoning_effort: "none"`, `enable_thinking: false`, etc.). - enable_thinking: None, + // Placeholder: `try_handle_moa` always overwrites this with + // `effective_enable_thinking_for_moa`, which is `Some(false)` as a + // policy. Caller reasoning knobs are parsed for logging only. + enable_thinking: None,🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs` around lines 213 - 217, Update the comment above enable_thinking to reflect that effective_enable_thinking_for_moa always returns Some(false) and try_handle_moa assigns that value unconditionally; remove the outdated claim that inbound caller preferences control the override.crates/mesh-mixture-of-agents/src/normalize.rs (1)
55-60: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider passing truncation into the normalizer.
Every path in this file hardcodes
truncated: false, and each caller must remember to stamp the real value after the call (fanout::gather_workers_incremental,fanout::gather_references,refinement::refine_round). A new call site that omits the stamp silently reports a truncated answer as complete, which is the exact failure this field exists to prevent. Accepting the flag as a parameter ofnormalize_worker_outputwould make the compiler enforce it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/normalize.rs` around lines 55 - 60, Update normalize_worker_output to accept a truncation flag parameter and use it when constructing WorkerOutput instead of hardcoding truncated: false. Pass the actual flag from every caller, including fanout::gather_workers_incremental, fanout::gather_references, and refinement::refine_round, then remove their post-call stamping.crates/mesh-mixture-of-agents/src/backend.rs (1)
155-159: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueThe retry removes caller-supplied
chat_template_kwargscontent.
apply_enable_thinkingmergesenable_thinkinginto any existingchat_template_kwargsobject. The retry removes the wholechat_template_kwargskey, so template options the caller set for other purposes are lost on the retried request. Removing only the thinking keys keeps the rest intact.♻️ Proposed change
let mut retry_body = body.clone(); if let Some(obj) = retry_body.as_object_mut() { obj.remove("reasoning_effort"); - obj.remove("chat_template_kwargs"); + if let Some(kwargs) = obj + .get_mut("chat_template_kwargs") + .and_then(Value::as_object_mut) + { + kwargs.remove("enable_thinking"); + if kwargs.is_empty() { + obj.remove("chat_template_kwargs"); + } + } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/backend.rs` around lines 155 - 159, The retry preparation in the backend request flow must preserve caller-supplied chat_template_kwargs. Update the retry_body cleanup to remove only the thinking-related entries added by apply_enable_thinking, while retaining the rest of the chat_template_kwargs object; keep removal of reasoning_effort unchanged.crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs (1)
34-42: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConsider the token requirement and a deterministic pick when degrading.
degrade_to_single_modeltakes the first model returned bymodels_being_served(). Two effects follow:
- The picked model may not satisfy
required_tokens.build_moa_configappliescontext_selection::context_can_satisfyper worker, but this fallback path applies no context check, so a request that needs a large context can be routed to a small-context model.- The iteration order of
models_being_served()decides the answer. If that order is not stable, the same lone node can answer with different models across requests.A deterministic selection that prefers locally served models with sufficient context would make the degraded path predictable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs` around lines 34 - 42, Update the fallback model selection around degrade_to_single_model to filter out the virtual model and retain only models whose context satisfies required_tokens, preferring locally served models when available. Sort or otherwise select from the remaining candidates deterministically before choosing the target, while preserving the existing 503 response when no eligible model exists.crates/mesh-mixture-of-agents/src/fanout.rs (1)
350-363: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSimplify the reference stop condition.
Line 352 checks
!outputs.is_empty() && outputs.len() >= min_references. The first clause only matters whenmin_references == 0, and the only caller passesdispatched.len().div_ceil(2).max(1). Either drop the redundant clause or clampmin_referenceswith.max(1)inside this function so the intent is local.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/fanout.rs` around lines 350 - 363, In the reference collection loop, simplify the stop condition around outputs.len() and min_references: remove the redundant outputs non-empty check, or normalize min_references to at least one within the function before the loop. Preserve the behavior that collection stops once the configured minimum number of references is reached.crates/mesh-llm-host-runtime/src/network/openai/ingress.rs (1)
842-857: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSet the degraded flag in the match arm instead of comparing model values.
Line 856 infers
degradedfromrouting_model != decision.effective_model. The branch that produced the value is already known at line 847. The comparison holds today only becausedegrade_to_single_modelexcludesmoa::VIRTUAL_MODEL_NAME, so the rewritten name can never equal"mesh". If that exclusion changes, the pipeline classifier would run against a stale decision. A flag set in the arm removes the dependency.♻️ Proposed refactor
- let mut routing_model = decision.effective_model.clone(); - let tcp_stream = match try_handle_moa_intercept(tcp_stream, &mut request, &ctx, &decision).await - { - MoaInterceptResult::Handled => return, - MoaInterceptResult::NotMoa(stream) => stream, - MoaInterceptResult::Degraded { stream, model } => { - routing_model = model; - stream - } - }; - - let mut tcp_stream = tcp_stream; - // A degraded turn is a plain single-model request; skip the pipeline - // classifier (computed against "mesh") and route it directly. - let degraded = routing_model != decision.effective_model; + let mut routing_model = decision.effective_model.clone(); + let mut degraded = false; + let mut tcp_stream = + match try_handle_moa_intercept(tcp_stream, &mut request, &ctx, &decision).await { + MoaInterceptResult::Handled => return, + MoaInterceptResult::NotMoa(stream) => stream, + MoaInterceptResult::Degraded { stream, model } => { + routing_model = model; + degraded = true; + stream + } + }; + + // A degraded turn is a plain single-model request; skip the pipeline + // classifier (computed against "mesh") and route it directly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-llm-host-runtime/src/network/openai/ingress.rs` around lines 842 - 857, Update the request-routing flow around try_handle_moa_intercept to initialize a degraded flag before the match and set it explicitly in the MoaInterceptResult::Degraded arm. Replace the routing_model != decision.effective_model comparison with that flag, while leaving the Handled and NotMoa paths unchanged so pipeline classification is skipped whenever the intercept produced a degraded single-model request.crates/mesh-mixture-of-agents/src/context.rs (1)
532-545: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueName the advisor payload bound.
reducer_payload_budgetandREFINEMENT_DRAFT_BUDGETexpress their limits as named items with the reasoning attached. This loop hardcodes500and the derived497. Introduce a constant, for exampleADVICE_PAYLOAD_BUDGET, and derive the truncation length from it so the two values cannot drift.♻️ Proposed refactor
+/// How much of each advisor's prose the actor may see. +const ADVICE_PAYLOAD_BUDGET: usize = 500; + @@ - let payload = if r.payload.len() > 500 { - format!("{}...", crate::worker::truncate_chars(&r.payload, 497)) + let payload = if r.payload.len() > ADVICE_PAYLOAD_BUDGET { + format!( + "{}...", + crate::worker::truncate_chars(&r.payload, ADVICE_PAYLOAD_BUDGET - 3) + ) } else { r.payload.clone() };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/context.rs` around lines 532 - 545, In the advisor payload loop over references, replace the hardcoded 500-byte/character bound and derived 497 truncation length with a named constant such as ADVICE_PAYLOAD_BUDGET, documenting the limit’s purpose consistently with reducer_payload_budget and REFINEMENT_DRAFT_BUDGET. Derive the truncation length from that constant so the displayed payload plus ellipsis always stays within the same budget.crates/mesh-mixture-of-agents/src/tool_turn.rs (2)
255-336: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd tests for
actor_body.The policy tests are thorough.
actor_bodyis untested, and it encodes the response contract for the whole tool path: aToolProposalmust emittool_calls, anUncertaintywith a forced tool must emit the forced call, and anUncertaintywith no forced tool and no references must emitMOA_ERR_NO_USABLE_ANSWER. It is a pure function overWorkerOutput, so it needs no backend. Add cases for those three branches.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/tool_turn.rs` around lines 255 - 336, Add unit tests for the pure actor_body function covering three response-contract branches: ToolProposal produces tool_calls, Uncertainty with a forced tool produces that forced call, and Uncertainty without a forced tool or references returns MOA_ERR_NO_USABLE_ANSWER. Construct WorkerOutput fixtures directly without backend setup, and keep the existing policy tests unchanged.
143-147: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winBoth new fan-out modules rebuild an identical prompt inside the per-worker loop.
dispatch_and_gather_referencesandrefine_roundeach call a context packer once per spawned worker with arguments that do not vary by worker, so the samePackedContextis built N times. Hoist the packing above the loop and clone the messages into each task.
crates/mesh-mixture-of-agents/src/tool_turn.rs#L143-L147: movecontext::pack_for_reference(session, REFERENCE_HISTORY_MESSAGES)above thefor a in &assignmentsloop and clonepacked.messagesper task.crates/mesh-mixture-of-agents/src/refinement.rs#L100-L102: movecontext::pack_for_refinement(session, &texts)above the loop; this call also concatenates and truncates every draft, so the repeated work is larger here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/tool_turn.rs` around lines 143 - 147, The fan-out paths rebuild identical packed context for every worker. In crates/mesh-mixture-of-agents/src/tool_turn.rs lines 143-147, update dispatch_and_gather_references to call context::pack_for_reference once before the for a in &assignments loop and clone packed.messages into each task; in crates/mesh-mixture-of-agents/src/refinement.rs lines 100-102, update refine_round to call context::pack_for_refinement once before its worker loop and clone the resulting messages per task.crates/mesh-mixture-of-agents/src/refinement.rs (2)
194-291: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a
refine_roundtest for the shortfall path.The tests cover policy and budget arithmetic well. They do not exercise
refine_round, whose central safety claim is that fewer thanMIN_DRAFTSrefinements returnNoneso the caller keeps the round-1 outputs.reducer.rsalready has aFakeBackendhelper that can drive this. Add one test where a single worker succeeds and assertrefine_roundreturnsNone.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/refinement.rs` around lines 194 - 291, Add a unit test for refine_round using the existing reducer.rs FakeBackend helper, configuring one worker to succeed and produce fewer than MIN_DRAFTS refinement results. Assert that refine_round returns None, preserving the caller’s round-1 outputs; keep the test focused on this shortfall path.
141-148: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRecord a summary when a refinement reply is empty.
Line 142 skips a reply whose text is blank and continues without pushing a
WorkerSummary. The worker ran and consumed tokens, but it appears in no accounting. Every other outcome in this loop produces a summary, and the round-1 gather reconciles all dispatched workers. Add a failed summary so refinement attempts stay observable.♻️ Proposed change
Ok((model, role, Ok(reply), elapsed)) => { if reply.text.trim().is_empty() { + summaries.push(WorkerSummary { + model, + role, + succeeded: false, + elapsed_ms: elapsed, + output_kind: None, + confidence: None, + }); continue; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/refinement.rs` around lines 141 - 148, Update the empty-reply branch in the refinement result loop before continue so it pushes a failed WorkerSummary for the completed worker, preserving the worker identity and relevant timing/token accounting; then continue without normalizing output. Ensure every dispatched refinement attempt remains represented consistently with the other result branches and round-1 reconciliation.crates/mesh-mixture-of-agents/src/lib.rs (2)
1309-1309: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffConsider extracting the response builders into their own module.
These six builders are now
pub(crate)becausetool_turnconsumes them. They form a separable responsibility: OpenAI wire-response construction.lib.rsis over 2,000 lines and this PR modifies it.Move
best_answer,fallback_worker_response,tool_proposal_response,error_response,chat_response, andtool_call_responseplusresponse_builder_testsinto aresponsemodule. That keepslib.rsshrinking as the crate grows and moves the tests with the behavior.As per coding guidelines: "When modifying a Rust source file over 1,000 lines, extract any separable responsibility into a semantically named module, keep the new file under 1,000 lines, and move relevant tests with the extracted behavior."
Also applies to: 1326-1326, 1338-1338, 1374-1374, 1420-1420, 1434-1434
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/lib.rs` at line 1309, Extract the OpenAI wire-response builders best_answer, fallback_worker_response, tool_proposal_response, error_response, chat_response, and tool_call_response from lib.rs into a semantically named response module, preserving their pub(crate) visibility and existing behavior. Move response_builder_tests into the same module, update module declarations and call sites such as tool_turn, and keep the new module under 1,000 lines.Source: Coding guidelines
316-350: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSimplify the now-unreachable tool bookkeeping on the text path.
The guard at line 310 returns for every turn where
has_toolsis true orforced_toolis set. After that guard,query_uses_toolsat line 343 is alwaysfalse,selected_tool_namesis always empty, andgrace_mode_for_turn(session, has_tools)is always called withhas_tools == false, soGraceMode::Toolis unreachable here. The comment at lines 318-342 explains a decision that this path no longer makes.Replace the computed values with constants and move the historical rationale to
tool_turn. The change keeps behavior identical and removes a misleading contract for the next reader.♻️ Proposed simplification
- let query_uses_tools = forced_tool.is_some() || has_tools; - let selected_tool_names = if let Some(tool) = forced_tool { - vec![tool.name.clone()] - } else if query_uses_tools { - selected_tool_names_for_turn(session, allowed_tools) - } else { - Vec::new() - }; + // Tool-bearing turns returned above, so this path is always tool-free. + let query_uses_tools = false; + let selected_tool_names: Vec<String> = Vec::new();🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/lib.rs` around lines 316 - 350, In the post-guard text path, replace the computed tool state with constant no-tools values and call grace_mode_for_turn with tools unavailable, since forced_tool and has_tools cannot reach this branch. Move the historical tool-availability rationale from this block to tool_turn, retaining only comments relevant to the text path. Preserve the existing behavior while removing the misleading query_uses_tools and selected_tool_names contract.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs`:
- Around line 399-470: Extract the separable pool-assembly responsibilities from
this oversized module into semantically named modules, such as admission and
patience, keeping each new file under 1,000 lines. Move apply_admission_control
and its tests into the admission module; move cap_committee,
self_fill_from_extra_instances, assemble_worker_pool, and their tests into an
appropriate pool/committee module, and move PatienceProfile, patience_profile,
and related tests into the patience module. Update module declarations and call
sites while preserving existing behavior.
- Around line 522-548: Update assemble_worker_pool and
self_fill_from_extra_instances to carry the initial remote peer selected by
add_worker_backend, and skip that peer when adding extra instances to the MoA
committee. Pass the tracked peer through the call chain, remove the unused _http
parameter from self_fill_from_extra_instances and its callers, and preserve the
existing worker-count limit.
In `@crates/mesh-mixture-of-agents/src/arbiter.rs`:
- Around line 455-463: Update the match in single_output_decision so
OutputKind::ToolProposal returns Decision::NeedsReducer with the appropriate
reducer reason, alongside OutputKind::Uncertainty. Keep non-tool, non-uncertain
outputs on the existing Decision::Answer path.
In `@crates/mesh-mixture-of-agents/src/backend.rs`:
- Around line 356-361: Update the tool-proposal path around
BackendReply::complete to validate the raw args as JSON and set the reply’s
truncated state when parsing fails specifically with finish_reason == "length";
preserve the complete status for valid JSON or non-length finish reasons, and
avoid treating the unwrap_or("{}") fallback as proof that the original arguments
were complete.
In `@crates/mesh-mixture-of-agents/src/context.rs`:
- Around line 347-453: Extract the packing responsibilities from context.rs into
semantically named modules: move REFERENCE_PREAMBLE and pack_for_reference with
their tests to a reference module, pack_for_refinement and
REFINEMENT_DRAFT_BUDGET with related tests to refinement_context, and
pack_for_actor with its tests to actor. Update module declarations, imports, and
call sites so the existing behavior and APIs remain unchanged while keeping each
source file under 1,000 lines.
- Around line 475-497: Verify the refinement configuration in the code and
measurement setup around the PackedContext construction and its max_tokens
value. If the measured refinement configuration used the full worker token
budget rather than 1024 tokens, increase max_tokens to that budget so synthesis
is not unnecessarily truncated; otherwise preserve the current value.
In `@crates/mesh-mixture-of-agents/src/fanout.rs`:
- Around line 383-397: Update reconcile_dispatched, used by gather_references
and gather_workers_incremental, to match returned summaries to dispatched
workers by slot index or another unique per-dispatch identifier instead of model
name. Propagate that identifier through the dispatch/result flow so duplicate
same-model slots reconcile independently and failed or cancelled slots remain in
TurnResult.worker_summaries.
In `@crates/mesh-mixture-of-agents/src/reducer.rs`:
- Around line 154-158: Make reducer truncation observable in the output handling
around the result mapping that returns `(name, result.map(|reply| reply.text))`.
Preserve the full reducer text, but inspect `reply.truncated` and log when the
final synthesis response was truncated, using the existing logging mechanism and
context available in the reducer flow.
In `@crates/mesh-mixture-of-agents/src/tool_turn.rs`:
- Around line 178-187: Cap the optional advisory wait in the tool-turn flow
before calling gather_references, using the same worker-budget fraction
established by refinement.rs rather than the full config.worker_timeout. Keep
min_references unchanged and preserve pack_for_actor’s empty-reference fallback
when the bounded advisory round expires.
- Around line 79-92: Update the TurnResult construction in finalize_actor_output
so reducer_used reflects actor_ok instead of always being true. Preserve the
existing reducer_attempts value and response handling, ensuring failed reducer
attempts that return fallback_worker_response or error_response report
reducer_used as false.
- Around line 54-55: Update handle_tool_query to accept and propagate the
caller’s has_tools flag, passing it to context::pack_for_actor instead of
hardcoding true. Ensure tool_proposal_response and forced tool_call_response
paths use the same flag so tool-call responses are omitted when the request has
no tools array.
In `@crates/mesh-mixture-of-agents/src/worker.rs`:
- Around line 144-181: Align the documentation for canonical_base_name and
pool_is_homogeneous with the actual behavior: quant-tagged names remain distinct
from untagged names, so mixed quant variants are not considered homogeneous and
refinement::refinement_expected will not enable refinement for them. Update the
comments to remove the claim that quant variants share a base, and add coverage
for a plain name versus a quant-tagged alias to pin this contract.
In `@crates/mesh-mixture-of-agents/tests/eval_openrouter.rs`:
- Around line 2212-2215: Update the solo-baseline setup around small_mesh_pool
so it does not claim pool[0] is the strongest member: either select the
strongest model explicitly or revise the comment and output wording to describe
the first declaration-order entry. Also handle an empty pool before indexing,
using the test’s existing failure or early-return convention instead of allowing
pool[0] to panic.
- Around line 460-466: Replace the local truncate helper and its call sites with
the exported truncate_chars behavior, ensuring truncation never slices through a
UTF-8 character. Update crates/mesh-mixture-of-agents/src/lib.rs::truncate_chars
if necessary so its limit semantics match the required character-count boundary,
and apply the safe helper to model output, HTTP error bodies, and advisor
errors.
In `@crates/mesh-mixture-of-agents/tests/sim_real_traces.rs`:
- Around line 541-557: Update
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L541-L557 to remove the
result.reducer_used skip for tool-bearing turns so the majority-argument
assertions execute; update is_reducer_call at
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L146-L154 to recognize
the pack_for_actor system marker “## Advice from other models” and return
SYNTHESIZED for the actor call instead of replaying a worker body.
In `@crates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rs`:
- Around line 3-7: Update the module-level rationale near the refinement
description to remove the withdrawn “42/66/12, p=5.2e-05” result or replace it
with the controlled evaluation result from evals/moa-openrouter/RESULTS.md;
retain the explanation that refinement is best-effort and may introduce another
hanging fan-out.
In `@evals/moa-openrouter/probe_tools.py`:
- Around line 100-105: Update the tool-call formatting comprehension in the
result-printing block to read arguments defensively via the nested function
mapping, defaulting missing or falsy values to "{}" as in record_agentic.py,
while preserving the existing call-name formatting and probe flow.
In `@evals/moa-openrouter/record.py`:
- Around line 31-41: Update the worker record construction around the response
handling to persist the complete OpenAI response, including usage and unselected
fields, by storing resp directly in each record while retaining the existing
derived fields as needed.
In `@evals/moa-openrouter/RESULTS.md`:
- Around line 487-493: Update the documented merge-blocker status around the
harness-versus-production parity claim in moa::handle_turn: either provide
evidence that the orchestration gap is resolved, or explicitly record an
approved waiver. Do not present the production reasoning-quality improvement as
validated while this blocker remains unresolved.
---
Nitpick comments:
In `@crates/mesh-llm-host-runtime/src/network/openai/ingress.rs`:
- Around line 842-857: Update the request-routing flow around
try_handle_moa_intercept to initialize a degraded flag before the match and set
it explicitly in the MoaInterceptResult::Degraded arm. Replace the routing_model
!= decision.effective_model comparison with that flag, while leaving the Handled
and NotMoa paths unchanged so pipeline classification is skipped whenever the
intercept produced a degraded single-model request.
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs`:
- Around line 34-42: Update the fallback model selection around
degrade_to_single_model to filter out the virtual model and retain only models
whose context satisfies required_tokens, preferring locally served models when
available. Sort or otherwise select from the remaining candidates
deterministically before choosing the target, while preserving the existing 503
response when no eligible model exists.
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs`:
- Around line 213-217: Update the comment above enable_thinking to reflect that
effective_enable_thinking_for_moa always returns Some(false) and try_handle_moa
assigns that value unconditionally; remove the outdated claim that inbound
caller preferences control the override.
In `@crates/mesh-mixture-of-agents/src/backend.rs`:
- Around line 155-159: The retry preparation in the backend request flow must
preserve caller-supplied chat_template_kwargs. Update the retry_body cleanup to
remove only the thinking-related entries added by apply_enable_thinking, while
retaining the rest of the chat_template_kwargs object; keep removal of
reasoning_effort unchanged.
In `@crates/mesh-mixture-of-agents/src/context.rs`:
- Around line 532-545: In the advisor payload loop over references, replace the
hardcoded 500-byte/character bound and derived 497 truncation length with a
named constant such as ADVICE_PAYLOAD_BUDGET, documenting the limit’s purpose
consistently with reducer_payload_budget and REFINEMENT_DRAFT_BUDGET. Derive the
truncation length from that constant so the displayed payload plus ellipsis
always stays within the same budget.
In `@crates/mesh-mixture-of-agents/src/fanout.rs`:
- Around line 350-363: In the reference collection loop, simplify the stop
condition around outputs.len() and min_references: remove the redundant outputs
non-empty check, or normalize min_references to at least one within the function
before the loop. Preserve the behavior that collection stops once the configured
minimum number of references is reached.
In `@crates/mesh-mixture-of-agents/src/lib.rs`:
- Line 1309: Extract the OpenAI wire-response builders best_answer,
fallback_worker_response, tool_proposal_response, error_response, chat_response,
and tool_call_response from lib.rs into a semantically named response module,
preserving their pub(crate) visibility and existing behavior. Move
response_builder_tests into the same module, update module declarations and call
sites such as tool_turn, and keep the new module under 1,000 lines.
- Around line 316-350: In the post-guard text path, replace the computed tool
state with constant no-tools values and call grace_mode_for_turn with tools
unavailable, since forced_tool and has_tools cannot reach this branch. Move the
historical tool-availability rationale from this block to tool_turn, retaining
only comments relevant to the text path. Preserve the existing behavior while
removing the misleading query_uses_tools and selected_tool_names contract.
In `@crates/mesh-mixture-of-agents/src/normalize.rs`:
- Around line 55-60: Update normalize_worker_output to accept a truncation flag
parameter and use it when constructing WorkerOutput instead of hardcoding
truncated: false. Pass the actual flag from every caller, including
fanout::gather_workers_incremental, fanout::gather_references, and
refinement::refine_round, then remove their post-call stamping.
In `@crates/mesh-mixture-of-agents/src/refinement.rs`:
- Around line 194-291: Add a unit test for refine_round using the existing
reducer.rs FakeBackend helper, configuring one worker to succeed and produce
fewer than MIN_DRAFTS refinement results. Assert that refine_round returns None,
preserving the caller’s round-1 outputs; keep the test focused on this shortfall
path.
- Around line 141-148: Update the empty-reply branch in the refinement result
loop before continue so it pushes a failed WorkerSummary for the completed
worker, preserving the worker identity and relevant timing/token accounting;
then continue without normalizing output. Ensure every dispatched refinement
attempt remains represented consistently with the other result branches and
round-1 reconciliation.
In `@crates/mesh-mixture-of-agents/src/tool_turn.rs`:
- Around line 255-336: Add unit tests for the pure actor_body function covering
three response-contract branches: ToolProposal produces tool_calls, Uncertainty
with a forced tool produces that forced call, and Uncertainty without a forced
tool or references returns MOA_ERR_NO_USABLE_ANSWER. Construct WorkerOutput
fixtures directly without backend setup, and keep the existing policy tests
unchanged.
- Around line 143-147: The fan-out paths rebuild identical packed context for
every worker. In crates/mesh-mixture-of-agents/src/tool_turn.rs lines 143-147,
update dispatch_and_gather_references to call context::pack_for_reference once
before the for a in &assignments loop and clone packed.messages into each task;
in crates/mesh-mixture-of-agents/src/refinement.rs lines 100-102, update
refine_round to call context::pack_for_refinement once before its worker loop
and clone the resulting messages per task.
In `@crates/mesh-mixture-of-agents/tests/eval_openrouter.rs`:
- Around line 1453-1459: Update the homogeneous candidate loop in the arm C
setup to generate exactly diverse.len() candidates, removing the max(1)
fallback. Preserve the existing structured_proposal failure handling so zero
diverse candidates produces zero homogeneous candidates and the B-C comparison
measures family diversity.
In `@evals/moa-openrouter/make_fixture.py`:
- Line 23: Update the elapsed_ms expression in the fixture-building logic to
remove the redundant int wrapper and rely on round((w["elapsed"] or 0) * 1000)
returning an integer, preserving the existing elapsed fallback and rounding
behavior.
- Around line 14-16: Update the fixture path setup in the script to resolve OUT
and both input paths relative to the script’s __file__, so execution is
independent of the current working directory. Wrap every input and output open
call, including the corpus.jsonl loop, in context managers to close handles
promptly.
In `@evals/moa-openrouter/probe_tools.py`:
- Around line 65-70: Manage both output file handles with context managers: in
evals/moa-openrouter/probe_tools.py lines 65-70, move the module-level open call
into main and scope it with with; in evals/moa-openrouter/record_agentic.py
lines 236-293, replace the out handle and explicit close with a with open block
covering scenario processing. Ensure importing probe_tools.py does not create
the output file and exceptions still close both handles.
In `@evals/moa-openrouter/record_agentic.py`:
- Around line 275-291: Update the message accumulation expression in the agentic
recording flow to use iterable unpacking for the existing messages and the two
new message dictionaries, replacing list concatenation while preserving their
order and contents.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1f1bf079-4ba5-464a-8bc2-9d17fd49c867
📒 Files selected for processing (39)
crates/mesh-llm-guardrails/src/rescue.rscrates/mesh-llm-host-runtime/src/network/openai/ingress.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rscrates/mesh-mixture-of-agents/src/arbiter.rscrates/mesh-mixture-of-agents/src/backend.rscrates/mesh-mixture-of-agents/src/context.rscrates/mesh-mixture-of-agents/src/fanout.rscrates/mesh-mixture-of-agents/src/lib.rscrates/mesh-mixture-of-agents/src/normalize.rscrates/mesh-mixture-of-agents/src/reducer.rscrates/mesh-mixture-of-agents/src/refinement.rscrates/mesh-mixture-of-agents/src/tool_guard.rscrates/mesh-mixture-of-agents/src/tool_turn.rscrates/mesh-mixture-of-agents/src/worker.rscrates/mesh-mixture-of-agents/tests/eval_openrouter.rscrates/mesh-mixture-of-agents/tests/fixtures/ablation_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/committee_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/real_traces.jsoncrates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rscrates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rscrates/mesh-mixture-of-agents/tests/sim_partial_worker_survival.rscrates/mesh-mixture-of-agents/tests/sim_real_traces.rscrates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rscrates/mesh-mixture-of-agents/tests/sim_strong_patience.rscrates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rscrates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rscrates/mesh-mixture-of-agents/tests/sim_worker_accounting.rsevals/moa-openrouter/README.mdevals/moa-openrouter/RESULTS.mdevals/moa-openrouter/agentic.jsonlevals/moa-openrouter/analyze_ablation.pyevals/moa-openrouter/corpus.jsonlevals/moa-openrouter/fanout.jsonlevals/moa-openrouter/make_fixture.pyevals/moa-openrouter/orclient.pyevals/moa-openrouter/probe_tools.pyevals/moa-openrouter/record.pyevals/moa-openrouter/record_agentic.py
Comment on lines
+455
to
463
| // Tool turns take the asymmetric actor path, so this text-path arbiter | ||
| // never sees an executable ToolProposal (tool-shaped text is demoted to | ||
| // Uncertainty by `enforce_tool_call_contract` when tools are disabled). | ||
| match output.kind { | ||
| OutputKind::ToolProposal if has_tools => { | ||
| if let Some(ref name) = output.tool_name { | ||
| Decision::ToolCall { | ||
| name: name.clone(), | ||
| arguments: output | ||
| .tool_arguments | ||
| .clone() | ||
| .unwrap_or(Value::Object(Default::default())), | ||
| } | ||
| } else { | ||
| Decision::Answer(output.payload.clone()) | ||
| } | ||
| } | ||
| OutputKind::Uncertainty => Decision::NeedsReducer { | ||
| reason: "single worker uncertain".into(), | ||
| }, | ||
| _ => Decision::Answer(output.payload.clone()), | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Confirm empty-allow-list behaviour of enforce_tool_call_contract.
fd -t f 'tool_guard.rs' crates -x sed -n '1,80p'Repository: Mesh-LLM/mesh-llm
Length of output: 2911
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the relevant arbiter implementation and nearby tests without running repo code.
fd -t f 'arbiter.rs' crates/mesh-mixture-of-agents/src -x sh -c '
echo "=== $1 ==="
wc -l "$1"
echo "--- outline ---"
ast-grep outline "$1" || true
echo "--- relevant sections ---"
sed -n "420,490p" "$1"
echo "--- empty_allowed_list references in arbiter ---"
rg -n "enforce_tool_call_contract|single_output_decision|arbitrate|ToolProposal|OutputKind::" "$1" || true
' sh {}
echo "=== tool_guard tests ==="
fd -t f 'tool_guard.rs' crates -x sh -c '
sed -n "60,160p" "$1"
' sh {}Repository: Mesh-LLM/mesh-llm
Length of output: 13581
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the multi-output branch and make sure ToolProposal is absent from answer/critique/uncertainty collections.
fd -t f 'arbiter.rs' crates/mesh-mixture-of-agents/src -x sh -c '
echo "=== $1 relevant branch ==="
sed -n "33,75p" "$1"
echo "--- ToolProposal/Arbitrate tests ---"
rg -n "ToolProposal|empty_allowed_list_is_noop|enforce_tool_call_contract|arbitrate" "$1" || true
' sh {}
# Read-only behavioral probe of the relevant guard condition and decision arms from the source text.
python3 - <<'PY'
from pathlib import Path
arbiter = Path("crates/mesh-mixture-of-agents/src/arbiter.rs").read_text()
for pattern in [
"let allowed_tools = enforce_tool_call_contract",
"let mut allowed_tools = allowed_tools.clone()",
"enforce_tool_call_contract(&mut out, allowed_tools",
]:
print(f"{pattern!r}: {'present' if pattern in arbiter else 'absent'}")
print("single_output_decision guard references allowed_tools:", "allowed_tools" in arbiter[arbiter.find("fn single_output_decision"):arbiter.find("\n}", arbiter.find("fn single_output_decision"))])
PYRepository: Mesh-LLM/mesh-llm
Length of output: 2795
Route lone ToolProposal outputs to the reducer.
enforce_tool_call_contract leaves an empty allow-list unchanged, and this arbiter has no allowed_tools guard. A single tool-shaped payload reaches single_output_decision as OutputKind::ToolProposal and falls into the _ arm as Decision::Answer(payload), returning proposal prose to a tool-free request. Send ToolProposal to NeedsReducer so synthesis produces real text.
🐛 Proposed fix
- // Tool turns take the asymmetric actor path, so this text-path arbiter
- // never sees an executable ToolProposal (tool-shaped text is demoted to
- // Uncertainty by `enforce_tool_call_contract` when tools are disabled).
+ // Tool turns take the asymmetric actor path, so a `ToolProposal` here is
+ // tool-shaped text on a tools-disabled turn. Its payload is not an answer,
+ // so synthesize instead of returning the proposal prose verbatim.
match output.kind {
- OutputKind::Uncertainty => Decision::NeedsReducer {
+ OutputKind::Uncertainty => Decision::NeedsReducer {
reason: "single worker uncertain".into(),
},
+ OutputKind::ToolProposal => Decision::NeedsReducer {
+ reason: "single worker proposed a tool on a tool-free turn".into(),
+ },
_ => Decision::Answer(output.payload.clone()),
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| // Tool turns take the asymmetric actor path, so this text-path arbiter | |
| // never sees an executable ToolProposal (tool-shaped text is demoted to | |
| // Uncertainty by `enforce_tool_call_contract` when tools are disabled). | |
| match output.kind { | |
| OutputKind::ToolProposal if has_tools => { | |
| if let Some(ref name) = output.tool_name { | |
| Decision::ToolCall { | |
| name: name.clone(), | |
| arguments: output | |
| .tool_arguments | |
| .clone() | |
| .unwrap_or(Value::Object(Default::default())), | |
| } | |
| } else { | |
| Decision::Answer(output.payload.clone()) | |
| } | |
| } | |
| OutputKind::Uncertainty => Decision::NeedsReducer { | |
| reason: "single worker uncertain".into(), | |
| }, | |
| _ => Decision::Answer(output.payload.clone()), | |
| } | |
| // Tool turns take the asymmetric actor path, so a `ToolProposal` here is | |
| // tool-shaped text on a tools-disabled turn. Its payload is not an answer, | |
| // so synthesize instead of returning the proposal prose verbatim. | |
| match output.kind { | |
| OutputKind::Uncertainty => Decision::NeedsReducer { | |
| reason: "single worker uncertain".into(), | |
| }, | |
| OutputKind::ToolProposal => Decision::NeedsReducer { | |
| reason: "single worker proposed a tool on a tool-free turn".into(), | |
| }, | |
| _ => Decision::Answer(output.payload.clone()), | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/mesh-mixture-of-agents/src/arbiter.rs` around lines 455 - 463, Update
the match in single_output_decision so OutputKind::ToolProposal returns
Decision::NeedsReducer with the appropriate reducer reason, alongside
OutputKind::Uncertainty. Keep non-tool, non-uncertain outputs on the existing
Decision::Answer path.
Comment on lines
+541
to
+557
| // The reducer may legitimately rewrite the turn into prose; only | ||
| // assert when a tool call was emitted from worker proposals. | ||
| if tools.is_empty() || result.reducer_used { | ||
| continue; | ||
| } | ||
|
|
||
| let majority_val: Value = serde_json::from_str(&majority).unwrap_or(Value::Null); | ||
| for (name, args) in &tools { | ||
| let got: Value = serde_json::from_str(args).unwrap_or(Value::Null); | ||
| assert_eq!( | ||
| got, majority_val, | ||
| "case `{}`: tool `{name}` should use the majority arguments \ | ||
| ({majority_n} workers proposed {majority}), not a minority variant. \ | ||
| Got {args}", | ||
| case.id | ||
| ); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
The replay harness still models the symmetric reducer path for tool turns. Tool-bearing turns now route to handle_tool_query, which packs the final request with pack_for_actor and always reports reducer_used: true. Both sites assume the old reducer fan-out shape, so the tool assertions no longer measure the shipped path.
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L541-L557: replace theresult.reducer_usedskip, which is alwaystrueforhas_toolscases and makes the majority-argumentassert_eq!unreachable.crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L146-L154: extendis_reducer_callto recognize thepack_for_actorsystem marker (## Advice from other models), so the actor call is answered withSYNTHESIZEDinstead of a replayed worker body.
📍 Affects 1 file
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L541-L557(this comment)crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L146-L154
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/mesh-mixture-of-agents/tests/sim_real_traces.rs` around lines 541 -
557, Update crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L541-L557 to
remove the result.reducer_used skip for tool-bearing turns so the
majority-argument assertions execute; update is_reducer_call at
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L146-L154 to recognize
the pack_for_actor system marker “## Advice from other models” and return
SYNTHESIZED for the actor call instead of replaying a worker body.
Comment on lines
+31
to
+41
| text, tcs = oc.first_choice(resp) | ||
| ch = (resp.get("choices") or [{}])[0] | ||
| return { | ||
| "model": model, | ||
| "tier": oc.tier(model), | ||
| "elapsed": round(elapsed, 2), | ||
| "error": resp.get("error"), | ||
| "finish_reason": ch.get("finish_reason"), | ||
| "text": text, | ||
| "tool_calls": tcs, | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Persist the complete worker response.
The recorder discards usage and all unselected OpenAI response fields. This conflicts with evals/moa-openrouter/README.md, which requires full response preservation, including usage. Store resp in each worker record, or add the omitted fields and narrow the documented contract.
Proposed fix
return {
"model": model,
"tier": oc.tier(model),
"elapsed": round(elapsed, 2),
+ "response": resp,
"error": resp.get("error"),
"finish_reason": ch.get("finish_reason"),
"text": text,
"tool_calls": tcs,
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| text, tcs = oc.first_choice(resp) | |
| ch = (resp.get("choices") or [{}])[0] | |
| return { | |
| "model": model, | |
| "tier": oc.tier(model), | |
| "elapsed": round(elapsed, 2), | |
| "error": resp.get("error"), | |
| "finish_reason": ch.get("finish_reason"), | |
| "text": text, | |
| "tool_calls": tcs, | |
| } | |
| text, tcs = oc.first_choice(resp) | |
| ch = (resp.get("choices") or [{}])[0] | |
| return { | |
| "model": model, | |
| "tier": oc.tier(model), | |
| "elapsed": round(elapsed, 2), | |
| "response": resp, | |
| "error": resp.get("error"), | |
| "finish_reason": ch.get("finish_reason"), | |
| "text": text, | |
| "tool_calls": tcs, | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/record.py` around lines 31 - 41, Update the worker
record construction around the response handling to persist the complete OpenAI
response, including usage and unselected fields, by storing resp directly in
each record while retaining the existing derived fields as needed.
i386
reviewed
Aug 4, 2026
i386
left a comment
i386
left a comment
Collaborator
There was a problem hiding this comment.
One functional issue found in the committee-cap path.
i386
reviewed
Aug 4, 2026
i386
left a comment
i386
left a comment
Collaborator
There was a problem hiding this comment.
A second issue concerns the capacity signal that gates admission.
Collaborator
|
Follow-up on the tiering signal: parsing The better source is GGUF inspection: enumerate the GGUF tensors and sum their |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rs`:
- Around line 242-244: Update assemble_worker_pool and
self_fill_from_extra_instances to retain and pass the resolved alias group and
required_tokens. Query all aliases in that group when self-filling, and apply
the same remote-host context eligibility check as add_worker_backend before
adding each endpoint, so insufficient-context hosts are excluded. Add
regressions covering equivalent canonical aliases and hosts below the required
context capacity.
- Around line 19-29: Update resolve_one_worker_from_aliases to accept and use
the existing WorkerBackendResolution value for immutable resolution inputs,
removing its separate targets, aliases, and required_tokens parameters as
appropriate. Remove the unused local_count parameter and eliminate the
#[allow(clippy::too_many_arguments)] suppression, updating all call sites to
match.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: ba7e182b-4708-4854-986a-b213df4ea9cb
📒 Files selected for processing (3)
crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs
michaelneale
force-pushed
the
feat/moa-synthesis
branch
from
August 4, 2026 08:26
365c470 to
01d717d
Compare
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rs`:
- Around line 314-333: Ensure self-fill never duplicates an existing peer or
re-adds the local backend. Update add_worker_backend to return or store the
selected remote mesh::PeerId, collect peer identities already represented by the
current backends, and filter hosts_for_model(&name) in the self-fill loop before
creating each RemoteModelBackend; use the actual peer-id type from
RemoteModelBackend.
In `@evals/moa-openrouter/analyze_ablation.py`:
- Around line 175-196: Update the verdict logic in the analysis flow after the
B/C estimates to bootstrap a paired B−C differential using the same task
resamples, producing its own confidence interval. Base the “gain is CONTENT”
classification on that differential interval being entirely above zero, rather
than on b_lo; retain the existing B−A verdict and non-content-specific
comparison separately.
In `@evals/moa-openrouter/orclient.py`:
- Around line 74-76: Update the HTTPError handling in chat to decode the
response body defensively, falling back safely when the body is not valid UTF-8
so UnicodeDecodeError cannot escape the handler. Preserve the existing HTTP
status/detail formatting and structured error return behavior.
In `@evals/moa-openrouter/probe_tools.py`:
- Around line 27-31: Update each function schema in probe_tools.py, including
the schemas around the shown properties and the additional locations noted in
the review, to set additionalProperties to false alongside type, properties, and
required. Ensure every probe tool schema matches the production constraint
enforced by tool_guard.
- Line 79: Update the oc.chat call in the probe flow, including the
corresponding calls around the additionally affected lines, to pass
no_think=True. Preserve the existing model, messages, tools, token, and
temperature arguments so all workers and aggregators use the production MoA
thinking configuration.
- Line 199: Update the fanout_tool_result record call to store the complete
tool-result conversation by passing msgs_b as messages alongside the existing
results payload. Preserve the current record name and result data.
In `@evals/moa-openrouter/record_agentic.py`:
- Around line 131-140: Update consensus_tool to normalize valid JSON arguments
before counting tool-call proposals: parse each argument string, re-serialize
parsed objects with stable key ordering, and use that canonical form in the
consensus key. Preserve malformed arguments under a separate raw-string fallback
key, while keeping tool names and existing consensus selection behavior
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 80edd6cb-badd-48c4-9245-c592678da89a
📒 Files selected for processing (40)
crates/mesh-llm-guardrails/src/rescue.rscrates/mesh-llm-host-runtime/src/network/openai/ingress.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rscrates/mesh-mixture-of-agents/src/arbiter.rscrates/mesh-mixture-of-agents/src/backend.rscrates/mesh-mixture-of-agents/src/context.rscrates/mesh-mixture-of-agents/src/fanout.rscrates/mesh-mixture-of-agents/src/lib.rscrates/mesh-mixture-of-agents/src/normalize.rscrates/mesh-mixture-of-agents/src/reducer.rscrates/mesh-mixture-of-agents/src/refinement.rscrates/mesh-mixture-of-agents/src/tool_guard.rscrates/mesh-mixture-of-agents/src/tool_turn.rscrates/mesh-mixture-of-agents/src/worker.rscrates/mesh-mixture-of-agents/tests/eval_openrouter.rscrates/mesh-mixture-of-agents/tests/fixtures/ablation_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/committee_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/real_traces.jsoncrates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rscrates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rscrates/mesh-mixture-of-agents/tests/sim_partial_worker_survival.rscrates/mesh-mixture-of-agents/tests/sim_real_traces.rscrates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rscrates/mesh-mixture-of-agents/tests/sim_strong_patience.rscrates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rscrates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rscrates/mesh-mixture-of-agents/tests/sim_worker_accounting.rsevals/moa-openrouter/README.mdevals/moa-openrouter/RESULTS.mdevals/moa-openrouter/agentic.jsonlevals/moa-openrouter/analyze_ablation.pyevals/moa-openrouter/corpus.jsonlevals/moa-openrouter/fanout.jsonlevals/moa-openrouter/make_fixture.pyevals/moa-openrouter/orclient.pyevals/moa-openrouter/probe_tools.pyevals/moa-openrouter/record.pyevals/moa-openrouter/record_agentic.py
🚧 Files skipped from review as they are similar to previous changes (31)
- crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs
- crates/mesh-mixture-of-agents/src/tool_guard.rs
- crates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rs
- crates/mesh-mixture-of-agents/src/reducer.rs
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs
- evals/moa-openrouter/fanout.jsonl
- crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs
- crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs
- crates/mesh-mixture-of-agents/src/normalize.rs
- crates/mesh-mixture-of-agents/tests/sim_partial_worker_survival.rs
- crates/mesh-mixture-of-agents/tests/fixtures/committee_tasks.json
- crates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs
- crates/mesh-mixture-of-agents/tests/fixtures/ablation_tasks.json
- crates/mesh-mixture-of-agents/src/worker.rs
- crates/mesh-mixture-of-agents/src/backend.rs
- crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs
- crates/mesh-llm-host-runtime/src/network/openai/ingress.rs
- crates/mesh-mixture-of-agents/tests/fixtures/real_traces.json
- evals/moa-openrouter/agentic.jsonl
- crates/mesh-mixture-of-agents/tests/sim_real_traces.rs
- crates/mesh-mixture-of-agents/src/refinement.rs
- evals/moa-openrouter/corpus.jsonl
- crates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rs
- crates/mesh-mixture-of-agents/src/tool_turn.rs
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs
- crates/mesh-mixture-of-agents/src/context.rs
- crates/mesh-llm-guardrails/src/rescue.rs
- crates/mesh-mixture-of-agents/tests/eval_openrouter.rs
- crates/mesh-mixture-of-agents/src/lib.rs
- crates/mesh-mixture-of-agents/src/fanout.rs
- crates/mesh-mixture-of-agents/src/arbiter.rs
Comment on lines
+175
to
+196
| b_pt, b_k = point_estimate(tasks, "B") | ||
| c_pt, c_k = point_estimate(tasks, "C") | ||
| b_lo, b_hi = bootstrap_ci(tasks, "B", args.iters, args.seed) | ||
| c_lo, c_hi = bootstrap_ci(tasks, "C", args.iters, args.seed) | ||
|
|
||
| print(" net uplift = P(rescue) - P(harm), equal-weight mean over tasks") | ||
| print(f" B (real) uplift {b_pt:+.3f} 95% CI [{b_lo:+.3f}, {b_hi:+.3f}] (tasks n={b_k})") | ||
| print(f" C (shuffled) uplift {c_pt:+.3f} 95% CI [{c_lo:+.3f}, {c_hi:+.3f}] (tasks n={c_k})") | ||
| print(f" differential B-C: {b_pt - c_pt:+.3f} (content effect beyond token/prompt effect)") | ||
| print() | ||
|
|
||
| # Verdicts (directional; the CI is what matters for a claim). | ||
| if b_lo > 0: | ||
| print(" => references HELP: B net uplift CI is entirely > 0") | ||
| elif b_hi < 0: | ||
| print(" => references HARM: B net uplift CI is entirely < 0") | ||
| else: | ||
| print(" => inconclusive: B net uplift CI spans 0") | ||
| if b_pt - c_pt > 0 and b_lo > 0: | ||
| print(" => and the gain is CONTENT (B > C), not just extra tokens/prompt") | ||
| elif abs(b_pt - c_pt) < 0.02: | ||
| print(" => gain (if any) is NOT content-specific (B ~ C)") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Bootstrap the B−C differential before classifying a content effect.
Line 193 tests b_lo > 0, but b_lo is the lower confidence bound for B−A. It is not a confidence bound for B−C. This condition can report a content-specific gain when B−C is statistically uncertain.
Use the same paired task and draw resamples to calculate a B−C confidence interval. Base the content verdict on that interval.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/analyze_ablation.py` around lines 175 - 196, Update the
verdict logic in the analysis flow after the B/C estimates to bootstrap a paired
B−C differential using the same task resamples, producing its own confidence
interval. Base the “gain is CONTENT” classification on that differential
interval being entirely above zero, rather than on b_lo; retain the existing B−A
verdict and non-content-specific comparison separately.
| """ | ||
|
|
||
| def one(model): | ||
| resp, elapsed = oc.chat(model, messages, tools=tools, max_tokens=max_tokens, temperature=0.8) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Use the production MoA thinking setting.
oc.chat defaults no_think to False. Its contract states that no_think=True mirrors the MoA default. These calls can therefore measure reasoning-enabled workers and aggregators instead of the production configuration. Pass no_think=True; orclient.chat already retries without the flag for providers that require reasoning.
Proposed fix
- resp, elapsed = oc.chat(model, messages, tools=tools, max_tokens=max_tokens, temperature=0.8)
+ resp, elapsed = oc.chat(
+ model, messages, tools=tools, max_tokens=max_tokens, temperature=0.8, no_think=True
+ )
...
- aggregator, msgs, tools=TOOLS if with_tools else None, max_tokens=384, temperature=0.3
+ aggregator, msgs, tools=TOOLS if with_tools else None,
+ max_tokens=384, temperature=0.3, no_think=TrueAlso applies to: 135-137
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/probe_tools.py` at line 79, Update the oc.chat call in
the probe flow, including the corresponding calls around the additionally
affected lines, to pass no_think=True. Preserve the existing model, messages,
tools, token, and temperature arguments so all workers and aggregators use the
production MoA thinking configuration.
| }, | ||
| ] | ||
| res_b = summarize(fan_out(msgs_b, TOOLS), "B. TOOL-RESULT TURN (agentic step 2)") | ||
| record("fanout_tool_result", results=res_b) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Record the tool-result input conversation.
This record contains only results. It omits the user prompt, assistant tool call, tool-call ID, and tool result in msgs_b. A replay corpus cannot reconstruct this tool-result turn. Store messages=msgs_b with the result.
Proposed fix
- record("fanout_tool_result", results=res_b)
+ record("fanout_tool_result", messages=msgs_b, results=res_b)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| record("fanout_tool_result", results=res_b) | |
| record("fanout_tool_result", messages=msgs_b, results=res_b) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/probe_tools.py` at line 199, Update the
fanout_tool_result record call to store the complete tool-result conversation by
passing msgs_b as messages alongside the existing results payload. Preserve the
current record name and result data.
Comment on lines
+131
to
+140
| def consensus_tool(workers): | ||
| """Most-proposed (name, arguments) across workers, or None.""" | ||
| counts = {} | ||
| for w in workers: | ||
| for c in w.get("tool_calls") or []: | ||
| key = (c["function"]["name"], c["function"].get("arguments") or "{}") | ||
| counts[key] = counts.get(key, 0) + 1 | ||
| if not counts: | ||
| return None | ||
| return max(counts.items(), key=lambda kv: kv[1])[0] |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Normalize JSON arguments before counting tool-call consensus.
Equivalent argument objects can use different key order or whitespace. The current raw-string key splits those votes. A lower-support proposal can then control the canned observation and all later recorded steps.
Parse valid argument JSON and serialize it with stable key ordering before counting proposals. Keep malformed arguments in a separate raw fallback key.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/record_agentic.py` around lines 131 - 140, Update
consensus_tool to normalize valid JSON arguments before counting tool-call
proposals: parse each argument string, re-serialize parsed objects with stable
key ordering, and use that canonical form in the consensus key. Preserve
malformed arguments under a separate raw-string fallback key, while keeping tool
names and existing consensus selection behavior unchanged.
…laky peers
Agentic MoA turns now reliably produce tool calls, and diverging answers are
synthesized instead of relaying one arbitrarily-chosen worker's text.
Fixes found by replaying recorded traces from 9 open-weight models:
* Tools were withheld from workers. `query_uses_tools` was derived from
`looks_like_tool_intent`, an English keyword match on the user's text. "The
test suite is failing. Find out which test fails and why" matches nothing, so
workers were dispatched without tool schemas and the arbiter ran with
has_tools=false — a unanimous tool proposal then fell through to the answer
path and leaked "calling search" as prose. 5 of 10 recorded scenarios hit
this. Tool availability is now the caller's declaration.
* Argument outliers could win a unanimous tool call. Native tool calls are all
normalized to a fixed 0.9 confidence, so the confidence-only tiebreak could
not separate 8 workers proposing {"path":"src"} from one hallucinating
{"path":"rust_project/src"}. Added argument clustering (key-order
independent); the largest cluster wins and argument-free calls never outvote
filled-in ones.
* Early exit committed on tied arguments. Early exit runs on whoever has
arrived, so two fast models agreeing on a tool name with different arguments
was a coin flip that also aborted the workers who would have broken the tie.
It now waits.
* Truncation was invisible. `finish_reason` was never read; 39/140 recorded
responses came back "length" and 24 carried partial text, which parsed as a
normal answer and could be returned verbatim. Plumbed through as
BackendReply.truncated -> WorkerOutput.truncated; such answers are barred
from consensus and verbatim output but still feed synthesis, labelled
incomplete.
* A strict endpoint could kill a worker permanently. minimax-m2.5 returns
HTTP 400 "Reasoning is mandatory for this endpoint" and failed 12/12
requests; HttpBackend now drops the thinking flags and retries once.
Policy changes:
* Thinking is always off for MoA workers, not merely defaulted off. The
previous escape hatch only let callers request the broken configuration:
qwen3-32b spent 408 reasoning tokens against a 384-token cap and returned
null content. Ignored overrides are logged.
* Diverging answers go to the reducer. Returning the top-confidence answer was
near-arbitrary because models rarely emit our confidence envelope, so
everything defaults to 0.5 and max_by returns whichever worker finished
first. Agreement still short-circuits without paying for synthesis.
* Reducer prompt adopts Together's aggregator framing (synthesize, critically
evaluate, agreement is not proof), keeping our per-worker attribution,
structured tool proposals, and 500-char payload bound.
Tests: new sim_real_traces.rs replays 56 recorded cases from 9 models through
handle_turn, plus regression unit tests for each bug. Recorders and corpora
live in evals/moa-openrouter/.
…r acts Tool-bearing turns now use the Hermes/Nous-style asymmetric shape instead of a majority vote across workers: - references run TOOL-FREE and only advise in prose - the single best tool-caller (the "actor") acts on that advice with the real tools and emits the tool call Tool authority now tracks capability, not popularity. The old path let several weak models proposing a popular-but-wrong tool outvote the one strong tool-caller that picked correctly (observed: qwen3-32b alone chose run_command for a failing-test triage while the smaller models chose list_dir, and the vote shipped list_dir). An actor model removes that failure class instead of patching the vote arithmetic. Stays a pure stateless /v1/chat/completions turn — references are regenerated from the caller's transcript each request, and the external client still owns tool execution. Text-only turns are unchanged (symmetric fan-out + synthesis-on-divergence). Actor selection is capability-first: the host ranks candidates by gossiped `tool_use` level (Supported > Likely > None), then model size tier, then stable order, and passes the ordering to the engine via the new `GatewayConfig.actor_candidates`. Empty ordering falls back to the engine's name-derived size tier, so existing callers and tests are unaffected. Mesh guardrails for mixed/public meshes: references are gathered with a bounded wait (proceed at a majority of advisors, never block on the slow tail), and the actor is called through the existing hedged ladder so a slow/broken best candidate falls through to the next tool-capable peer. New: - crates/mesh-mixture-of-agents/src/tool_turn.rs — asymmetric tool-turn handler - context::pack_for_actor — "advise, then act" framing (vs synthesis framing) - fanout::gather_references — bounded, tool-free reference collection - GatewayConfig.actor_candidates + reducer_candidates honours it - host compute_actor_candidates from gossiped tool_use + size The dead majority-vote code in the arbiter (now unreachable — tool turns bypass it) is removed in the follow-up commit. Tests: eval_openrouter.rs adds a live OpenRouter harness (ignored by default) that runs the real handle_turn over 6 open-weight models with mesh-realism latency/failure injection, for benchmarking asymmetric MoA vs single models. 192 engine tests pass; clippy -D warnings and fmt clean on both crates.
Tool turns now take the asymmetric actor path (previous commit), so the arbiter's tool-proposal voting is unreachable — and it *was* the majority-of-weakness bug: a popular-but-wrong tool choice from several weak models could outvote the one strong tool-caller that picked correctly. Removes it rather than leaving dead code: - delete best_tool_proposal, best_tool_proposal_by_consensus, and decisive_argument_cluster - drop the tool-arbitration and tool-vs-answer branches from arbitrate, and the tool-consensus branch from try_early_decision - drop the now-unused has_tools parameter from arbitrate, try_early_decision, single_output_decision, and gather_workers_incremental - the arbiter is now purely an answer/critique/uncertainty arbiter; tool-shaped text on the text path is already demoted to Uncertainty by enforce_tool_call_contract (tools disabled ⇒ empty allow-list) - drop the arbiter tests that pinned the removed tool-vote behavior; keep and re-point every answer-consensus, truncation, synthesis, and tier-gate test No behavior change for text turns. 178 engine tests pass; host-runtime lib tests (1851) pass; clippy -D warnings and fmt clean on mesh-mixture-of-agents, mesh-llm-host-runtime, and mesh-llm.
Replaces the confounded "MoA vs best single model" comparison with a within-actor ablation that isolates the one variable that matters: the references. One pinned actor, identical sampling / token budget / system prompt across all arms; only the advice changes: A. actor alone (no references) B. actor + real references (the production tool path) C. actor + shuffled references (advice from a different task, length-similar) Metric: rescue (A✗→B✓) minus harm (A✓→B✗). Arm C is the key control — it separates "useful information" from "extra tokens + a think-carefully prompt". Why the change: the earlier best-single comparison ran solo models at different sampling (0.8/512) than the actor (0.3/2048) and with a different system prompt, and scored transient 429/502/504 and tool-unsupported endpoints as capability failures. Those numbers measured sampling+prompt+infra, not the design, and are not citable. This ablation removes all of that by construction: the same scorer hits all three arms of the same actor, so an imperfect label largely cancels in the rescue-minus-harm delta. Also: - chat_completion_retrying: predeclared retry on transient infra errors, which are then excluded (∅) from the capability analysis rather than scored as wrong answers. - MOA_ABLATION_ACTOR / MOA_ABLATION_DRAWS env overrides. - the old best-single test is kept but documented as a confounded smoke test, not evidence. Pilot findings (5 draws, 4 tasks, ignored/live): - Strong actor (qwen3-32b): 20/20 all arms — aces every task alone, so references are inert-but-harmless (no headroom to measure rescue). - Weak actor (qwen3-8b): A 15/20, B 20/20, C 15/20 — net uplift +5, all on the one task with headroom (triage). Real advice rescued the weak actor every draw; SHUFFLED advice did not (C=A), so the gain is advice CONTENT, not token count or the decision prompt. Zero harm. Reading: reference value tracks actor-alone headroom — a safety net when the actor is weak, dead weight (not damage) when strong. Suggests gating the reference phase on actor strength. This is a directional pilot, not the merge-blocking study (that needs ~40 preregistered stratified tasks, k>=10 draws, a paired hierarchical bootstrap CI, and the production-selected actor).
…yzer The defensible version of the pilot, committed BEFORE running so the labels are preregistered and can't be seen to chase results. - tests/fixtures/ablation_tasks.json: 40 preregistered tasks, 4 strata x 10 (inspect / search / execute / no_tool). Set-valued accept labels (accept_tools is a SET; empty = "no tool call"), with optional arg substring constraints — addresses the brittle single-tool label from the pilot. - ablation_scaled_study: loads the fixture, runs A/B/C arms at k draws (default 10) with bounded concurrency, writes one JSONL trial per (draw,task,arm) to MOA_ABLATION_OUT. Live measurement only; no stats here. - evals/moa-openrouter/analyze_ablation.py: deterministic paired HIERARCHICAL bootstrap — resample tasks within stratum, then draws within task — for a 95% CI on net uplift = P(rescue) - P(harm), plus the shuffled-arm control (content effect = B_uplift - C_uplift). Validated on synthetic data: 3/40 rescue tasks -> +0.075 point estimate, shuffled +0.000, and the CI honestly spans 0 when rescue is sparse (no over-claiming). Arms (one pinned actor, identical sampling/prompt; only references vary): A actor alone · B actor + real references · C actor + shuffled references. Env: MOA_ABLATION_ACTOR, MOA_ABLATION_DRAWS, MOA_ABLATION_CONCURRENCY, MOA_ABLATION_OUT. Ignored by default (live network + cost).
Cut the essay-length doc/inline comments added with the actor design down to the non-obvious "why". No code change. - tool_turn.rs: 26-line module preamble -> 6 lines; inline comments tightened (20% -> 10% comment density) - context.rs: pack_for_actor docstring 16 -> 5 lines; reducer synthesis-framing block 14 -> 6 lines - workers.rs: compute_actor_candidates docstring 21 -> 6 lines (kept the terse sort-key comments, which are load-bearing)
Two more ablation arms to answer "can peers help tool selection at all?", reusing the A/B/C JSONL schema + analyze_ablation.py. matched_peer_structured_study — do similar-strength, different-family peers help a fixed finalizer via STRUCTURED candidate tool calls (not prose)? A solo · B diverse (2 different-family peers) · C homogeneous (2 resamples of the finalizer's own model). B-C isolates cross-family diversity from extra sampling; also records oracle-union. correction_rescues_weak_tool_caller — the mesh scenario neither Hermes nor Together handles: a weak tool-caller with no strong peer. Tests correction of the CONCRETE drafted call instead of pre-hoc advice. A draft-alone · B deterministic (schema-validate + re-prompt on structural failure) · C semantic (different-family critic reviews the concrete call, finalizer revises once). Findings (live, directional): every tool-selection intervention was inert-to-harmful vs routing to a capable model. Pre-hoc structured proposals were flat (37-39/40 all arms). Deterministic correction fired ~never (qwen3-8b already drafts structurally valid calls ~95%); the residual failures are semantic (wrong tool choice), which neither validation nor a strong critic fixed because the revision still runs through the weak actor. Conclusion: tool selection is a "best capable model acts" task; MoA's value is on the answer/chat path (untested).
Tests where MoA's value should live per Together's validated claim: open-ended answer QUALITY on realistic agent-session turns (reason-over-tool-output, planning, explanation) — not tool selection. Fixed aggregator; only its input varies: A alone · B committee (synthesize 3 diverse-peer drafts, 1 round) C layered (peers refine seeing each other first, then synthesize — Together's `layers`) Judged pairwise by an out-of-pool different-family judge (gpt-4o-mini), position-swapped (win only if consistent both orders), output lengths logged. - tests/fixtures/committee_tasks.json: 15 realistic reasoning/answer turns - committee_beats_solo_on_reasoning: writes per-trial JSONL Pilot findings (2 draws, live; DIRECTIONAL, underpowered): - committee(B) vs solo(A): win 6 / tie 2 / loss 2 — positive but sign-test p=0.29, NOT significant at n=10. Every B win was also the longer answer, so length is not cleanly ruled out. - layered(C) vs committee(B): loss 6 / tie 2 / win 2 — Together's extra refinement round is negative value on these tasks, at extra cost. - 20/30 trials skipped because the aggregator (qwen3-32b) returned empty content (reasoning-budget exhaustion — the content:null bug Hermes' troubleshooting.md documents). A flaky aggregator degrades the committee; the instrument needs empty-output handling before a real run. Net: first directionally-positive result for MoA in this investigation, but unproven. Contrast with 5 tool-selection experiments that were all null-to-harmful. Supports the task-split: route tools to one caller, convene a (single-round) committee on reasoning turns.
…l transcript Our references were packed very differently from Hermes', and it was measurably costly. Adds `pack_for_reference`, which gives advisors only the conversation's user/assistant prose: - strips the agent system prompt (an advisor told "you are a coding agent, run the tests" role-plays the actor instead of advising it) - strips the tool transcript (prior tool_calls + results anchored every advisor on the trajectory already taken, collapsing the error-independence that makes aggregation worth anything) - drops the "respond with your best answer or tool call" instruction: advisors hold no schemas, so asking for a tool call yields tool-shaped prose — exactly the advice that pulled the actor off its own better choice - uniform view across advisors (no per-role trimming), so the packing is a stable function of history and caches across iterations - caps advisor output at 600 tokens (advisor generation dominates turn latency; the turn waits for the slowest advisor) Head-to-head on the same preregistered study (strong actor qwen3-32b, 40 tasks x 10 draws, identical everything except packing): packing B pass net uplift 95% CI original 359/400 -0.102 [-0.170, -0.045] hermes 385/400 -0.037 [-0.090, -0.003] Harm cut by ~64%. Per-category: search -14 -> -2, execute -25 -> -13, inspect -2 -> 0. The content-specific component (B-C differential) fell from -0.075 to -0.015, i.e. with correct packing the residual harm is no longer mostly "bad advice content". Honest read: most of the harm I previously attributed to "references" was an artifact of how WE packed them, not a property of reference-based MoA. The direction still stands though — even correctly packed, references remain a small but statistically real regression for a STRONG actor on tool selection (CI still entirely below zero). Adds 4 unit tests pinning the packing contract (no tool-call request, no system prompt leak, no tool-transcript leak, prose preserved), and MOA_REFERENCE_PACKING=hermes to select the style in the eval harness.
…t failed Writes up every live study in one place (evals/moa-openrouter/RESULTS.md) so the conclusions and their caveats survive the investigation. Headline 2x2 (40 preregistered tasks x 10 draws, paired hierarchical bootstrap): actor packing B pass net uplift 95% CI strong original 359/400 -0.102 [-0.170, -0.045] <- the bug strong hermes 385/400 -0.037 [-0.090, -0.003] weak original 365/400 -0.013 [-0.090, +0.070] weak hermes 377/400 +0.017 [-0.053, +0.100] Two monotonic effects: fixing the packing helps in both actor conditions, and references are worth more to a weaker actor. The only significant cell is the original-packing strong-actor harm — i.e. our bug, not a property of MoA. Per-stratum (weak + hermes) shows references help exactly where the actor has headroom (search +10, execute +4) and hurt where it was already perfect (inspect -7). That is a gating signal. Also records what did NOT work for tool selection (pre-hoc structured proposals: flat; deterministic correction: never fires, the weak actor already emits valid calls ~95%; semantic correction: negative, since the revision still runs through the weak actor), and the committee pilot on reasoning turns (directionally positive, n=10, not significant; Together's layering loses to single-round).
The Hermes-style packing was measured and unit-tested but only wired into the eval harness — the production tool path still used pack_for_worker_selected, the exact packing measured at -0.102 net uplift. tool_turn now calls pack_for_reference: conversation prose only, no agent system prompt, no tool transcript, no request for a tool call. Same head-to-head (strong actor, 40 tasks x 10 draws) puts this at -0.037 vs -0.102, and it is the only configuration where references show a positive point estimate for a weak actor (+0.017). See evals/moa-openrouter/RESULTS.md.
…no-upside Arm C (32B x2 + one 8B, weak node admitted) vs arm B (32B x2): B 48W 23T 2L p=2e-12 decided winrate 96% C 50W 25T 5L p=2e-10 decided winrate 91% Fisher B-vs-C p=0.44 (not separable at n=80) but one-way: admitting the weak node never helped and raised losses 2 -> 5. Both still beat solo, so a weak node doesn't collapse the pool, but it adds cost for no upside and a small tail risk. This is the measured basis for tier-based apply_admission_control.
You asked the right question: we showed admitting a weak node HURTS when the pool already has two strong (arm C), but never tested where admitting HELPS. The missing cell — one strong + one weak — is decisive and overturns the first-cut rule: 32B + 8B layered vs solo 32B: 47W 27T 5L p=1.3e-9 length-clean (r=-0.01) Admission control rejected the 8B here, collapsing the pool to a solo 32B. But the mixed committee beats solo decisively, so rejecting it threw away MoA in exactly the core "modest node joins a strong node and helps" case. Rule corrected: drop small-tier workers only when >=2 big-tier remain (a real committee survives). If dropping would collapse to solo, keep the mix. 32B x2 + 8B -> drop 8B (committee survives; 8B adds nothing, arm C) 32B + 8B -> keep both (dropping = solo, and the mix wins 47/5) Two admission tests updated/added to pin both cases. 69 moa_gateway tests pass; clippy -D warnings and fmt clean.
Before: a `model=mesh` request on a lone node (or a mesh with <2 workers)
returned 503 "MoA requires ≥2 models". That makes `mesh` unusable as a default
model — it only worked once a committee could form.
Now: when `build_moa_config` can't form a committee but a real model is
available, `try_handle_moa` rewrites `model=mesh` to that model and hands the
stream back to normal single-model routing. `mesh` works everywhere —
passthrough on one node, committee once a second worker joins. Only a node with
no model at all still 503s.
Both call sites handle the new fall-through:
- passive/transport path already routes a returned `Some(stream)` normally;
- host/ingress path: `try_handle_moa_intercept` gained a `Degraded { stream,
model }` result. The pre-computed `decision.effective_model` is stale
("mesh") after degradation, so the degraded model name is threaded into
`route_request`, and the pipeline classifier (computed against "mesh") is
skipped for degraded turns.
Unchanged from main was the hard 503; this is the missing piece for `mesh` as a
superior always-on drop-in. 1859 host tests pass; clippy -D warnings and fmt
clean.
Fan-out cost is ~2N+1 model calls per turn (N drafts + N refines + 1 synthesis), and measured quality is flat past ~4 workers while latency and spend keep climbing. On a big shared mesh — say 20 nodes — an uncapped pool fanned out to all of them: 41 calls for no quality gain. `cap_committee` now trims the assembled pool to the best MAX_COMMITTEE_WORKERS (4) by the same capability ranking used for actor selection (gossiped tool_use, then size, then stable index). Nodes beyond the cap are standbys — they still serve direct traffic, just not this committee. Runs after admission control and self-fill, so the cap applies to the final worker set. Extracted as a helper to keep assemble_worker_pool under the cognitive-complexity limit. 69 moa_gateway tests pass; clippy -D warnings and fmt clean.
sim_all_workers_fail covers total failure (clean structured error). This covers the common mesh reality: nodes flicker, so a subset of the committee dies mid-turn while the rest answer. MoA must degrade to survivors, not the error path. Three cases pinned: - half the committee dies, rest answer -> turn completes, all 4 accounted (2 succeeded, 2 recorded failed, none silently dropped) - lone survivor (3 of 4 die) -> turn still answers - slow-failing worker -> does not stall the turn past the survivors Locks the pre-existing best-effort fanout behaviour against regression now that assembly (admission, self-fill, cap) is more complex. 214 moa tests pass; clippy -D warnings and fmt clean.
workers.rs had grown to 1215 lines (over the 1k guideline) mixing two responsibilities: worker-pool assembly/selection and the model backends + config orchestration. Per CodeRabbit review, extract the assembly half. pool.rs (658 lines) now owns: assemble_worker_pool, resolve_one_worker_from_ aliases, add_worker_backend, group_aliases_by_canonical_base, is_locally_served, apply_admission_control, self_fill_from_extra_instances, cap_committee, compute_actor_candidates, canonical_base_name, WorkerBackendResolution, the two size caps, and their tests. workers.rs (578 lines) keeps: build_moa_config (orchestrator), the thinking-override helpers, patience_profile, the Local/Remote model backends + parse_quic_http_response, and their tests. Pure mechanical move, no behavior change. Backends and canonical_base_name made pub(super) for cross-module use. 69 moa_gateway tests pass (unchanged); clippy -D warnings and fmt clean.
`turn_completes_when_some_workers_die` asserted exactly succeeded==2/failed==2, but early-exit consensus can abort a live worker once a usable answer is in hand, so the succeeded/aborted split is timing-dependent. It passed locally but raced red in CI (Rust crate tests shard 0). The robustness contract is unchanged and still pinned: all four dispatched workers appear in summaries (none silently dropped) and the two dead ones are recorded as failed (>=2). Dropped the exact success-count assertion. Stable across 5 consecutive local runs.
michaelneale
force-pushed
the
feat/moa-synthesis
branch
from
August 4, 2026 08:45
01d717d to
2309bca
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@evals/moa-openrouter/RESULTS.md`:
- Around line 164-180: Reconcile the trial-count inconsistency in the mesh-study
section by either changing “40 prompts × 3 draws” to match the reported n=80, or
explicitly documenting which 40 trials were excluded and the reason. Keep the
win/tie/loss totals and statistical results consistent with the final stated
sample size.
- Around line 23-38: Correct the significance summary in the results discussion
to acknowledge that the strong-actor Hermes-style cell is statistically
significant harm because its interval excludes zero. Update the statement that
no post-fix cell is significant, or explicitly document the adjusted statistical
test and its basis if that is intended to change the conclusion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: b6a3251c-8915-4646-b0b2-a9fbd22aa488
📒 Files selected for processing (40)
crates/mesh-llm-guardrails/src/rescue.rscrates/mesh-llm-host-runtime/src/network/openai/ingress.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rscrates/mesh-mixture-of-agents/src/arbiter.rscrates/mesh-mixture-of-agents/src/backend.rscrates/mesh-mixture-of-agents/src/context.rscrates/mesh-mixture-of-agents/src/fanout.rscrates/mesh-mixture-of-agents/src/lib.rscrates/mesh-mixture-of-agents/src/normalize.rscrates/mesh-mixture-of-agents/src/reducer.rscrates/mesh-mixture-of-agents/src/refinement.rscrates/mesh-mixture-of-agents/src/tool_guard.rscrates/mesh-mixture-of-agents/src/tool_turn.rscrates/mesh-mixture-of-agents/src/worker.rscrates/mesh-mixture-of-agents/tests/eval_openrouter.rscrates/mesh-mixture-of-agents/tests/fixtures/ablation_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/committee_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/real_traces.jsoncrates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rscrates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rscrates/mesh-mixture-of-agents/tests/sim_partial_worker_survival.rscrates/mesh-mixture-of-agents/tests/sim_real_traces.rscrates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rscrates/mesh-mixture-of-agents/tests/sim_strong_patience.rscrates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rscrates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rscrates/mesh-mixture-of-agents/tests/sim_worker_accounting.rsevals/moa-openrouter/README.mdevals/moa-openrouter/RESULTS.mdevals/moa-openrouter/agentic.jsonlevals/moa-openrouter/analyze_ablation.pyevals/moa-openrouter/corpus.jsonlevals/moa-openrouter/fanout.jsonlevals/moa-openrouter/make_fixture.pyevals/moa-openrouter/orclient.pyevals/moa-openrouter/probe_tools.pyevals/moa-openrouter/record.pyevals/moa-openrouter/record_agentic.py
🚧 Files skipped from review as they are similar to previous changes (32)
- crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs
- crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs
- crates/mesh-llm-host-runtime/src/network/openai/ingress.rs
- crates/mesh-llm-guardrails/src/rescue.rs
- crates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rs
- crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs
- crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs
- crates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs
- crates/mesh-mixture-of-agents/src/normalize.rs
- crates/mesh-mixture-of-agents/src/worker.rs
- crates/mesh-mixture-of-agents/src/tool_guard.rs
- crates/mesh-mixture-of-agents/src/backend.rs
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs
- crates/mesh-mixture-of-agents/tests/sim_partial_worker_survival.rs
- crates/mesh-mixture-of-agents/tests/fixtures/real_traces.json
- evals/moa-openrouter/agentic.jsonl
- crates/mesh-mixture-of-agents/src/fanout.rs
- crates/mesh-mixture-of-agents/tests/fixtures/committee_tasks.json
- crates/mesh-mixture-of-agents/src/arbiter.rs
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rs
- evals/moa-openrouter/corpus.jsonl
- crates/mesh-mixture-of-agents/tests/eval_openrouter.rs
- evals/moa-openrouter/fanout.jsonl
- crates/mesh-mixture-of-agents/tests/fixtures/ablation_tasks.json
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs
- crates/mesh-mixture-of-agents/src/reducer.rs
- crates/mesh-mixture-of-agents/src/refinement.rs
- crates/mesh-mixture-of-agents/src/context.rs
- crates/mesh-mixture-of-agents/src/lib.rs
- crates/mesh-mixture-of-agents/src/tool_turn.rs
- crates/mesh-mixture-of-agents/tests/sim_real_traces.rs
- crates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rs
Comment on lines
+23
to
+38
| | strong (qwen3-32b) | original | 359/400 | −0.102 | [−0.170, −0.045] | | ||
| | strong (qwen3-32b) | Hermes-style | 385/400 | −0.037 | [−0.090, −0.003] | | ||
| | weak (qwen3-8b) | original | 365/400 | −0.013 | [−0.090, +0.070] | | ||
| | weak (qwen3-8b) | Hermes-style | **377/400** | **+0.017** | [−0.053, +0.100] | | ||
|
|
||
| Two monotonic effects: | ||
|
|
||
| 1. **Fixing the packing helps in both actor conditions** (+0.065 strong, | ||
| +0.030 weak). | ||
| 2. **References are worth more to a weaker actor** (+0.054 weak-vs-strong at | ||
| matched packing). | ||
|
|
||
| The only *statistically significant* cell in the matrix is the original-packing | ||
| strong-actor harm — i.e. the bug. After the fix, nothing is significant: | ||
| strong is marginal (upper bound −0.003), weak is a positive point estimate with | ||
| a CI spanning zero. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Correct the significance conclusion.
The strong/Hermes interval is [-0.090, -0.003]. It excludes zero. The text then says no post-fix cell is significant. State that this cell is significant harm, or document the adjusted test that changes this conclusion.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/RESULTS.md` around lines 23 - 38, Correct the
significance summary in the results discussion to acknowledge that the
strong-actor Hermes-style cell is statistically significant harm because its
interval excludes zero. Update the statement that no post-fix cell is
significant, or explicitly document the adjusted statistical test and its basis
if that is intended to change the conclusion.
# Conflicts: # crates/mesh-llm-guardrails/src/rescue.rs # crates/mesh-mixture-of-agents/src/normalize.rs # crates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rs
`model=mesh` now works as a universal virtual model from one node upward: 1 node / 1 model -> serves that model (single-model passthrough) >=2 workers -> real MoA committee (fan-out + synthesis) 0 models -> 503 (correct: nothing to serve) Two bugs found and fixed against a live 2-node mesh (MacBook 3B + mini 1B): 1. request.raw not rewritten. degrade patched model_name/body_json but the forwarded request rides on request.raw bytes, which still said "mesh", so the embedded openai-frontend 404'd. Now uses rewrite_model_field (patches raw + body + Content-Length together; already unit-tested). 2. wrong model source. degrade read models_being_served(), empty on a fresh serve node. /v1/models draws from three sources; degrade now tries them in order: callable_models(targets) -> models_being_served() -> serving_models(). Live-verified: single node model=mesh returns HTTP 200 with a real answer (was 503/404); 2-node mesh model=mesh returns x-moa-turn=fanout workers=2. Known cold-start transient: the very first mesh request on a fresh serve node can 404 for ~1-2s until the routing table populates; solid once warm.
ndizazzo
self-requested a review
August 5, 2026 02:39
Addresses two i386 P1s: destructive admission and committee-cap decisions were
made from a name heuristic that treats every unparseable name as big-tier, so a
small fine-tune/alias (`my-assistant`) could bypass the weak-worker filter or
become the reducer — contradicting the branch's core "weak can't soil strong"
claim.
Producer (each serving node, has its own GGUF): scan_gguf_total_parameters sums
tensor element counts for the exact stored-parameter count and publishes it via
the existing ServedModelMetadata.parameter_count_b. profile.rs prefers this over
name parsing; missing => None (unknown), never a guess.
Consumer (MoA orchestrator, has NO peer GGUFs): reads the gossiped
parameter_count_b off served descriptors. New SizeTier {Small,Big,Unknown}:
* admission excludes only VERIFIED small, counts only VERIFIED big toward the
">=2 big remain" gate; Unknown is never strong and never filtered.
* cap_committee ranks by verified size, NOT compute_actor_candidates
(tool_use-first), which could evict a big model for tool-advertising small
ones on an answer turn.
Name parsing (model_param_size_b) remains a lower-confidence fallback only.
71 moa_gateway tests (2 new: unknown-never-filtered, name-parse fallback);
model-artifact + host clippy -D warnings and fmt clean.
Collaborator
Author
|
@i386 thanks - have it reading real data now |
Collaborator
Author
|
@i386 — fixed the tiering P1s you flagged (name-based tier / unparseable-name-as-big driving destructive admission). Now exactly your prescription: authoritative size from GGUF tensor sum, gossiped via the existing Commit: 1. Producer computes the real size — pub fn scan_gguf_total_parameters(path: &Path) -> Option<u64> {
let GgufHeader { file: mut f, n_tensors, n_kv } = open_gguf_header(path)?;
skip_all_kv_pairs(&mut f, n_kv)?;
let mut total: u64 = 0;
for _ in 0..n_tensors {
let _name = read_gguf_string(&mut f).ok()?;
let n_dims = read_u32(&mut f).ok()?;
if n_dims > MAX_GGUF_TENSOR_DIMS { return None; }
let mut elements: u64 = 1;
for _ in 0..n_dims {
let dim = read_u64(&mut f).ok()?;
elements = elements.checked_mul(dim)?;
}
let _ggml_type = read_u32(&mut f).ok()?;
let _offset = read_u64(&mut f).ok()?;
total = total.checked_add(elements)?;
}
Some(total)
}2. Producer prefers it, falls back to name — 3. Consumer tiers off the gossiped value — enum SizeTier { Small, Big, Unknown }
fn tier_for(name: &str, sizes: &HashMap<String, f64>) -> SizeTier {
if let Some(b) = sizes.get(&canonical_base_name(name)) {
return if *b < SMALL_TIER_MAX_B { SizeTier::Small } else { SizeTier::Big };
}
match mesh_llm_guardrails::model_param_size_b(name) { // name-parse fallback
Some(b) if (b as f64) < SMALL_TIER_MAX_B => SizeTier::Small,
Some(_) => SizeTier::Big,
None => SizeTier::Unknown, // missing = Unknown, not big
}
}Admission excludes only verified Verified live on a 2-node mesh (MacBook 3B + mini Qwen3.5-4B): gossiped sizes went from a bogus name-parsed 4910 to real 3.21 / 4.21, and admission now no-ops on unknown-size workers. Two new unit tests: Note deliberately deferred to follow-up: MoE active-parameter signal (total tensor count is the safety gate for now, as you said). Does this match what you had in mind? |
Collaborator
I think the fallback to name should be removed. if you cant get this info from gguf, the model must be ranked as the lowest param model. As would be worth chucking anything in your cache at this and checking if that tensor counts matches the name to validate. |
Per i386: parsing NNb from a served name is brittle and a destructive admission decision must not rest on an unverified label. Removed the name fallback entirely. Producer (profile.rs): parameter_count_b comes ONLY from the GGUF tensor sum. No local GGUF to sum ⇒ None (no guessed count). Deleted the now-dead parameter_count_b_from_text + its test. Consumer (pool.rs): tier_for uses only the gossiped verified size. Dropped SizeTier::Unknown — a model with no verified size is Small (weakest), so an unverifiable label can never masquerade as big and displace a real strong worker. cap_committee ranking updated (no-size ranks last). Admission excludes only verified-small; with no verified big there is nothing to protect so an all-unsized pool is untouched. Tests updated: unsized-worker-is-weakest (excluded next to verified big), no-verified-sizes keeps all. 71 moa_gateway + 28 profile tests pass; clippy -D warnings and fmt clean.
Collaborator
MeshLLM PR 1116 Homelab Validation Report
General FindingsPR 1116's MoA machinery works through the shipped OpenAI HTTP path on a real The PR is nevertheless NOT_READY for its headline claim that the shipped Two additional release defects were found: repository Scope And ExclusionsValidated:
Canonical Change Inventory And Validation Ledger
Environment And Build ProvenanceAll worktrees were created at
Product Validation Results
Private Mesh EvidenceFinal topology (left running):
API, Logs, UI, And Inference FindingsThe same-model early-exit request produced 2 worker summaries, matching the The tool request selected one actor and returned structurally correct arguments. No final log contained a structured Defects And Anomalies
Risk And Follow-Up Register
Final Gate Assessment
Sign-OffDecision: NOT_READY for merge/release on the stated quality-improvement |
Self-fill could re-add the same physical node (or an endpoint already backing the sole worker), turning ONE box into a fake 2-worker committee that hits it twice for near-identical drafts. Catastrophic for mesh mode: a lone node must degrade to single-model serving, never pretend to be a committee. Rewrote self_fill_from_extra_instances to build the pool from DISTINCT physical endpoints only: the local skippy port (if this node serves the model and context fits) plus each distinct remote peer from hosts_for_model. If fewer than two distinct endpoints serve the model, the pool stays the single worker and MoA degrades to single-model. A single endpoint can no longer appear twice. Also fixes the CodeRabbit self-fill bugs (re-adds same peer; skipped context eligibility) — context fit is now checked on the local endpoint, and endpoints are distinct by construction. 71 moa_gateway tests pass; clippy -D warnings and fmt clean.
A `model=mesh` request with no `messages`, a non-array `messages`, or an empty
array fell through to the workers and fabricated a 200 answer from nothing
(homelab validation API-1). try_handle_moa now requires a present, non-empty
`messages` array and returns 400 otherwise, before any model call.
Verified live: {"model":"mesh"}, string messages, and [] all now 400; a valid
request still returns 200. 71 moa_gateway tests pass; clippy -D warnings + fmt
clean.
Collaborator
Author
|
@ndizazzo the cuda thing seems unrelated, and can you try that again with non trivial model as i can't see that model yielding much at all, so probably results are too noisy. |
Collaborator
Author
|
@ndizazzo yeah the claim is only for larger models combining (20B ish) unfortunately, so need to test with that (still probing myself). Also want to make sure this improves over the main branch. but this reminds me to try it with larger combos... so standby.. (in the mean time feel free to try it with larger - and some subsequent fixes) |
Width sprint (evals/moa-openrouter, aggregator qwen3-8b, 8B peers, shipped committee path) shows the fan-out cap was throttling the pools that benefit most: 6× diverse 8B vs best member: 12W/65T/2L, p=0.013 (wins) 4× diverse 8B: 5W/73T/0L, p=0.06 (marginal) 2× diverse 8B: 2W/77T/1L, p=1.0 (null) Small, weak drafts need WIDTH — more independent proposals — before aggregation clears the best member. A verified big model, by contrast, wins at 2 and gains nothing past ~4. - committee_cap is now tier-aware: COMMITTEE_CAP_SMALL=6 for all-small pools, COMMITTEE_CAP_BIG=4 when a verified big model is present (replaces the flat MAX_COMMITTEE_WORKERS=4). - eval harness: the three per-trial judge comparisons now run concurrently (tokio::join!) instead of three serial awaits — ~3x faster judging phase, no behaviour change. Also measured (recorded in RESULTS.md): the refine round never beats single aggregation at 8B (refine-vs-single null in every cell) — Hermes' cheaper single-synth cadence matches Together's layered shape here; and at 8B diversity matters (6 diverse >> 6 same), unlike mid-scale. 72 moa_gateway tests (new: committee_cap_is_wide_for_small_pools_tight_for_big); clippy -D warnings and fmt clean.
Collaborator
Author
Update: width sprint + review fixes (since
|
| pool | single-agg vs best member | refine vs single-agg |
|---|---|---|
| 2× 8B diverse | 2W/77T/1L, p=1.0 | null |
| 4× 8B diverse | 5W/73T/0L, p=0.06 | null |
| 6× 8B diverse | 12W/65T/2L, p=0.013 | null |
| 6× 8B same | 4W/75T/1L, p=0.38 | null |
Three results:
- Six diverse 8B models beat the best single member (p=0.013). The old flat
MAX_COMMITTEE_WORKERS=4throttled exactly the small pools that need width. Cap is now tier-aware: 6 for all-small, 4 when a verified big model is present (a 24–32B pair already wins at 2). - The refine round never earns its serial cost —
refine vs single-aggis null in every cell. Hermes' single-aggregation cadence matches Together's layered shape here at half the latency. (Noted for follow-up; not changed in this PR.) - At 8B, diversity matters (6 diverse 12W/2L ≫ 6 same 4W/1L) — unlike mid-scale where Self ≈ Mixed.
Review fixes pushed
- i386 P1 ×2: tiering now uses verified GGUF tensor-sum size gossiped via
parameter_count_b(not name parsing);cap_committeeranks by size, nottool_use. Name fallback removed per follow-up — no verified size ⇒ ranked weakest. Validated: cached 3B→3.21, 4B→4.21;gemma-4-E4Bname says 4B but sums to 7.5B (proves the name heuristic was unsafe). - Iron law: self-fill rebuilt to use distinct physical endpoints only — a single node can never fake a 2-worker committee; it degrades to single-model. (Also closes the two CodeRabbit self-fill bugs.)
- API-1:
model=meshwith missing/non-array/emptymessagesnow returns 400, not a fabricated 200. Verified live. - CodeRabbit:
pool.rsextracted; eval judges parallelised.
Still open (agreed)
- Capable-model (24–32B) shipped-path run + second judge for the headline quality claim — needs paid runs, separate effort.
- aarch64
check-releaseparity + CUDA-toolkit-default: pre-existing, not touched by this branch (check-releasepasses on this HEAD; branch changes no release/packaging code).
Full data + method in evals/moa-openrouter/RESULTS.md.
…is dead cost The width sprint measured refine-vs-single-aggregation as null in every 8B cell (2/4/6 workers, diverse and same). Small pools win by WIDTH under a single aggregation, not by the extra serial refine pass. Previously RefinementPolicy:: Auto ran the round for ALL-small pools — paying two synthesis passes for no measured gain. Auto now refines only for a homogeneous pool that is NOT all-small (i.e. same-model at real scale, where correlated drafts + the round measured 48/2 vs 35/10). All-small and diverse pools skip it — matching Hermes' cheaper single-synth cadence, where refine buys nothing. Effect: an all-small mesh turn drops from 2 serial synthesis passes to 1, roughly halving added latency, with no measured quality loss. Tests: auto_skips_an_all_small_pool (was auto_refines_...); the 5 all-small mechanics sim tests (straggler/grace/degradation) pinned with Always so they still exercise the round; big-pool gate tests keep Auto. 178 + sim tests pass; clippy -D warnings and fmt clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ready for review. Every claim below is measured through real open-weight models; see
evals/moa-openrouter/RESULTS.mdfor method, numbers, and withdrawn results.Closes part of #1115.
What this delivers
MoA rebuilt on Together + Hermes research, tuned for a heterogeneous mesh: the virtual
meshmodel gets stronger as nodes join, without a weak node ever dragging it down.layers), then one aggregator synthesizes.The core result
A pool of ≥2 sufficiently-capable participants beats the single best member of that pool on reasoning turns — and the active ingredient is test-time ensembling, not model diversity:
Two samples of one model do as well as two different models (matches Self-MoA, arXiv:2502.00674). So a same-model mesh should get MoA too.
The three mesh changes
Also fixed (found by measuring the shipped path, not the harness)
Nine eval-vs-production divergences that each erased the gain until fixed (grace finalizing before refinement; reducer/refiner truncation discarding most of each answer; prompts that told workers to "be concise"/"be direct"; named vs anonymous reducer inputs; role-tier token starvation; arbiter shipping one worker's answer instead of synthesizing). Plus a real production panic in
mesh-llm-guardrails(byte-slicing on a multi-byte character), with regression tests.Two hypotheses were tested and rejected by measurement, then reverted — not rationalised.
Protocol / compatibility
No wire, gossip, plugin, or skippy-ABI changes. Actor ranking reads the existing gossiped
tool_usecapability. New behaviour is in-memory config (ReferencePolicy,RefinementPolicy). One caller-visible change:reasoning_effort/enable_thinkingon amodel=meshrequest is ignored (that override only selected a broken config where reasoning models returncontent: null).Validation
Live studies are
#[ignore]d and gated onOPENROUTER_API_KEY; CI skips them.Honest limits (for the reviewer)
real_traces.json,*.jsonl,RESULTS.md); the code surface is 13 files.Summary by CodeRabbit