fix(parser): recover tool calls that have a trailing comma

[statxc]

Summary

The tool-call parser mis-handled a trailing comma — one of the most common malformed-JSON quirks local models emit. {"tool":"set_timer","arguments":{"seconds":300},} was parsed as a phantom tool named "seconds", and a trailing comma in an array dropped every call after the first. This recovers the correct call instead.

Changes

• extract_embedded_json advanced by a single byte after a balanced candidate failed serde_json validation, so it resumed scanning inside the object it had just rejected — and returned an inner fragment of the real call.
• When a balanced top-level candidate fails to parse, extract_balanced_json_candidate now retries once after stripping trailing commas via a string-aware pass (strip_trailing_commas), so commas inside string values are preserved. Valid JSON still returns unchanged on the first attempt; genuinely-invalid JSON (e.g. {"seconds":60*60*12}, issue fix(tools): don't leak unparsed tool-call JSON to the user (closes #378) #380) still returns None and is caught by is_unparsed_tool_call.

Real Behavior Proof

• I have built and run the affected code locally (or noted why I could not).
• I have NOT verified on Jetson hardware, and I explain the equivalent verification path or validation gap below.

Tested profile / hardware (check all that apply):

• laptop

This is a deterministic, hardware-independent change to tools/parser.rs (the shared tool-call extraction used by both the runtime path and the BFCL eval path). No Jetson, audio, model, or network dependency — an x86_64 Linux dev box exercises the exact code path that runs on-device. Because the same parser backs parse_tool_calls_for_eval, this directly affects BFCL scoring fidelity: a trailing-comma response that previously scored as a wrong/missing call now scores as the correct call.

What I ran

On x86_64 Linux (Ubuntu, kernel 6.8), rustc 1.96.0:

cargo test -p genie-core
cargo test -p genie-core --lib tools::parser
cargo clippy -p genie-core --tests
cargo clippy -p genie-core --no-default-features --tests
cargo fmt -p genie-core -- --check

I added the four regression tests first and confirmed three of them fail on the current code (proving the bug) before applying the fix.

What I observed

• Before the fix, the new tests failed exactly as the bug predicts:
  • extract_recovers_trailing_comma_object: left: "seconds" vs right: "set_timer".
  • extract_recovers_trailing_comma_array: left: 1 vs right: 2 (second call dropped).
  • extract_trailing_comma_recovery_preserves_string_commas: panicked (no JSON extracted).
• After the fix: cargo test -p genie-core — 734 passed, 0 failed (plus integration/doc suites green). The parser module: 21 passed.
• extract_leaves_valid_json_untouched confirms valid JSON is returned byte-for-byte unchanged (no behavior change on the happy path).
• clippy clean in both default and --no-default-features; cargo fmt --check clean.

Test plan

1. cargo test -p genie-core --lib tools::parser — all 21 pass, including the four new extract_* tests.
2. Feed {"tool":"set_timer","arguments":{"seconds":300},} through parse_tool_calls_for_eval and confirm one set_timer call with seconds = 300 (not a "seconds" tool).

[matedev01]

Approve ✅

Clean, correct recovery. strip_trailing_commas is properly string- and escape-aware: " toggles in_string, a \ inside a string sets escape so \" doesn't close the string, and a comma is only dropped when its next non-whitespace char is }/] outside a string. It's also conservative — the repaired candidate is only returned when stripping actually changed something and the result parses, so it can't "recover" into a different call. And it's pure additions, no churn in the existing scan path.

The tests are on point: trailing comma in an object, trailing comma in an array (the "drops every call after the first" case), and the important one — "a, b,}" proving in-string commas and braces survive untouched — plus the valid-JSON-untouched guard.

Verified locally (pure parser string logic, no arch sensitivity): cargo test -p genie-core --lib tools::parser → 28 passed (the 4 new + existing, no regression).

One tiny non-blocking idea: a case with an escaped quote next to a trailing comma (e.g. {"x":"a\",}",}) would explicitly pin the escape handling — the code already gets it right, this would just lock it.

LGTM — needs the catch-up with main (currently behind).