Docs: ai-agent-with-tools tutorial references unavailable Weather tool modules

[ortegacmanuel]

Problem

The AI Agent with Tools livebook tutorial references Jido.Tools.Weather.* modules as if they ship with the jido or jido_ai hex packages:

• Jido.Tools.Weather.Geocode
• Jido.Tools.Weather.LocationToGrid
• Jido.Tools.Weather.Forecast
• Jido.Tools.Weather.CurrentConditions

The docs state: "Jido ships weather tools that wrap the free NWS (National Weather Service) API."

However, these modules were removed from jido_action and currently only exist as fallback/example code in jido_ai's examples/lib/compat/weather_tools_fallback.ex. Since examples/ is excluded from hex packages, users following the tutorial get UndefinedFunctionError when trying to use them.

Additional issue found

Even after copying the tools into a project, the LLM agent crashes because the tool schemas use {:in, [:detailed, :summary]} (atom enums), but LLM tool calls send string values ("summary"). The schema validation rejects them with Jido.Action.Error.InvalidInputError, which then crashes the ReAct loop because that error struct doesn't implement Jason.Encoder.

Suggested fixes

1. Update the livebook to either:
   • Point users to copy the weather tools from jido_ai/examples/lib/compat/weather_tools_fallback.ex, or
   • Include the tool definitions inline in the tutorial, or
   • Note that users need {:jido_ai, github: "agentjido/jido_ai"} (git dep, not hex) and must compile the examples path
2. Fix the enum schemas in the example tools to accept strings (since LLMs always send JSON string values, not Elixir atoms)
3. Consider publishing the weather tools in a hex package (e.g. jido_action or a new jido_tools package)

Steps to reproduce

1. Follow the tutorial at https://jido.run/docs/learn/ai-agent-with-tools
2. Add {:jido, "~> 2.1"} and {:jido_ai, "~> 2.0"} to mix.exs
3. Run Jido.Tools.Weather.Geocode.run(%{location: "Denver, CO"}, %{}) in iex
4. Get UndefinedFunctionError

Environment

• jido 2.2.0
• jido_ai 2.1.0
• jido_action 2.2.0
• Elixir 1.18.1 / OTP 27

[ortegacmanuel]

Additional issue: Inspecting tool activity section is misleading

The guide's "Inspect tool activity" section suggests this code to see which tools the agent called:

tool_activity =
  case Jido.AgentServer.status(pid) do
    {:ok, status} ->
      %{
        result: status.snapshot.result,
        tool_calls: status.snapshot.details[:tool_calls] || [],
        model: status.snapshot.details[:model]
      }
  end

However, status.snapshot.details[:tool_calls] only contains pending (in-flight) tool calls. Once the agent completes, pending_tool_calls is cleared to [], and then the empty list is filtered out entirely by Enum.reject in build_snapshot_details/2 (strategy.ex:418).

So users following the guide will always see tool_calls: [] after a successful run, even though tools were clearly used (visible in the log output).

To actually inspect tool activity after completion, users should look at:

{:ok, status} = Jido.AgentServer.status(pid)
# Full conversation history including tool calls and results
status.snapshot.details[:conversation]
# Trace summary showing event counts per request
status.snapshot.details[:trace_summary]

The guide should either:

• Update the code snippet to use :conversation to show tool call history
• Or note that :tool_calls only reflects in-flight state, not history

---

Summary of all issues found following this guide

1. Weather tool modules don't exist in published hex packages (jido, jido_ai, jido_action) — they're only in jido_ai/examples/lib/compat/ which is excluded from hex
2. Enum schemas crash with LLM agents — tool schemas use {:in, [:detailed, :summary]} but LLMs send JSON strings ("summary"), causing InvalidInputError which then crashes the ReAct loop due to missing Jason.Encoder implementation
3. Tool activity inspection returns empty results — snapshot.details[:tool_calls] is always [] after completion since it only tracks pending calls

[nshkrdotcom]

Thanks for the thorough issue, @ortegacmanuel. This was already a valid docs bug, and your follow-up comment about tool activity exposed a second important problem in the tutorial itself.

After reading the original report and the update here:
#117 (comment)

I think the strongest long-term solution is not to keep the current weather-based AI agent with tools tutorial and just patch the wording. The more robust, architecturally correct path is to separate concerns more cleanly across this repo and the package repos.

What this issue now shows

At this point there are really three distinct problems bundled together:

1. The tutorial references weather tool modules that are not actually available in the published Hex flow

   • Jido.Tools.Weather.Geocode
   • Jido.Tools.Weather.LocationToGrid
   • Jido.Tools.Weather.Forecast
   • Jido.Tools.Weather.CurrentConditions

   Those references make the tutorial appear to be using stable, shipped package modules, but for a user installing released jido / jido_ai from Hex, that is not true.

2. The framework/tool boundary appears to have a real JSON/schema mismatch

   • tools declare atom enums such as {:in, [:detailed, :summary]}
   • LLM tool calls arrive as JSON strings such as "summary"
   • validation rejects them
   • the resulting error is not serialized safely, which crashes the ReAct loop instead of turning into a normal tool failure

3. The tutorial’s inspection guidance is misleading for completed runs

   • status.snapshot.details[:tool_calls] appears to reflect pending/in-flight calls
   • after a successful run, the tutorial implies users will see historical tool usage there
   • in practice they get [], even though the model did call tools
   • the correct post-run inspection surfaces appear to be :conversation and :trace_summary

So this is now bigger than just “fix one broken code sample.” It is a combination of:

• docs drift
• package boundary confusion
• a framework behavior gap at the JSON tool-call boundary
• and a runtime-observability/docs mismatch around completed tool activity

Recommended architecture going forward

My recommendation is:

1. Make /docs/learn/ai-agent-with-tools a generic, self-contained ReAct tutorial
2. Keep weather as a separate domain guide
3. Fix enum coercion / serializable tool errors / snapshot semantics in jido_ai and/or jido_action
4. Add anti-drift tests so this class of docs mismatch does not recur

I think that is the best long-term shape for the system.

Why I do not think the long-term answer is “just publish weather tools and leave this tutorial as-is”

Publishing reusable weather tools may still be worthwhile later as an optional ecosystem package, but I do not think the foundational “learn tool calling” tutorial should depend on that decision.

A foundational tutorial should optimize for:

• conceptual clarity
• self-containment
• low setup friction
• stable behavior across releases
• low dependency on external service quirks
• easy CI drift detection

The current weather-based tutorial does the opposite in several ways:

• it couples a core ReAct teaching page to a domain-specific integration
• it depends on public APIs and the details of a weather workflow
• it creates confusion about what ships in core vs optional packages
• it makes it harder to keep the tutorial aligned with released packages

This repo already has a better home for weather-specific content in:
priv/pages/docs/guides/building-a-weather-agent.livemd

That page is a domain guide. That is exactly where weather-specific examples belong.

Proposed long-term plan

1. Re-architect the docs in jido_run

A. Rewrite AI agent with tools as a generic tool-calling tutorial

/docs/learn/ai-agent-with-tools should teach:

• what a Jido.Action tool is
• how the ReAct loop works
• how to define tool schemas
• how the LLM selects and calls tools
• how to inspect a finished run correctly

It should not be the primary place where users learn a weather/NWS integration.

Instead, the tutorial should use a small set of notebook-local tools that are:

• self-contained
• easy to understand
• stable
• fast
• ideally not network-dependent
• still multi-step enough to demonstrate tool chaining

For example, a simple tool chain could be something like:

• normalize_location
• lookup_trip_conditions
• packing_advice
• convert_temperature

Or:

• lookup_city_profile
• compare_temperature_units
• recommend_gear

The exact example matters less than the architectural property: the tutorial should stand on its own and teach tool-calling without depending on undeclared package modules.

B. Keep weather in the weather guide

/docs/guides/building-a-weather-agent is already the better location for:

• real public APIs
• external service headers and fetch logic
• geocoding + grid lookup + forecast chaining
• domain-specific operational concerns

If reusable packaged weather tools are published later, that guide can optionally mention them. But the foundational learn page should stay independent.

C. Fix the “Inspect tool activity” section immediately

The updated issue comment is important here.

If status.snapshot.details[:tool_calls] only represents pending/in-flight tool calls, then the current tutorial is misleading users when it implies they can inspect completed tool history there.

The tutorial should be updated to inspect completed runs using:

• status.snapshot.details[:conversation]
• status.snapshot.details[:trace_summary]

And the docs should explicitly say something along the lines of:

• :tool_calls reflects transient in-flight state during execution
• :conversation contains the full conversation history, including tool calls and tool results
• :trace_summary gives a compact post-run summary

That distinction is important, because otherwise users follow the tutorial, see tool_calls: [], and conclude that either the docs are wrong or the runtime is broken.

D. Remove all claims that released Hex packages ship the weather tools

Anywhere in this repo that presents Jido.Tools.Weather.* as generally available from released package installs should be corrected.

That includes at least:

• the tutorial page itself
• test assertions that currently lock in those references
• any marketing/getting-started snippets that still show those modules as if they are the default path
• ecosystem/package descriptions that currently overstate packaged weather-tool availability

2. Fix the framework/package behavior in jido_ai and/or jido_action

This is the part that should not be solved in docs alone.

A. Fix enum handling at the LLM tool boundary

This seems like a real framework ergonomics bug.

At the LLM boundary, tool arguments arrive as JSON. That means enum-like values naturally arrive as strings. If an action schema declares atoms, the framework should handle that boundary safely and predictably.

The best long-term behavior would be:

• action declares an enum like [:detailed, :summary]
• provider sends "summary"
• tool execution path safely coerces to the existing atom :summary
• invalid values still fail validation in a normal, controlled way

That coercion should happen centrally, not be pushed onto every tutorial or every action author.

Otherwise every example author has to remember “never declare atom enums for LLM-facing tools,” which is exactly the sort of hidden rule that keeps causing drift and breakage.

B. Make tool/input validation failures safely serializable

A validation failure should never crash the ReAct loop because an exception struct or error struct does not implement Jason.Encoder.

The strategy/runtime should normalize those failures into a stable serializable error payload, for example a simple map or another deliberately encoded structure.

Desired behavior:

• invalid tool args do not crash the agent loop
• the failure is surfaced clearly to the caller and/or the LLM
• the run remains observable and debuggable

That is a framework correctness issue, not just a docs issue.

C. Clarify snapshot semantics around pending vs historical tool activity

The issue comment strongly suggests an API/documentation mismatch in snapshot semantics.

If :tool_calls is intentionally “currently pending tool calls,” that is fine, but then:

• docs need to say that clearly
• the field name may be too easy to misread
• and/or the runtime should expose a better explicit completed-history field for post-run inspection

Possible improvements:

• keep :tool_calls, but document it as pending-only
• rename it if that is feasible in the package API
• add a dedicated completed tool history or summary field
• make :conversation / :trace_summary the clearly documented inspection surfaces for finished runs

Right now the name/usage combination is too easy for users to misunderstand.

D. Add regression tests in the package repos

The package repos should have explicit coverage for:

• JSON string enum values coming from tool calls
• invalid enum values producing structured, serializable failures
• completed runs exposing inspectable history via the intended snapshot fields
• pending tool-call state being distinguished from completed tool history

Without those tests, this class of problem is likely to recur.

3. Add anti-drift contract tests in jido_run

This repo already has the right instinct here: docs are being checked against released package metadata rather than GitHub main assumptions.

That approach should be extended.

A. Add stronger docs assertions for package availability claims

If a docs page says a module or capability ships in released packages, there should be a contract test or metadata validation guarding that claim.

This issue exists because the docs made a stronger claim than the released packages supported.

B. Add assertions around completed-run inspection guidance

The docs should not regress back to encouraging :tool_calls as historical inspection for completed runs.

A test should guard that the tutorial uses the intended post-run surfaces.

C. Prefer runnable drift tests for foundational tutorials

If AI agent with tools can be rewritten to avoid fragile external dependencies, I would strongly prefer promoting it from reference-only to runnable CI coverage.

Foundational tutorials are exactly the docs that most need executable drift detection.

Recommended phased delivery

Phase 1: Immediate corrections in jido_run

1. Correct the tutorial so it no longer references unavailable Jido.Tools.Weather.* modules as part of the released Hex path
2. Fix the inspection section to use :conversation and :trace_summary
3. Remove or correct stale snippets elsewhere in the repo that imply those weather modules are generally available
4. Update ecosystem metadata/docs that currently overstate weather-tool availability
5. Adjust tests accordingly

Phase 2: Structural docs improvement in jido_run

1. Rewrite AI agent with tools as a generic self-contained tool-calling tutorial
2. Keep weather-specific walkthroughs in building-a-weather-agent
3. If possible, make AI agent with tools runnable in CI after removing the weather/API dependency

Phase 3: Framework/package fixes in jido_ai / jido_action

1. Add central handling for string enum values at the JSON tool-call boundary
2. Ensure validation failures become serializable runtime/tool errors instead of crashing the loop
3. Clarify or improve snapshot semantics for completed vs pending tool activity
4. Add regression tests for all of the above

Phase 4: Optional ecosystem packaging

If the project still wants reusable weather tools, publish them as a dedicated optional package with clear ownership and versioning. But I would treat that as a separate ecosystem/product decision, not as a prerequisite for fixing this tutorial correctly.

My opinionated summary

The most robust and architecturally correct path is:

• Do not keep the foundational “AI agent with tools” page weather-specific
• Do not rely on GitHub-only or examples-only modules in released docs
• Do fix the framework’s JSON enum and error-serialization behavior centrally
• Do correct the inspection guidance to use completed-run history surfaces
• Do add stronger anti-drift tests so docs and released packages stay aligned

Thanks again, @ortegacmanuel, for the careful report and the follow-up investigation. The extra note about :tool_calls being pending-only materially changes what the best long-term fix looks like.

@mikehostetler could you review/comment on this proposed direction, especially the package-level changes in jido_ai / jido_action around:

• JSON string enum coercion for LLM tool calls
• serializable validation/tool errors
• and clearer snapshot semantics for pending vs completed tool activity?

[nshkrdotcom]

Mike provided clarification that narrows the scope. I agree with the narrower scope for this issue (fewer tokens to fix, after all).

I was aiming for the long-term architectural shape, but for this ticket the right plan is simpler:

Updated plan for this issue

1. Update this Livebook directly

   • Remove the broken assumption that Jido.Tools.Weather.* is available from the published Hex packages
   • Replace those references with a self-contained, notebook-local implementation or otherwise inline the working example code needed for the tutorial
   • Keep the tutorial focused on getting the existing page correct rather than reworking the docs structure more broadly

2. Fix the inspection section in this Livebook

   • Stop telling readers to inspect status.snapshot.details[:tool_calls] after a completed run
   • Use the completed-run inspection surfaces that actually reflect history, such as :conversation and :trace_summary
   • Add a short note that :tool_calls is transient/in-flight state, not completed history

3. Update the example schemas in this Livebook so they match actual LLM tool-call behavior

   • Avoid enum examples that will fail when the LLM sends JSON strings
   • If the tutorial includes enum-like tool params, make the example accept the values in the format the runtime will actually receive

4. Add or update tests in jido_run

   • Update the existing page-content assertions for this Livebook
   • Add assertions that the corrected inspection guidance is present
   • Make sure the test suite guards against reintroducing the unavailable weather-module references

5. Ship the docs fix

   • This issue should be closed by a concrete docs/test patch in jido_run

Follow-up work that should be tracked separately

I still think there are likely real follow-ups outside the scope of this issue, but they should be handled independently:

• package/runtime handling for string enum values in LLM tool calls
• serialization of validation errors so they do not crash the ReAct loop
• clearer semantics/documentation around pending vs completed tool activity

Those are worth discussing, but I agree they should not block fixing and shipping this Livebook.

So the scoped version is: update the page, test the page, and ship the page.