fix: Propagate model and base URL in LLMCallException; improve error handling

[JashG]

Description

An area of improvement we'd like to address in the NeMo Guardrails microservice is error messages. Specifically, they tend to lack enough context and are not actionable.

This PR contains a few changes around error handling:

• Propagate relevant context in error messages. Specifically, since a config can invoke multiple models hosted at different URLs, it'd be helpful for the error message to include the model, base URL, and most relevant error.
• Add a few custom exceptions to enable consumers to better handle specific errors.

Related Issue(s)

Checklist

• I've read the CONTRIBUTING guidelines.
• I've updated the documentation if applicable.
• I've added tests if applicable.
• @mentions of the person or team responsible for reviewing proposed changes.

[greptile-apps]

Greptile Overview

Greptile Summary

This PR improves error handling in NeMo Guardrails by adding context to error messages and introducing custom exception types.

Key Changes:

• Created new exception hierarchy (ConfigurationError, InvalidModelConfigurationError, InvalidRailsConfigurationError) to enable better error handling by consumers
• Enhanced LLMCallException to include model name and endpoint URL context when LLM calls fail
• Added _raise_llm_call_exception() helper that extracts model/endpoint info from multiple provider attribute patterns (OpenAI, Azure, TRT-LLM, etc.)
• Replaced generic ValueError exceptions throughout config validation with specific exception types
• Improved error messages to be more descriptive and actionable (e.g., showing available model types when validation fails)
• Added comprehensive test coverage for exception enrichment across different LLM providers

Impact:
The changes significantly improve debuggability when working with multiple models/endpoints in a single config. Error messages now clearly identify which model and endpoint failed, making it easier to diagnose issues in production environments.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The changes are well-structured, focused on error handling improvements, and include comprehensive test coverage. The code maintains backward compatibility while adding helpful context to error messages. The refactoring from generic exceptions to specific exception types is a best practice that improves code maintainability.
• No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
nemoguardrails/exceptions.py	5/5	New exception hierarchy with LLMCallException now accepting optional context message for better error reporting
nemoguardrails/actions/llm/utils.py	5/5	Added _raise_llm_call_exception helper that enriches exceptions with model name and endpoint URL from multiple provider attribute patterns
nemoguardrails/rails/llm/config.py	5/5	Replaced generic ValueError with specific InvalidModelConfigurationError and InvalidRailsConfigurationError for better error handling and clearer messages
nemoguardrails/rails/llm/llmrails.py	5/5	Updated to use new exception types and improved colang version validation using dictionary lookup pattern
tests/test_actions_llm_utils.py	5/5	Added comprehensive test coverage for LLM exception enrichment with model/endpoint context across different provider types

Sequence Diagram

sequenceDiagram
    participant User
    participant LLMRails
    participant ActionDispatcher
    participant LLMUtils
    participant LLMProvider
    participant ExceptionHandler

    User->>LLMRails: generate_async(messages)
    LLMRails->>ActionDispatcher: execute_action()
    ActionDispatcher->>LLMUtils: llm_call(llm, prompt)
    LLMUtils->>LLMUtils: _invoke_with_string_prompt/message_list()
    
    alt LLM Call Succeeds
        LLMUtils->>LLMProvider: llm.ainvoke(prompt)
        LLMProvider-->>LLMUtils: response
        LLMUtils-->>ActionDispatcher: response
        ActionDispatcher-->>LLMRails: result
        LLMRails-->>User: generated response
    else LLM Call Fails
        LLMUtils->>LLMProvider: llm.ainvoke(prompt)
        LLMProvider-->>LLMUtils: Exception (e.g., ConnectionError)
        LLMUtils->>LLMUtils: _raise_llm_call_exception(exception, llm)
        LLMUtils->>LLMUtils: Extract model_name from llm_call_info_var or _infer_model_name()
        LLMUtils->>LLMUtils: Extract endpoint_url from BASE_URL_ATTRIBUTES or client.base_url
        LLMUtils->>ExceptionHandler: raise LLMCallException(exception, context_message)
        ExceptionHandler-->>ActionDispatcher: LLMCallException with enriched context
        ActionDispatcher->>ActionDispatcher: catch LLMCallException and re-raise
        ActionDispatcher-->>LLMRails: LLMCallException
        LLMRails-->>User: Error with model and endpoint details
    end

Loading

[greptile-apps]

Additional Comments (2)

1. tests/test_embeddings_only_user_messages.py, line 21 (link)

   syntax: Import needs updating to use the new location

2. tests/test_tool_calling_utils.py, line 22 (link)

   syntax: Import needs updating - LLMCallException was moved

_{8 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile
React with 👍 or 👎 to share your feedback on this new summary format}

[greptile-apps]

_{10 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile
React with 👍 or 👎 to share your feedback on this new summary format}

[codecov]

Codecov Report

❌ Patch coverage is 90.41096% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
nemoguardrails/rails/llm/config.py	78.94%	4 Missing ⚠️
nemoguardrails/rails/llm/llmrails.py	81.81%	2 Missing ⚠️
nemoguardrails/actions/llm/utils.py	96.42%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

_{12 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile
React with 👍 or 👎 to share your feedback on this new summary format}

[greptile-apps]

_{12 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{13 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{13 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{11 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{10 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}