Fix CI test order and tests (part 1) (#31)

* Fix CI test order - install util-genai before dependent packages

The test for opentelemetry-util-genai-emitters-splunk fails with:
ModuleNotFoundError: No module named 'opentelemetry.util.genai.emitters.spec'

This happens because emitters-splunk depends on opentelemetry-util-genai
being installed first (which contains the emitters.spec module).

Reordered the test steps to install and test the base util-genai package
before the packages that depend on it (emitters-splunk, evals, etc.).

(cherry picked from commit e8bfae2)

* Fix CI: Install all packages before running tests

Some tests have cross-dependencies between packages:
- util-genai tests import from util-genai-evals
- emitters-splunk tests import from util-genai

Changed strategy to install all packages first (with --no-deps),
then run each package's tests separately. This ensures all
inter-package dependencies are available during testing.

(cherry picked from commit c8a3bdd)

* Fix remaining undefined logger references in handler.py

Changed all 'logger' references to '_LOGGER' to match the module's
logger variable name (line 239 and 245). This fixes the NameError
that was occurring during test execution.

(cherry picked from commit d5b7953)

* Fix test mocking path for load_completion_callbacks

Tests were trying to mock 'handler._load_completion_callbacks' but
the function is actually 'utils.load_completion_callbacks' (imported
from utils module). Updated the mock patch paths to point to the
correct location.

(cherry picked from commit ccc7665)

* Fix emitters-splunk: use event_logger instead of content_logger in factories, update tests to match current implementation

(cherry picked from commit a1910cf)

* Fix test: correct evaluation metric name from toxicity_v1 to toxicity/v1

(cherry picked from commit d2a5711)

* Fix remaining mock paths and dynamic aggregation test in util-genai-evals

- Fix 4 instances of handler._load_completion_callbacks -> utils.load_completion_callbacks in test_evaluators.py
- Fix test_evaluation_dynamic_aggregation.py to set _aggregate_results to None instead of False to enable dynamic environment variable reading as per actual implementation

(cherry picked from commit a20742f)

* Fix CI test order - install util-genai before dependent packages

The test for opentelemetry-util-genai-emitters-splunk fails with:
ModuleNotFoundError: No module named 'opentelemetry.util.genai.emitters.spec'

This happens because emitters-splunk depends on opentelemetry-util-genai
being installed first (which contains the emitters.spec module).

Reordered the test steps to install and test the base util-genai package
before the packages that depend on it (emitters-splunk, evals, etc.).

(cherry picked from commit e8bfae2)

* Fix CI: Install all packages before running tests

Some tests have cross-dependencies between packages:
- util-genai tests import from util-genai-evals
- emitters-splunk tests import from util-genai

Changed strategy to install all packages first (with --no-deps),
then run each package's tests separately. This ensures all
inter-package dependencies are available during testing.

(cherry picked from commit c8a3bdd)

* Add test dependencies for langchain instrumentation

- Add [test] extra to pyproject.toml with langchain-core, langchain-openai, pytest-recording, vcrpy, pyyaml, flaky
- Update CI workflow to install [instruments,test] dependencies
- Fixes CI failure: ModuleNotFoundError: No module named 'langchain_openai'

* Upgrade flaky to >=3.8.1 for pytest 7.4.4 compatibility

Fixes ImportError: cannot import name 'call_runtest_hook' from 'flaky.flaky_pytest_plugin'

The flaky 3.7.0 version is incompatible with pytest 7.4.4 used in CI.
Upgrading to flaky>=3.8.1 resolves the compatibility issue.

* Fix deepeval test patching issues

- Fix incorrect patch targets for _instantiate_metrics and _run_deepeval
  * These are module-level functions, not class methods
  * Change from patch.object(class, method) to patch(module.function)
  * Update function signatures to match actual implementations

- Fix _build_llm_test_case lambda signatures
  * Function takes only 1 argument (invocation), not 2

- Fix test assertions for bias metric labels
  * 'Not Biased' for success=True (not 'pass')
  * 'Biased' for success=False (not 'fail')

- Fix default metrics test expectation
  * Remove 'faithfulness' from expected defaults
  * Actual defaults: bias, toxicity, answer_relevancy, hallucination, sentiment

All deepeval tests now pass (15 passed, 2 warnings)

* Fix langchain instrumentation callback handler tests

- Fixed LangchainCallbackHandler initialization to use telemetry_handler parameter
- Fixed _resolve_agent_name to not return 'agent' tag as agent name
- Added missing workflow methods to _StubTelemetryHandler (start_workflow, stop_workflow, fail_workflow, fail_by_run_id)
- Filter gen_ai.tool.* metadata from ToolCall attributes (stored in dedicated fields)
- Process invocation_params in on_chat_model_start:
  * Extract model_name from invocation_params with higher priority
  * Add invocation params with request_ prefix to attributes
  * Move ls_* metadata to langchain_legacy sub-dict
  * Set provider from ls_provider metadata
  * Add callback.name and callback.id from serialized data
- Configure pytest-recording (VCR) for cassette playback
- Fix vcr fixture scopes and cassette directory configuration

All 7 callback handler agent tests now pass.

* Fix langchain instrumentation test isolation issue - unwrap all wrapper layers

Root cause: The opentelemetry.instrumentation.utils.unwrap() function only
unwraps ONE layer of wrapt wrappers. When tests call uninstrument() and then
re-instrument(), the second wrapping creates nested wrappers. The unwrap only
removed the outer layer, leaving the old wrapper still active.

Fix: Modified _uninstrument() to unwrap ALL layers by looping while
__wrapped__ exists, ensuring complete cleanup before re-instrumentation.

Also added _callback_handler reference to instrumentor for test access.

Result: ALL 9 langchain tests now passing (was 8 passed, 1 skipped)

* Add response model extraction to callback_handler

- Extract response_model_name from generation.message.response_metadata
- Fixes test_langchain_llm.py and test_langchain_llm_util.py
- Ensures gen_ai.response.model attribute is set in spans

* Fix deepeval compatibility: make CacheConfig optional

- CacheConfig is only available in deepeval >= 3.7.0
- Use try-except import to support older versions
- Conditionally add cache_config to eval_kwargs
- Fixes CI ImportError on older deepeval versions

* Upgrade deepeval minimum version to 3.7.0

- Update pyproject.toml: deepeval>=0.21.0 -> deepeval>=3.7.0
- Revert deepeval_runner.py to original simple implementation
- CacheConfig is required in deepeval >= 3.7.0
- Removes compatibility code, cleaner solution

* Add CacheConfig to test stubs for deepeval

- Add CacheConfig class to stub modules in test files
- Update evaluate() stub signature to accept cache_config parameter
- Fixes CI import errors when using deepeval stubs
- Both test_deepeval_evaluator.py and test_deepeval_sentiment_metric.py updated

* Add runtime check for deepeval module availability

- Add check in run_evaluation to detect if deepeval is patched to None
- Raises ImportError when sys.modules['deepeval'] is None
- Fixes test_dependency_missing which patches deepeval to None
- Ensures proper error handling when dependency is missing

Previously, CacheConfig wasn't in the stubs, so import would fail.
Now that we added CacheConfig to stubs (for newer deepeval >= 3.7.0),
we need runtime check to handle the dependency_missing test case.

* Apply code formatting (auto-formatter cleanup)

- pyproject.toml: Format include list to single line
- deepeval_runner.py: Add blank lines for PEP 8 compliance
- test files: Format function parameters to multi-line style

No functional changes, formatting only.

* Fix ruff linting issues

- Remove unused imports (MagicMock, TracerProvider)
- Sort imports in test_splunk_emitters.py
- Remove trailing whitespace

* Apply ruff formatting

* Remove CacheConfig import from deepeval_runner

- CacheConfig parameter has default value in deepeval.evaluate()
- No need to import or pass it explicitly
- Works with all deepeval versions (>=0.21.0)
- Simpler code without conditional imports

* Complete CacheConfig cleanup and fix flaky test

- Remove CacheConfig from deepeval_runner.py (use default parameter)
- Remove sys.modules runtime check (no longer needed)
- Remove CacheConfig stubs from test files
- Remove cache_config parameter from stub evaluate() functions
- Fix flaky timestamp test: use >= instead of > for end_time
  (Windows CI can have identical timestamps in same nanosecond)

* Revert production code to main branch

Revert to main branch versions:
- deepeval_runner.py (keep CacheConfig import/usage)
- langchain/callback_handler.py
- langchain/__init__.py

These changes will be submitted in a separate MR.
Test file improvements are kept in this branch.

* ruff++

---------

Co-authored-by: Sergey Sergeev <[email protected]>

Author: shuningc

.github/workflows/ci-main.yaml

@@ -22,43 +22,51 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest, macos-latest]
-        python-version: ['3.10', '3.11', '3.12', '3.13']
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
 
     steps:
-    - uses: actions/checkout@v5
+      - uses: actions/checkout@v5
 
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v6
-      with:
-        python-version: ${{ matrix.python-version }}
+      - name: Enable long paths on Windows
+        if: runner.os == 'Windows'
+        run: |
+          git config --system core.longpaths true
 
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install pytest==7.4.4 pytest-cov==4.1.0
-        pip install -r dev-genai-requirements.txt
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python-version }}
 
-    - name: Run tests - opentelemetry-util-genai-emitters-splunk
-      run: |
-        pip install -e util/opentelemetry-util-genai-emitters-splunk --no-deps
-        python -m pytest util/opentelemetry-util-genai-emitters-splunk/tests/ -v
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install pytest==7.4.4 pytest-cov==4.1.0
+          pip install -r dev-genai-requirements.txt
 
-    - name: Run tests - opentelemetry-util-genai-evals
-      run: |
-        pip install -e util/opentelemetry-util-genai-evals --no-deps
-        python -m pytest util/opentelemetry-util-genai-evals/tests/ -v
+      - name: Install all genai packages
+        run: |
+          pip install -e util/opentelemetry-util-genai --no-deps
+          pip install -e util/opentelemetry-util-genai-evals --no-deps
+          pip install -e util/opentelemetry-util-genai-evals-deepeval --no-deps
+          pip install -e util/opentelemetry-util-genai-emitters-splunk --no-deps
+          pip install -e "instrumentation-genai/opentelemetry-instrumentation-langchain[instruments,test]"
 
-    - name: Run tests - opentelemetry-util-genai-evals-deepeval
-      run: |
-        pip install -e util/opentelemetry-util-genai-evals-deepeval --no-deps
-        python -m pytest util/opentelemetry-util-genai-evals-deepeval/tests/ -v
+      - name: Run tests - opentelemetry-util-genai
+        run: |
+          python -m pytest util/opentelemetry-util-genai/tests/ -v --cov=opentelemetry.util.genai --cov-report=term-missing
 
-    - name: Run tests - opentelemetry-instrumentation-langchain
-      run: |
-        pip install -e instrumentation-genai/opentelemetry-instrumentation-langchain --no-deps
-        python -m pytest instrumentation-genai/opentelemetry-instrumentation-langchain/tests/ -v
+      - name: Run tests - opentelemetry-util-genai-emitters-splunk
+        run: |
+          python -m pytest util/opentelemetry-util-genai-emitters-splunk/tests/ -v
 
-    - name: Run tests - opentelemetry-util-genai
-      run: |
-        pip install -e util/opentelemetry-util-genai --no-deps
-        python -m pytest util/opentelemetry-util-genai/tests/ -v --cov=opentelemetry.util.genai --cov-report=term-missing
+      - name: Run tests - opentelemetry-util-genai-evals
+        run: |
+          python -m pytest util/opentelemetry-util-genai-evals/tests/ -v
+
+      - name: Run tests - opentelemetry-util-genai-evals-deepeval
+        run: |
+          python -m pytest util/opentelemetry-util-genai-evals-deepeval/tests/ -v
+
+      - name: Run tests - opentelemetry-instrumentation-langchain
+        run: |
+          python -m pytest instrumentation-genai/opentelemetry-instrumentation-langchain/tests/ -v

dev-genai-requirements.txt

@@ -12,7 +12,7 @@ markupsafe>=2.0.1
 codespell==2.1.0
 requests==2.32.3
 ruamel.yaml==0.17.21
-flaky==3.7.0
+flaky>=3.8.1
 pre-commit==3.7.0; python_version >= '3.9'
 pre-commit==3.5.0; python_version < '3.9'
 ruff==0.6.9

dev-requirements.txt

@@ -12,7 +12,7 @@ markupsafe>=2.0.1
 codespell==2.1.0
 requests==2.32.4
 ruamel.yaml==0.17.21
-flaky==3.7.0
+flaky>=3.8.1
 pre-commit==3.7.0; python_version >= '3.9'
 pre-commit==3.5.0; python_version < '3.9'
 ruff==0.6.9

instrumentation-genai/opentelemetry-instrumentation-langchain/examples/multi_agent_travel_planner/client_server_version/client.py

@@ -98,21 +98,21 @@ def run_client(
     poison_config = None
     if use_poison:
         poison_config = generate_random_poison_config()
-        print(f"\n💉 Poison Configuration:")
+        print("\n💉 Poison Configuration:")
         print(f"  Probability: {poison_config['prob']}")
         print(f"  Types: {', '.join(poison_config['types'])}")
         print(f"  Max snippets: {poison_config['max']}")
         print(f"  Seed: {poison_config['seed']}")
 
     # Generate user request
     user_request = generate_travel_request(origin, destination)
-    print(f"\n✉️  User Request:")
+    print("\n✉️  User Request:")
     print(f"  {user_request}")
 
     # Get server URL from environment or default to localhost
     server_url = os.getenv("SERVER_URL", "http://localhost:8080")
 
-    print(f"\n🔌 Connecting to Flask server...")
+    print("\n🔌 Connecting to Flask server...")
     print(f"  URL: {server_url}")
 
     # Prepare request data
@@ -149,31 +149,31 @@ def run_client(
         print(f"👥 Travellers: {result['travellers']}")
 
         if result.get('poison_events'):
-            print(f"\n💉 Poison Events Triggered:")
+            print("\n💉 Poison Events Triggered:")
             for event in result['poison_events']:
                 print(f"  - {event}")
 
-        print(f"\n✈️  Flight Summary:")
+        print("\n✈️  Flight Summary:")
         print(f"  {result['flight_summary']}")
 
-        print(f"\n🏨 Hotel Summary:")
+        print("\n🏨 Hotel Summary:")
         print(f"  {result['hotel_summary']}")
 
-        print(f"\n🎭 Activities Summary:")
+        print("\n🎭 Activities Summary:")
         print(f"  {result['activities_summary']}")
 
-        print(f"\n🎉 Final Itinerary:")
+        print("\n🎉 Final Itinerary:")
         print("─" * 60)
         print(result['final_itinerary'])
         print("─" * 60)
 
         if result.get('agent_steps'):
-            print(f"\n🤖 Agent Steps:")
+            print("\n🤖 Agent Steps:")
             for step in result['agent_steps']:
                 print(f"  - {step['agent']}: {step['status']}")
 
     except requests.exceptions.Timeout:
-        print(f"\n❌ Error: Request timed out after 5 minutes")
+        print("\n❌ Error: Request timed out after 5 minutes")
         sys.exit(1)
     except requests.exceptions.RequestException as e:
         print(f"\n❌ Error: Failed to connect to server: {e}")
@@ -184,7 +184,7 @@ def run_client(
         sys.exit(1)
     except KeyError as e:
         print(f"\n❌ Error: Missing key in response: {e}")
-        print(f"Response:")
+        print("Response:")
         pprint(result)
         sys.exit(1)
 

instrumentation-genai/opentelemetry-instrumentation-langchain/examples/multi_agent_travel_planner/client_server_version/main.py

@@ -190,17 +190,15 @@
 
 from __future__ import annotations
 
-import argparse
 import json
 import os
 import random
 import sys
 from datetime import datetime, timedelta
-from typing import Annotated, Any, Dict, List, Optional, TypedDict
+from typing import Annotated, Dict, List, Optional, TypedDict
 from uuid import uuid4
 from pprint import pprint
 
-from dotenv import load_dotenv
 from flask import Flask, request, jsonify
 from langchain_core.messages import (
     AIMessage,
@@ -220,7 +218,7 @@
 
 from opentelemetry.sdk.trace import TracerProvider
 from opentelemetry.sdk.trace.export import BatchSpanProcessor
-from opentelemetry.trace import SpanKind, Status, StatusCode, Tracer
+from opentelemetry.trace import SpanKind
 from opentelemetry import _events, _logs, metrics, trace
 from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
 from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
@@ -529,7 +527,7 @@ def pretty_print_message(message, indent=False):
 
         indented = "\n".join("\t" + c for c in pretty_message.split("\n"))
         print(indented, file=sys.stderr, flush=True)
-    except Exception as e:
+    except Exception:
         # Fallback if pretty_repr fails
         print(f"Message: {message}", file=sys.stderr, flush=True)
 
@@ -985,7 +983,7 @@ def plan():
             poison_config=poison_config,
         )
 
-        print(f"[SERVER] Travel plan completed successfully", file=sys.stderr, flush=True)
+        print("[SERVER] Travel plan completed successfully", file=sys.stderr, flush=True)
         print("\n" + "="*80, file=sys.stderr)
         print("TRAVEL PLAN RESULT:", file=sys.stderr)
         pprint(result, stream=sys.stderr)

instrumentation-genai/opentelemetry-instrumentation-langchain/pyproject.toml

@@ -34,6 +34,14 @@ dependencies = [
 instruments = [
   "langchain >= 0.3.21",
 ]
+test = [
+  "langchain-core >= 1.0.0",
+  "langchain-openai >= 1.0.0",
+  "pytest-recording >= 0.13.0",
+  "vcrpy >= 7.0.0",
+  "pyyaml >= 6.0.0",
+  "flaky >= 3.8.1",
+]
 
 [project.entry-points.opentelemetry_instrumentor]
 langchain = "opentelemetry.instrumentation.langchain:LangChainInstrumentor"

instrumentation-genai/opentelemetry-instrumentation-langchain/tests/conftest.py

@@ -138,7 +138,7 @@ def chatOpenAI_client():
     return ChatOpenAI()
 
 
-@pytest.fixture(scope="module")
+@pytest.fixture(scope="function")
 def vcr_config():
     return {
         "filter_headers": [
@@ -149,9 +149,19 @@ def vcr_config():
         ],
         "decode_compressed_response": True,
         "before_record_response": scrub_response_headers,
+        "serializer": "yaml",
     }
 
 
+@pytest.fixture(scope="session")
+def vcr_cassette_dir():
+    """Override the default cassette directory to avoid nested subdirectories."""
+    import os
+
+    # Return the cassettes directory path
+    return os.path.join(os.path.dirname(__file__), "cassettes")
+
+
 @pytest.fixture(scope="function")
 def instrument_no_content(tracer_provider, event_logger_provider, meter_provider):
     if LangChainInstrumentor is None:  # pragma: no cover - skip when dependency missing
@@ -175,16 +185,35 @@ def instrument_with_content(tracer_provider, event_logger_provider, meter_provid
     if LangChainInstrumentor is None:  # pragma: no cover
         pytest.skip("opentelemetry-instrumentation-langchain not available")
     set_prompt_capture_enabled(True)
+
+    # Reset util-genai singleton handler to ensure clean state
+    import opentelemetry.util.genai.handler as _util_handler_mod  # noqa: PLC0415
+
+    if hasattr(_util_handler_mod.get_telemetry_handler, "_default_handler"):
+        setattr(_util_handler_mod.get_telemetry_handler, "_default_handler", None)
+
+    # Create new instrumentor for each test
     instrumentor = LangChainInstrumentor()
+
+    # If already instrumented (from previous test), uninstrument first
+    if instrumentor._is_instrumented_by_opentelemetry:
+        instrumentor.uninstrument()
+
     instrumentor.instrument(
         tracer_provider=tracer_provider,
         event_logger_provider=event_logger_provider,
         meter_provider=meter_provider,
     )
 
     yield instrumentor
+
     set_prompt_capture_enabled(True)
-    instrumentor.uninstrument()
+    # Clean up: uninstrument and reset singleton
+    if instrumentor._is_instrumented_by_opentelemetry:
+        instrumentor.uninstrument()
+
+    if hasattr(_util_handler_mod.get_telemetry_handler, "_default_handler"):
+        setattr(_util_handler_mod.get_telemetry_handler, "_default_handler", None)
 
 
 @pytest.fixture(scope="function")
@@ -222,21 +251,37 @@ def instrument_with_content_util(
             OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT: "SPAN_ONLY",  # util-genai content gate
         }
     )
+
     # Reset singleton so new env vars are applied
     import opentelemetry.util.genai.handler as _util_handler_mod  # noqa: PLC0415
 
     if hasattr(_util_handler_mod.get_telemetry_handler, "_default_handler"):
         setattr(_util_handler_mod.get_telemetry_handler, "_default_handler", None)
+
+    # Create new instrumentor for each test
     instrumentor = LangChainInstrumentor()
+
+    # If already instrumented (from previous test), uninstrument first
+    if instrumentor._is_instrumented_by_opentelemetry:
+        instrumentor.uninstrument()
+
     instrumentor.instrument(
         tracer_provider=tracer_provider,
         event_logger_provider=event_logger_provider,
         meter_provider=meter_provider,
     )
+
     yield instrumentor
+
     os.environ.pop(OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, None)
     set_prompt_capture_enabled(True)
-    instrumentor.uninstrument()
+
+    # Clean up: uninstrument and reset singleton
+    if instrumentor._is_instrumented_by_opentelemetry:
+        instrumentor.uninstrument()
+
+    if hasattr(_util_handler_mod.get_telemetry_handler, "_default_handler"):
+        setattr(_util_handler_mod.get_telemetry_handler, "_default_handler", None)
 
 
 class LiteralBlockScalar(str):
@@ -305,6 +350,11 @@ def deserialize(cassette_string):
 
 try:  # pragma: no cover - optional pytest-vcr dependency
     import pytest_recording  # type: ignore # noqa: F401
+    import vcr as vcr_module  # type: ignore # noqa: F401
+
+    # Register custom YAML serializer globally
+    vcr_module.VCR().register_serializer("yaml", PrettyPrintJSONBody)
+
 except ModuleNotFoundError:  # pragma: no cover - provide stub when plugin missing
 
     @pytest.fixture(name="vcr", scope="module")
@@ -316,9 +366,10 @@ def register_serializer(self, *_args, **_kwargs):
         return _VCRStub()
 
 
-@pytest.fixture(scope="module", autouse=True)
+@pytest.fixture(scope="function", autouse=True)
 def fixture_vcr(vcr):
-    vcr.register_serializer("yaml", PrettyPrintJSONBody)
+    # When pytest-recording is installed, vcr is a Cassette and we don't need to do anything
+    # The serializer is already registered on the VCR module above
     return vcr