Skip to content

validation_framework.md

Latest commit

 

History

History
396 lines (304 loc) · 11.7 KB

validation_framework.md

File metadata and controls

396 lines (304 loc) · 11.7 KB

Validation Framework

Overview

The validation framework prevents biochemical hallucinations in media recommendations through three layers of defense:

  1. Layer 1: Generation-Time Constraints (Preventive)

    • Block problematic recommendations during agent execution
    • Enforce biochemical rules before output generation

  2. Layer 2: Post-Generation Validation (Reactive)

    • Comprehensive fact-checking of all recommendations
    • Cross-reference against design constraints

  3. Layer 3: Human Review Triggers (Safety Net)

    • Flag high-risk recommendations for expert review
    • Provide contradiction reports and evidence

Motivation: The Calcium Hallucination Error

In versions v7 and v10 of the media design system, calcium chloride was incorrectly recommended at 10-100 µM (v7) and 0.1-10 mM (v10) for lanthanide depletion experiments. The system claimed calcium:

  • "Acts as XoxF cofactor enhancer"
  • "Acts synergistically with lanthanides"

This was biochemically incorrect. The correct biochemistry:

  • Calcium competes with lanthanides for XoxF binding sites
  • Calcium suppresses XoxF expression
  • Calcium should be minimized (<10 µM) in lanthanide-dependent growth

This error demonstrated critical gaps in the recommendation validation pipeline.

Components

1. CompetitiveInteractionValidator

Validates metal competition and biochemical interactions.

Location: src/microgrowagents/validators/competitive_interaction_validator.py

Key Features:

  • Calcium-lanthanide competition detection
  • Iron-zinc competition warnings
  • Chelation balance checking
  • Pathway-specific exclusions

Example:

from microgrowagents.validators import CompetitiveInteractionValidator

validator = CompetitiveInteractionValidator()

ingredients = [
    {"ingredient_name": "Calcium chloride", "concentration": 50, "unit": "µM"},
    {"ingredient_name": "Neodymium chloride", "concentration": 5, "unit": "µM"}
]

result = validator.validate_metal_competition(
    ingredients=ingredients,
    experimental_goal="lanthanide depletion",
    organism="SAMN31331780"
)

if not result.passed:
    print(f"Validation failed: {result.message}")
    print(f"Action: {result.action}")
    print(f"Alternative: {result.alternative}")

2. ContradictionDetector

Cross-checks recommendations against excluded lists and documentation.

Location: src/microgrowagents/validators/contradiction_detector.py

Key Features:

  • Excluded ingredient detection
  • Documentation constraint enforcement
  • Concentration limit checking
  • Chemical formula synonym matching (CaCl2 ↔ Calcium chloride)

Example:

from microgrowagents.validators import ContradictionDetector

detector = ContradictionDetector()

recommendations = [
    {"ingredient_name": "Calcium chloride", "concentration": 50, "unit": "µM"}
]

excluded = ["CaCl2", "Citrate"]

contradictions = detector.check_against_excluded_list(
    recommendations=recommendations,
    excluded_ingredients=excluded
)

for c in contradictions:
    print(f"Contradiction: {c.message}")
    print(f"Severity: {c.severity}")
    print(f"Resolution: {c.resolution}")

3. Biochemical Constraints Database

Centralized database of interaction rules and thresholds.

Location: src/microgrowagents/knowledge/biochemical_constraints.yaml

Contents:

  • Competitive interactions (Ca-Ln, Fe-Zn, Mg-Ca)
  • Pathway exclusions (XoxF vs MxaF)
  • Chelation thresholds (citrate-Ln, EDTA-Fe)
  • Metal properties and organism mappings

Example Entry:

competitive_interactions:
  calcium_lanthanide:
    description: "Calcium competes with lanthanides for binding sites"
    mechanism: "Ca²⁺ and Ln³⁺ compete for same coordination sites in proteins"
    rule: |
      IF (lanthanide_present OR xoxF_pathway_targeted)
      THEN calcium_concentration < 10 µM
      REASON: High Ca²⁺ suppresses XoxF expression and competes for Ln³⁺ binding
    threshold:
      max_calcium_with_lanthanides: 10.0  # µM
      severity: "CRITICAL"
    references:
      - "docs/MP_media_optimization_plan.md:353-365"
      - "doi:10.1038/nature16174"
    alternative: "Minimize calcium to <10 µM or remove entirely from base medium"

Usage Workflow

1. Validate During Generation

from microgrowagents.validators import CompetitiveInteractionValidator

validator = CompetitiveInteractionValidator()

# Before finalizing recommendations
result = validator.validate_metal_competition(
    ingredients=proposed_ingredients,
    experimental_goal=experiment_description,
    organism=organism_id
)

if not result.passed and result.severity == "CRITICAL":
    # Reject recommendation
    raise ValueError(f"Validation failed: {result.message}")

2. Validate After Generation

from microgrowagents.validators import ContradictionDetector

detector = ContradictionDetector()

# After generating recommendations
contradictions = detector.check_all(
    recommendations=final_recommendations,
    excluded_ingredients=design_constraints.get("excluded", []),
    experimental_goal=experiment_description
)

if contradictions:
    critical = [c for c in contradictions if c.severity == "CRITICAL"]
    if critical:
        # Flag for review or reject
        print(f"Critical contradictions found: {len(critical)}")
        for c in critical:
            print(f"  - {c.message}")

3. Combined Validation

from microgrowagents.validators import (
    CompetitiveInteractionValidator,
    ContradictionDetector
)

def validate_media_formulation(
    recommendations,
    experimental_goal,
    organism,
    excluded_ingredients=None
):
    """Full validation pipeline."""

    # Layer 1: Competitive interactions
    interaction_validator = CompetitiveInteractionValidator()
    interaction_result = interaction_validator.validate_metal_competition(
        ingredients=recommendations,
        experimental_goal=experimental_goal,
        organism=organism
    )

    # Layer 2: Contradictions
    contradiction_detector = ContradictionDetector()
    contradictions = contradiction_detector.check_all(
        recommendations=recommendations,
        excluded_ingredients=excluded_ingredients,
        experimental_goal=experimental_goal
    )

    # Aggregate results
    validation_report = {
        "passed": interaction_result.passed and len(contradictions) == 0,
        "interaction_result": interaction_result,
        "contradictions": contradictions,
        "critical_issues": [],
        "warnings": []
    }

    # Collect critical issues
    if not interaction_result.passed and interaction_result.severity == "CRITICAL":
        validation_report["critical_issues"].append(interaction_result.message)

    for c in contradictions:
        if c.severity == "CRITICAL":
            validation_report["critical_issues"].append(c.message)

    return validation_report

Validation Result Interpretation

Severity Levels

  • INFO: Informational, no action needed
  • WARNING: Review recommended but not blocking
  • CRITICAL: Blocking error, recommendation must be rejected

Action Types

  • ACCEPT: Recommendation is valid
  • WARN: Recommendation triggers warnings, review recommended
  • REJECT: Recommendation must be rejected

Example Results

Critical Error (Calcium with Lanthanides):

ValidationResult(
    passed=False,
    severity="CRITICAL",
    message="Ca²⁺ concentration (50 µM) exceeds 10 µM limit. High calcium competes with lanthanides for XoxF binding sites and suppresses XoxF expression. See docs/MP_media_optimization_plan.md:353-365",
    violations=[...],
    action="REJECT",
    alternative="Minimize calcium to <10 µM or remove entirely from base medium"
)

Warning (High Fe:Zn Ratio):

ValidationResult(
    passed=True,  # Not blocking
    severity="WARNING",
    message="High Fe:Zn ratio (200:1). Iron and zinc compete for transporters. Consider increasing zinc to maintain ratio < 50:1",
    warnings=["High Fe:Zn ratio"],
    action="WARN",
    alternative=None
)

Testing

Running Tests

# All validation tests
uv run pytest tests/test_validators/ -v

# Integration tests (calcium hallucination prevention)
uv run pytest tests/test_integration/test_prevent_calcium_hallucination.py -v

# All together
uv run pytest tests/test_validators/ tests/test_integration/test_prevent_calcium_hallucination.py -v

Test Coverage

CompetitiveInteractionValidator (9 tests):

  • Calcium-lanthanide conflict detection
  • Trace calcium allowance
  • Pathway exclusion enforcement
  • Iron-zinc competition
  • Multiple violations reporting

ContradictionDetector (7 tests):

  • Excluded ingredient detection
  • Documentation contradiction detection
  • Concentration limit enforcement
  • Chemical synonym matching

Integration Tests (7 tests):

  • v7/v10 scenario regression tests
  • Correct rationale validation
  • Documentation reference validation

Adding New Constraints

1. Add to biochemical_constraints.yaml

competitive_interactions:
  copper_zinc:
    description: "Copper and zinc compete for metalloproteins"
    mechanism: "Cu²⁺ and Zn²⁺ compete for same binding motifs"
    rule: |
      IF (Cu_concentration > 10 µM AND Zn_concentration < 5 µM)
      THEN warn_zinc_displacement
    threshold:
      max_cu_zn_ratio: 2.0
      severity: "WARNING"
    references:
      - "PMID:12345678"
    alternative: "Increase zinc or reduce copper"

2. Add Validation Method

def _check_copper_zinc_competition(self, ingredients: List[Dict]) -> Dict:
    """Check for copper-zinc competition."""
    cu_conc = self._get_metal_concentration(ingredients, ["copper", "Cu"])
    zn_conc = self._get_metal_concentration(ingredients, ["zinc", "Zn"])

    if cu_conc and zn_conc:
        constraint = self.constraints["competitive_interactions"]["copper_zinc"]
        max_ratio = constraint["threshold"]["max_cu_zn_ratio"]

        if cu_conc / zn_conc > max_ratio:
            return {
                "warning": True,
                "type": "competitive_interaction",
                "interaction": "copper_zinc",
                "severity": "WARNING",
                "message": f"High Cu:Zn ratio. {constraint['mechanism']}"
            }

    return {}

3. Add Test

def test_copper_zinc_competition(self, validator):
    """Test that high Cu with low Zn triggers warning."""
    ingredients = [
        {"ingredient_name": "Copper sulfate", "concentration": 20, "unit": "µM"},
        {"ingredient_name": "Zinc sulfate", "concentration": 5, "unit": "µM"}
    ]

    result = validator.validate_metal_competition(
        ingredients=ingredients,
        experimental_goal="general growth",
        organism="any"
    )

    warnings = [w for w in result.warnings if "copper" in w.lower()]
    assert len(warnings) > 0

References

  • v7/v10 Calcium Error: .claude/prompts/GENERATE_*_v7.md, GENERATE_*_v10.md
  • MP Media Plan: docs/MP_media_optimization_plan.md:353-365
  • XoxF Biochemistry: doi:10.1038/nature16174
  • Implementation Plan: Plan file provided to implementation session

Future Enhancements

Planned (P1)

  • DocumentationFactChecker: Validate mechanism claims against documentation
  • PromptConstraintParser: Auto-extract constraints from prompt files
  • Extended constraint database: More interaction types (Cu-Zn, Mn-Fe, etc.)

Possible (P2)

  • Thermodynamic validation: Kd values, binding competition calculations
  • Real-time constraint learning: Update constraints from experimental outcomes
  • LLM-based rationale validation: Check if rationales contradict known biochemistry

Support

For questions or issues, see:

  • Test files in tests/test_validators/
  • Integration tests in tests/test_integration/
  • Biochemical constraints in src/microgrowagents/knowledge/biochemical_constraints.yaml