Thesis: Causal Graph Discovery with LLMs + Classical Methods

This package discovers and evaluates causal graphs by combining:

• Statistical causal discovery (e.g., PC, GES, DAGMA-Linear)
• LLM-based generation (prompted with dataset context + optional stat baselines)
• LLM judging (scoring, pairwise tournaments)
• Unified metrics, encodings, reproducible logging & results export

It supports three experiment modes:

• Exp1 – Single LLM pipeline (with statistical hints) → one graph + metrics
• Exp2 - Ensemble + LLM scorers + tournament of LLM judges (dual comparison)
  • Scoring – Ensemble of LLM graph generators + LLM scorers → best graph + metrics
  • Pairwise – Ensemble + tournament of LLM judges (dual comparison) → final graph + metrics

---

Contents

• Quick Start
• Configuration
• How It Works
  • Data Loading
  • Graph Encoders
  • LLMs and Chains
  • Statistical Algorithms
  • Graph Generation (Runners)
  • Scoring & Tournament
  • Metrics
  • Results & Logging
• CLI Usage
• Extending the Package
• Environment Variables

---

Quick Start

1. Install dependencies (Python 3.12 recommended), required packages are listed in setup.py:

pip install -e .

2. Set API keys (see Environment Variables).

3. Prepare a config (YAML). Example minimal Exp1:

dataset_dir: "./data"
dataset: "covid_infection"   # or covid_complete, injection_molding, valeo
results_path: "./runs/exp1"

statistical_algorithms:
  algorithms: ["pc", "ges"]   # optional empty list

graph_llm:
  name: "o3-2025-04-16"         # or local model, deepseek, gemini, etc.
  temperature: 0.2
  chain:
    name: "simple_chain"      # or "smart_chain"
    n_ideas: 5                # only used by smart_chain
  runner: "bfs_single_source" # see Runners below
  graph_encoding: "multi-node" # see Encoders below
  use_corr: true

4. Run:

python -m src.exps.exp1.run --config-path path/to/config.yaml

The script saves results to <results_path>/results.json.

---

Configuration

All configs validate via Pydantic models in src/config.py.

• Base fields

  • dataset_dir (Path), dataset (str), results_path (Path)
  • statistical_algorithms.algorithms (list[str])

• LLM configs

  • LLM.name, LLM.temperature, LLM.chain.{name,n_ideas}
  • GraphLLM.runner, GraphLLM.use_corr, graph_encoding
  • JudgeLLM.graph_encoding

• Experiment wrappers

  • Exp1Config – single graph_llm
  • Exp2Config – graph_llms (list), judge_llms (list)

Example Exp2 (ensemble + (scoring, pairwise, or both))

dataset_dir: "./data"
dataset: "valeo"
results_path: "./runs/exp2"

statistical_algorithms:
  algorithms: ["ges"]

graph_llms:
  - name: "o3-2025-04-16"
    temperature: 0.1
    chain: { name: "simple_chain" }
    runner: "bfs_single_source"
    graph_encoding: "multi-node"
    use_corr: true

  - name: "gemini-2.5-flash"
    temperature: 0.2
    chain: { name: "smart_chain", n_ideas: 5 }
    runner: "bfs_chat_source"
    graph_encoding: "multi-node"
    use_corr: false

judge_llms:
  - name: "gemini-2.5-pro"
    temperature: 0.0
    chain: { name: "simple_chain" }
    graph_encoding: "multi-node"

---

How It Works

Data Loading

src/data/data_manager.py delegates to DataLoader variants:

• COVID (covid_infection, covid_complete) from .xdsl Bayesian network file
• Injection Molding from CSV (filtered to selected material, normalized)
• Valeo DAG from JSON + Parquet observations (normalized & sampled)

Returns:

• Ground-truth adjacency (pd.DataFrame)
• Observational data (pd.DataFrame)
• Numpy arrays for model input + small Gaussian noise (stability)

Graph Encoders

Encoders convert between internal graph dicts and strings for LLM prompts:

• multi-node

  A causes B, C
  B has no outgoing edges
  

• single-node

  A causes B
  A causes C
  B has no outgoing edges
  

• digraph (Graphviz edge lines)

  A -> B
  A -> C
  

• graphml – XML graph format

• json – { node: {parents: [], children: []} } (prompt-safe)

LLMs and Chains

• Models: resolved in src/algs/llm/llms.get_llm
  Supports OpenAI, Google Gemini, DeepSeek, and local VLLM (HF).

• Supported models include:

  • OpenAI

    • gpt-4o-mini-2024-07-18
    • gpt-4o-2024-08-06
    • o4-mini-2025-04-16
    • gpt-4.1-2025-04-14
    • gpt-4.1-mini-2025-04-14
    • o3-mini-2025-01-31
    • o3-2025-04-16

  • Google Gemini

    • gemini-2.0-flash
    • gemini-2.5-flash
    • gemini-2.5-pro

  • DeepSeek

    • deepseek-reasoner
    • deepseek-chat

  • HuggingFace / Local VLLM

    • mosaicml/mpt-7b
    • tiiuae/falcon-40b
    • meta-llama/Meta-Llama-3-70B-Instruct
    • meta-llama/Meta-Llama-3.1-405B-Instruct
    • meta-llama/Meta-Llama-3.1-70B
    • neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a8
    • neuralmagic/Qwen2-72B-Instruct-quantized.w8a16
    • neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w8a16
    • neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16
    • neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w4a16
    • hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4

• Chains:

  • SimpleChain – direct prompt
  • SmartChain – ideas → critique → resolution

• Prompts: context + dataset variable definitions + optional stat results

Statistical Algorithms

In src/algs/statistical/:

• PC, GES, DAGMA-Linear
• Unified API: StatisticalAlgorithm.run(...) -> adjacency DataFrame
• Orchestrated via run_statistical_algorithms_stage(...)
• Outputs used only as hints for LLMs (not ground truth)

Graph Generation (Runners)

• BFSSingleSourceRunner – independent requests per node
• BFSChatSourceRunner – ongoing chat across BFS queue
• Enforces DAG (rejects cycles, retries)
• Optional correlation context (Pearson r for current node vs others)

Scoring & Tournament

• Scorers (GraphScorer):

  • Vanilla: score ∈ [0,1]
  • G-Eval: rubric-based reasoning (DeepEval wrapper)

• Exp2 supports multiple scorers/judges

  • Scoring: Judge LLMs score → best avg
  • Pairwise: Dual comparison tournament, A vs B (twice, to detect bias). Tie → random tiebreak.

Metrics

From src/metrics.py:

• F1, Precision, Recall
• Accuracy (Jaccard of edge sets)
• Normalized Hamming Distance (NHD)
• Baseline Hamming Distance (BHD)
• NHD/BHD Ratio
• DAG check
• Edge counts

Results & Logging

• Results written to <results_path>/results.json
• Includes config snapshot, stat graphs + metrics, generated graphs + metrics, best/final graph, timing
• Logging via src/logger.py: console/file/socket handlers

---

CLI Usage

Each experiment is runnable as a module:

python -m src.exps.exp1.run --config-path /path/to/config.yaml
python -m src.exps.exp2.run --config-path /path/to/config.yaml
python -m src.exps.exp2.run_scoring --config-path /path/to/config.yaml
python -m src.exps.exp2.run_pairwise_comparison --config-path /path/to/config.yaml

---

Extending the Package

• Statistical algorithm – implement StatisticalAlgorithm.run, register in _STATISTICAL_ALGORITHMS
• Graph encoder – subclass GraphEncoder, register in _GRAPH_ENCODERS
• LLM model – extend get_llm
• Runner – subclass Runner, register in _GRAPH_RUNNERS
• Scorer – subclass GraphScorer, register in _GRAPH_SCORERS

---

Environment Variables

• OpenAI: OPENAI_API_KEY

• Google Gemini: GOOGLE_API_KEY

• Portkey gateway (optional):

  • PORTKEY_API_KEY, PORTKEY_OPENAI_VIRTUAL_KEY, PORTKEY_GOOGLE_VIRTUAL_KEY

• Local VLLM:

  • VLLM_TENSOR_PARALLEL_SIZE, VLLM_DOWNLOAD_DIR, HF_TOKEN

• Logging: LOGS_PATH

• Rate limits: GEMINI_RATE_LIMIT_SLEEP

---

Notes & Conventions

• Graph dict: dict[str, list[str]] → parent → children
• Encoders provide stable, LLM-friendly representations
• Cycle prevention enforced in BFS runners
• Prompts require <Answer>...</Answer> tagging