InSAR-VLM-Bench: Vision-Language Models on Interferogram Understanding

Can a multimodal LLM read a radar interferogram? Spoiler: sort of. This benchmark measures how far.

InSAR-VLM-Bench is an evaluation suite for vision-language models applied to Interferometric SAR imagery. It covers four tasks that probe distinct skills: coherence judgement, deformation localization, fringe counting, and noise attribution.

Table of Contents

• Why
• Tasks
• Installation
• Quick start
• Building the dataset
• Leaderboard
• Extending the benchmark
• Citation

Why

General-purpose VLMs are trained mostly on natural photographs. InSAR products live in a completely different visual domain — wrapped phase, decorrelation noise, fringe patterns — and a model that aces COCO captioning may have no idea what a tropospheric delay looks like. This benchmark gives a small, principled way to measure that gap and to track progress as remote-sensing-specific VLMs improve.

Tasks

Task	Type	Metric	Examples
coherence_classification	3-way classif.	accuracy	540
deformation_localization	grounding	IoU + token-F1	480
fringe_counting	regression	mean abs err	620
noise_assessment	open QA	token-F1	540

See configs/tasks.yaml for prompts and label spaces.

Installation

git clone https://github.com/EchoFish/insar-vlm-bench.git
cd insar-vlm-bench
pip install -e .

If you only want to evaluate a model and skip dataset-building dependencies:

pip install -e ".[eval]"   # skips rasterio etc.

Quick start

Evaluate LLaVA on all tasks of the bundled test split:

ivb-eval --model llava --data /path/to/insar-vlm-bench-v0.4 --out results/

Or from Python:

from insar_vlm_bench import IfgDataset, run_eval
from insar_vlm_bench.models import LlavaAdapter

ds = IfgDataset("/path/to/insar-vlm-bench-v0.4", task="fringe_counting")
model = LlavaAdapter()
print(run_eval(model, "/path/to/insar-vlm-bench-v0.4", "fringe_counting"))

Building the dataset

The benchmark is derived from openly available Sentinel-1 products and a curated selection of pairs spanning tectonic, volcanic, anthropogenic and slow-creep regions. To re-build from raw .tif interferograms and coherence maps:

python scripts/build_dataset.py --raw data/raw --out data/processed

The labels for coherence_classification and fringe_counting are derived directly from the underlying numerical maps, so they are reproducible. Labels for deformation_localization and noise_assessment are human-annotated; the annotation guidelines live in docs/annotation.md.

Leaderboard

See LEADERBOARD.md — updated as new models are added.

Extending the benchmark

Adding a new model adapter is one file. Subclass VLMAdapter and implement generate:

from insar_vlm_bench.models.base import VLMAdapter

class MyAdapter(VLMAdapter):
    name = "my-model"
    def generate(self, image, prompt, max_new_tokens=128):
        ...

Then register it in insar_vlm_bench/cli.py::MODEL_REGISTRY and you're done.

Citation

If you use this benchmark, please cite:

@misc{yu2024insarvlmbench,
  author = {Yu, Hui},
  title  = {{InSAR-VLM-Bench}: Benchmarking Vision-Language Models on Interferogram Understanding},
  year   = {2024},
  url    = {https://github.com/EchoFish/insar-vlm-bench}
}

License

MIT. See LICENSE.