Vernata: Self-Supervised Learning of LiDAR Point Representations

Accepted at IROS 2026

Oliver Lemke^1,2, Alexander Liniger¹, Abel Gawel¹, Marco Hutter^1,2

¹Robotics and AI Institute     ²ETH Zurich

We present Vernata, a multi-modal, multi-teacher distillation framework for self-supervised learning on outdoor LiDAR point clouds. Building upon the Sonata architecture, we introduce sparse view augmentation, a memory bank mechanism, and cross-modal distillation using dense, high-resolution 2D image features to enable fine-grained semantic guidance and robust inference across varying point densities.

Overview of the Framework. We present a multi-modal multi-teacher distillation architecture for self-supervised point cloud learning. Building upon the Sonata framework, highlighted in gray, our model is anchored by two teachers processing global views of the scene: A 3D teacher (top), employing a PTv3 encoder and updated via EMA, and a frozen 2D teacher (bottom), producing DINOv2-S features, which are upsampled with LoftUp and backprojected into the point cloud. In contrast, the student receives three variations of the global view: masked views, local crops, and sparse subsampled views. The student embeddings are aligned with the 3D teacher via cross-entropy loss and with the 2D teacher via cosine similarity.

Quick Start ⚡

Inference Only

For lightweight inference without the full training pipeline, see our standalone inference repository: vernata-inference (coming soon). It provides minimal dependencies and pre-trained model loading out of the box.

Full Training & Evaluation 🚀

1. Environment Setup

We use two separate conda environments. Follow the linked READMEs for step-by-step instructions:

• Main environment (Python 3.12, CUDA 12.4) — covers training, evaluation, and all model code. See install/vernata/README.md.
• Waymo parsing environment (Python 3.10) — only needed if you work with the Waymo Open Dataset, which requires older TensorFlow dependencies. See install/waymo/README.md.

2. Dataset Preparation

Download and preprocess the datasets you want to train on. Each dataset has its own processing pipeline and README under source/scripts/datasets/. See the dataset processing README for an overview.

3. Configure Scripts

Before launching any training or evaluation run, open the script you want to use and update the paths marked with !! MUST change before running at the top of each file. The key fields are:

Field	Where to set it	Description
paths.resources	All scripts	Root directory containing your datasets.
dataset.*.data_root	All scripts	Path to the WebDataset shards for each dataset (relative to paths.resources or absolute).
trainer.devices	All scripts	Number of GPUs on your machine.
trainer.strategy	All scripts	ddp for multi-GPU, auto for single GPU.
model.inference.local.model_path	Downstream & validation scripts	Path to the downloaded checkpoint (see Pre-trained Models). Passed automatically via the $1 positional argument.
model.inference.local.config_path	Downstream & validation scripts	Path to the downloaded config YAML. Passed automatically via the $1 positional argument.

4. Training & Evaluation

The scripts/ directory contains ready-to-use shell scripts for reproducing our results. The pipeline is split into three sequential stages:

Stage	Directory	Description
Self-supervised pre-training	scripts/ssl/	Trains the Vernata multi-teacher distillation model on unlabelled LiDAR data. This is the main pre-training step.
Downstream evaluation	scripts/downstream/	Freezes the pre-trained backbone and trains a linear segmentation head on labelled data to evaluate representation quality.
Validation	scripts/validate/	Runs inference on a trained checkpoint and reports metrics (e.g., mIoU).

Each script documents the required and optional configuration overrides at the top of the file. See the scripts/README.md for more details.

Pre-trained Models 🏋️

We release a pre-trained checkpoint that was self-supervised on Grand Tour + TartanGround and then downstream fine-tuned with linear probing on TartanGround.

Model	Pre-training data	Downstream data	Backbone	TartanGround mIoU	Download
Vernata (base)	Grand Tour + TartanGround	TartanGround (small split)	PTv3-base	0.5471	checkpoint + config

Download

# Download the checkpoint and config into a local directory
mkdir -p checkpoints/vernata-base
wget -O checkpoints/vernata-base/checkpoint.ckpt  "https://huggingface.co/theaiinstitute/vernata_ttg_SpSkCmd/resolve/main/ssl/checkpoint.ckpt"
wget -O checkpoints/vernata-base/unresolved-config.yaml "https://huggingface.co/theaiinstitute/vernata_ttg_SpSkCmd/resolve/main/ssl/unresolved-config.yaml"

Using the checkpoint

The checkpoint contains both the pre-trained backbone and the TartanGround segmentation head. You can use it in two ways:

Validation — evaluate the released checkpoint on TartanGround (uses the included segmentation head):

# Pass the checkpoint folder as the first argument:
bash scripts/validate/validate_ttg.sh /path/to/checkpoints/vernata

Downstream fine-tuning on a new dataset — load only the backbone and train a fresh segmentation head (e.g., on your own dataset):

# Pass the checkpoint folder as the first argument:
bash scripts/downstream/downstream_way.sh /path/to/checkpoints/vernata

Note: The downstream and validation scripts expect the path to the local checkpoint folder as the first positional argument. The folder should contain checkpoint.ckpt and unresolved-config.yaml.

TartanGround Evaluation Protocol 📊

As part of this work, we introduce a standardized evaluation protocol for TartanGround, including a custom 7-class semantic taxonomy, curated train/val splits, and scene-specific label corrections. This enables consistent, reproducible benchmarking across methods. See TARTANGROUND_EVAL.md for the full specification.

Code Structure 🎬

vernata/
├── install/                        # Environment setup (conda + pip)
│   ├── vernata/                #   Main environment (Python 3.12, CUDA 12.4)
│   └── waymo/                      #   Separate Waymo TFRecord parsing env (Python 3.10)
├── resources/
│   └── configs/                    # Hydra / OmegaConf configuration pipeline
│       ├── config.yaml             #   Root config (merged at runtime)
│       ├── dataset_configs/        #   Per-dataset configs (Waymo, Grand Tour, TartanGround, …)
│       ├── model_configs/          #   Backbone & head architecture definitions
│       └── stage_configs/          #   Training stage overrides (SSL, downstream, validation, …)
├── scripts/                        # Shell scripts for launching training & evaluation
│   ├── ssl/                        #   Self-supervised pre-training jobs
│   ├── downstream/                 #   Linear-probe downstream evaluation jobs
│   └── validate/                   #   Standalone validation jobs
└── source/                         # Python source code
    ├── main.py                     # Entry point — initialises Hydra config & runs the Orchestrator
    ├── orchestrator/               # Stage-based training orchestrator
    ├── models/                     # Model definitions
    │   ├── model/                  #   Vernata Lightning modules
    │   ├── blocks/                 #   Shared components (clustering, Sinkhorn-Knopp, …)
    │   └── point_transformer/      #   PointTransformerV3 backbone (serialization, modules)
    ├── dataset_manager/            # Dataset & DataModule implementations
    │   ├── data_module.py          #   PyTorch Lightning DataModule
    │   └── *_dataset.py            #   Per-dataset classes (Waymo, Grand Tour, TartanGround)
    ├── callbacks/                  # Lightning callbacks (logging, profiling, …)
    ├── optimizer/                  # Optimizer & LR scheduler construction
    ├── vernata_utils/              # Shared utilities
    │   ├── core/                   #   Config manager, registry, auto-instantiation, logging
    │   ├── data/                   #   Point cloud data structures, transforms, rigid-body math
    │   ├── training/               #   Dataloader helpers, W&B logging, rank utilities
    │   ├── metrics/                #   Loss functions & evaluation metrics
    │   ├── vis/                    #   Point cloud & similarity visualization
    │   └── inference/              #   Inference utilities
    ├── scripts/                    # Standalone Python scripts
    │   ├── datasets/               #   Dataset download & processing pipelines (per-dataset READMEs)
    │   └── vis/                    #   WebDataset inspection & visualization
    └── third_party/               # Vendored / submoduled external code
        ├── loftup/                 #   LoftUp — 2D feature upsampling & backprojection
        ├── pointcept/              #   Pointcept — point cloud learning toolkit
        └── tartanairpy/            #   TartanAir download API

Key READMEs

Topic	Location
Installation (main environment)	install/vernata/README.md
Installation (Waymo parsing)	install/waymo/README.md
Configuration pipeline	resources/configs/README.md
Training & evaluation scripts	scripts/README.md
Dataset processing pipelines	source/scripts/datasets/README.md

How it works

1. source/main.py initialises the Hydra configuration and launches the Orchestrator.
2. The Orchestrator (source/orchestrator/) handles the stages defined in the config (e.g. self-supervised pre-training → downstream evaluation).
3. Models (source/models/) implement the Vernata multi-teacher distillation architecture on top of a PointTransformerV3 backbone, wrapped as PyTorch Lightning modules.
4. Datasets (source/dataset_manager/) provide WebDataset-backed data loading for Waymo, Grand Tour, and TartanGround, with configurable transform pipelines.
5. Configuration (resources/configs/) is fully declarative — dataset, model, and stage configs are composed by Hydra at runtime. See the configuration README for a detailed guide.

News 📰

• [2026-08] We released the code, our pre-trained models, and the standardized TartanGround evaluation protocol. Check out our paper and website.

BibTeX 🙏

If you find this work useful, please consider citing:

@article{lemke2026vernata,
    author    = {Lemke, Oliver and Liniger, Alexander and Gawel, Abel and Hutter, Marco},
    title     = {Vernata: Self-Supervised Learning of LiDAR Point Representations},
    journal   = {arXiv preprint arXiv:2608.06919},
    year      = {2026}
}

Acknowledgements

Thank you to the creators and maintainers of the following projects that helped make this work possible:

• LoftUp for the feature upsampler.
• Pointcept / Sonata for their point cloud research code base.
• TartanGround / tartanairpy for the dataset and processing pipeline and support regarding dataset access.
• GrandTour for the dataset and support regarding dataset access.
• Waymo for the dataset.