Skip to content

Commit dbcbda1

Browse files
committed
Rework README: crazyflow layout, badges, animated splat header
1 parent 2679a14 commit dbcbda1

1 file changed

Lines changed: 104 additions & 45 deletions

File tree

README.md

Lines changed: 104 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,64 +1,123 @@
1+
<div align="center">
2+
<picture>
3+
<source media="(prefers-color-scheme: dark)" srcset="assets/logo_emerges_dark.gif">
4+
<img src="assets/logo_emerges_light.gif" alt="splax" width="560">
5+
</picture>
6+
</div>
7+
8+
--------------------------------------------------------------------------------
9+
10+
<div align="center">
11+
112
# splax
213

3-
High-performance gaussian splatting in JAX, implemented in [NVIDIA Warp](https://github.com/NVIDIA/warp)
4-
no system CUDA toolchain needed. Differentiable, vmap-batch-native (single
5-
kernel launch over the whole batch), faster than the CUDA reference it
6-
started from on every benchmarked config.
14+
**Differentiable 3D gaussian splatting for JAX, with rasterization kernels written in [NVIDIA Warp](https://github.com/NVIDIA/warp).**
715

8-
- `splax.inference.render` — grad-free rendering (residuals dropped, tight
9-
intersection, batchable via `jax.vmap` up to 1000+ parallel views).
10-
- `splax.training.render` (= `splax.render`) — differentiable w.r.t. means,
11-
scales, quats, colors, opacities.
12-
- `splax.io.write_ply` / `scripts/train_lego.py --out-ply` — fit splats to
13-
posed images and export standard 3DGS `.ply`.
14-
- `jaxsplat/` — the vendored CUDA reference implementation (parity + grad
15-
baseline), kept intact.
16+
[![Python](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org)
17+
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
18+
[![Ruff](https://github.com/amacati/splax/actions/workflows/ruff.yml/badge.svg)](https://github.com/amacati/splax/actions/workflows/ruff.yml)
19+
[![Docs](https://github.com/amacati/splax/actions/workflows/docs.yml/badge.svg)](https://amacati.github.io/splax)
1620

17-
## Setup
21+
</div>
1822

19-
Requires an NVIDIA GPU.
23+
splax renders and trains 3D gaussian splats inside JAX. Batched rendering and training run through `jax.vmap`, `jax.grad`, and `jax.jit`, with no system CUDA toolchain needed.
2024

21-
```sh
22-
uv sync # jax + warp + deps into .venv
23-
# optional: build the CUDA reference backend (hermetic toolchain, no sudo)
24-
scripts/setup_cuda.sh
25-
CUDACXX=$PWD/.cuda/bin/nvcc CUDAToolkit_ROOT=$PWD/.cuda \
26-
uv pip install --no-build-isolation -e ./jaxsplat
25+
## Examples
26+
27+
Render a `.ply` scene from one camera.
28+
29+
```python
30+
import jax.numpy as jnp
31+
import splax
32+
33+
means, scales, quats, colors, opacities = splax.load_ply("scene.ply")
34+
img = splax.inference.render(
35+
means, scales, quats, colors, opacities,
36+
viewmat=viewmat, background=jnp.ones(3),
37+
img_shape=(H, W), f=(fx, fy), c=(W // 2, H // 2),
38+
glob_scale=1.0, clip_thresh=0.01,
39+
) # (H, W, 3)
40+
```
41+
42+
Batch over a stack of camera poses with `jax.vmap`. One batched kernel launch, not a Python loop.
43+
44+
```python
45+
import jax
46+
47+
frames = jax.vmap(lambda vm: splax.inference.render(
48+
means, scales, quats, colors, opacities,
49+
viewmat=vm, background=jnp.ones(3), img_shape=(H, W),
50+
f=(fx, fy), c=(W // 2, H // 2), glob_scale=1.0, clip_thresh=0.01,
51+
))(viewmats) # (B, H, W, 3)
52+
```
53+
54+
Take gradients through the differentiable renderer with `jax.grad`. `splax.render` is `splax.training.render` and differentiates w.r.t. means, scales, quats, colors, opacities.
55+
56+
```python
57+
import jax
58+
59+
def loss(means, scales, quats, colors, opacities):
60+
img = splax.render(
61+
means, scales, quats, colors, opacities,
62+
viewmat=viewmat, background=jnp.ones(3), img_shape=(H, W),
63+
f=(fx, fy), c=(W // 2, H // 2), glob_scale=1.0, clip_thresh=0.01,
64+
)
65+
return jnp.mean((img - target) ** 2)
66+
67+
grads = jax.grad(loss, argnums=(0, 1, 2, 3, 4))(means, scales, quats, colors, opacities)
2768
```
2869

70+
## Documentation
71+
72+
Full documentation lives at [amacati.github.io/splax](https://amacati.github.io/splax): installation, a quickstart, a user guide for rendering, training, batching, and IO, and the API reference.
73+
74+
## Why
75+
76+
Gaussian splatting lives mostly in PyTorch and hand-written CUDA. splax puts it inside JAX so splat rendering composes with `jax.vmap`, `jax.grad`, and `jax.jit` and drops into research pipelines that already run on JAX, without leaving the ecosystem for the render step.
77+
78+
## Architecture
79+
80+
The renderer is not pure JAX because the core of splatting does not map to XLA primitives. Rasterization is tile-binned with a data-dependent sort of gaussian-tile intersections, per-pixel early termination once transmittance saturates, and a memory-frugal backward that recomputes the blend instead of storing per-pixel state. splax implements the projection, rasterization, and their backward passes as Warp kernels and wires them into JAX through FFI custom calls under `jax.custom_vjp`. The kernels are batch-native: `jax.vmap` maps to a single batched launch (camera id folded into the sort key) rather than a sequential per-sample loop.
81+
82+
## Relation to jaxsplat
83+
84+
splax started from [jaxsplat](https://github.com/yklcs/jaxsplat) as the reference and parity baseline. The rasterizer was rewritten from CUDA to Warp to drop the system toolchain, then extended with the feature and performance work below.
85+
2986
## Improvements
3087

31-
Ported from gsplat and the papers behind it (credit per item; details in `reports/`):
88+
Ported from [gsplat](https://github.com/nerfstudio-project/gsplat) and the papers behind it, which inspired most of the performance work (credit per item):
89+
90+
- Native multi-camera batched rendering, one launch with the camera id folded into the sort key (gsplat)
91+
- Opacity-aware tight tile intersection (StopThePop, Speedy-Splat, gsplat #927)
92+
- Packed 32-bit sort keys with quantized depth
93+
- Persistent sort and bin scratch across frames (gsplat caching allocator design)
94+
- Cooperative shared-memory tile blending with block-vote early exit (3DGS, gsplat)
95+
- Fixed-budget MCMC training with static shapes, relocation plus covariance noise (gsplat MCMCStrategy, Kheradmand et al. 2024)
96+
- Opacity and scale regularizers (gsplat mcmc preset)
97+
- Progressive resolution fine-tuning (coarse-to-fine, 3DGS)
98+
- Per-parameter Adam learning-rate schedules (gsplat, 3DGS)
99+
- L1 plus D-SSIM photometric loss (3DGS, gsplat, via dm-pix)
100+
- Camera pose gradients via `render(..., diff_wrt=("viewmat",))` (gsplat projection backward)
101+
- Batch-native backward passes, `jax.vmap(jax.grad(render))` runs as one batched launch (gsplat)
102+
- Batched training steps with sqrt-batch learning-rate scaling (gsplat `batch_size` and `steps_scaler`)
103+
- Anti-aliased opacity compensation (Mip-Splatting, gsplat), depth regularization from COLMAP points (gsplat `depth_loss`), per-image exposure correction (gsplat appearance optimization), all opt-in
32104

33-
- Native multi-camera batched rendering, single launch with camera id in the sort key — up to 5× per-frame at batch 8, ~60k renders/s at 128² batch 1024 (gsplat)
34-
- Opacity-aware tight tile intersection, −40–81% fewer intersections and −15–76% frame time on real trained scenes (lego 313k, drone 150k) (StopThePop §B.2; Speedy-Splat SNUGBOX/AccuTile, gsplat #927)
35-
- Persistent signature-keyed sort/bin scratch, kills per-frame realloc (gsplat caching-allocator design)
36-
- Cooperative shared-memory tile blending with block-vote early exit (3DGS; gsplat)
37-
- Verified negatives, kept documented: 64-thread/4-px CTA blend (1.5–3.7× slower) and warp-level backward reduction (1.9× slower on dense drone; wins random 1.05–1.80×) — both occupancy/barrier-limited under warp-lang, re-confirmed on clean lego/drone scenes (gsplat designs; reports/phase5_o4.md, phase6b.md, phase6g_lego_reverification.md)
38-
- Fixed-budget MCMC training in JAX (static shapes): dead-gaussian relocation + per-step covariance noise, +1.9 dB on the lego fit (gsplat MCMCStrategy; Kheradmand et al. 2024)
39-
- Opacity/scale regularizers from the MCMC paper — opacity_reg is load-bearing for a fixed budget with no pruning, +8.9 dB on the lego fit (gsplat mcmc preset)
40-
- Progressive 400²→800² resolution fine-tune, +5–6 dB by resolving fine detail at the eval resolution (coarse-to-fine, 3DGS)
41-
- Per-parameter Adam LR schedules, means exponentially decayed ~100× over training (gsplat/3DGS create_splats_with_optimizers)
42-
- L1 + 0.2·D-SSIM photometric loss mix, +0.7 dB on the lego fit (3DGS; gsplat, via dm-pix)
43-
- Camera-pose (viewmat) gradients via `render(..., diff_wrt=("viewmat",))` — dedicated projection-backward kernels sharing all vjp math; recovers a 5°/0.2 pose perturbation to <1e-3°/5e-7 on lego. The 12-float v_viewmat reduction uses a block `tile_sum`, 22–113× faster than plain atomics at 50k–742k gaussians (gsplat ProjectionEWA3DGSFused bwd, reports/phase6e.md)
44-
- Batch-native backward passes — `jax.vmap(jax.grad(render))` runs one batched backward launch (no per-sample loop) for every `diff_wrt`, up to 8.4× faster than the loop and ~2.1× for simultaneous multi-pose recovery (reports/phase8a_batched_grads.md)
45-
- Batched training steps + √batch LR scaling — B views/step amortize per-step fixed cost, −12% wall at PSNR parity on the mid-N (150k) drone fit; scale-dependent, so B=1 stays default (gsplat `batch_size`/`steps_scaler`, reports/phase8l_batched_training.md)
105+
## Installation
46106

47-
## Benchmarks & reports
107+
Requires an NVIDIA GPU and a CUDA-enabled JAX (`jax[cuda12]`, pulled in as a dependency).
48108

49109
```sh
50-
.venv/bin/python scripts/benchmark.py <label> --backend {warp,cuda} [--mem] # real scene: lego.ply by default
51-
.venv/bin/python scripts/benchmark_train.py <label> # training steps
52-
.venv/bin/python scripts/plot_results.py <label> [<label2>]
53-
.venv/bin/python scripts/render_lego.py # correctness gate vs ground truth
110+
uv pip install "git+https://github.com/amacati/splax"
54111
```
55112

56-
Per-phase reports with figures live in `reports/` (phase1–4: setup, JAX FFI
57-
migration, vmap, Warp port; phase5_*: one report per ported gsplat
58-
optimization, including verified negatives; phase6*: training support).
113+
Developer setup with [pixi](https://pixi.sh/), which installs splax editable with the dev tooling:
114+
115+
```sh
116+
git clone https://github.com/amacati/splax.git
117+
cd splax
118+
pixi shell
119+
```
59120

60121
## License
61122

62-
MIT (see [LICENSE](LICENSE)).
63-
gsplat-derived portions under Apache-2.0 ([licenses/Apache-2.0.txt](licenses/Apache-2.0.txt)).
64-
Started from [jaxsplat](https://github.com/yklcs/jaxsplat), rewritten in Warp to drop the CUDA toolchain, with optimizations ported from [gsplat](https://github.com/nerfstudio-project/gsplat) and the papers behind it (3DGS, StopThePop, Speedy-Splat).
123+
MIT (see [LICENSE](LICENSE)). gsplat-derived portions are under Apache-2.0 ([licenses/Apache-2.0.txt](licenses/Apache-2.0.txt)).

0 commit comments

Comments
 (0)