|
| 1 | +<div align="center"> |
| 2 | + <picture> |
| 3 | + <source media="(prefers-color-scheme: dark)" srcset="assets/logo_emerges_dark.gif"> |
| 4 | + <img src="assets/logo_emerges_light.gif" alt="splax" width="560"> |
| 5 | + </picture> |
| 6 | +</div> |
| 7 | + |
| 8 | +-------------------------------------------------------------------------------- |
| 9 | + |
| 10 | +<div align="center"> |
| 11 | + |
1 | 12 | # splax |
2 | 13 |
|
3 | | -High-performance gaussian splatting in JAX, implemented in [NVIDIA Warp](https://github.com/NVIDIA/warp) — |
4 | | -no system CUDA toolchain needed. Differentiable, vmap-batch-native (single |
5 | | -kernel launch over the whole batch), faster than the CUDA reference it |
6 | | -started from on every benchmarked config. |
| 14 | +**Differentiable 3D gaussian splatting for JAX, with rasterization kernels written in [NVIDIA Warp](https://github.com/NVIDIA/warp).** |
7 | 15 |
|
8 | | -- `splax.inference.render` — grad-free rendering (residuals dropped, tight |
9 | | - intersection, batchable via `jax.vmap` up to 1000+ parallel views). |
10 | | -- `splax.training.render` (= `splax.render`) — differentiable w.r.t. means, |
11 | | - scales, quats, colors, opacities. |
12 | | -- `splax.io.write_ply` / `scripts/train_lego.py --out-ply` — fit splats to |
13 | | - posed images and export standard 3DGS `.ply`. |
14 | | -- `jaxsplat/` — the vendored CUDA reference implementation (parity + grad |
15 | | - baseline), kept intact. |
| 16 | +[](https://www.python.org) |
| 17 | +[](LICENSE) |
| 18 | +[](https://github.com/amacati/splax/actions/workflows/ruff.yml) |
| 19 | +[](https://amacati.github.io/splax) |
16 | 20 |
|
17 | | -## Setup |
| 21 | +</div> |
18 | 22 |
|
19 | | -Requires an NVIDIA GPU. |
| 23 | +splax renders and trains 3D gaussian splats inside JAX. Batched rendering and training run through `jax.vmap`, `jax.grad`, and `jax.jit`, with no system CUDA toolchain needed. |
20 | 24 |
|
21 | | -```sh |
22 | | -uv sync # jax + warp + deps into .venv |
23 | | -# optional: build the CUDA reference backend (hermetic toolchain, no sudo) |
24 | | -scripts/setup_cuda.sh |
25 | | -CUDACXX=$PWD/.cuda/bin/nvcc CUDAToolkit_ROOT=$PWD/.cuda \ |
26 | | - uv pip install --no-build-isolation -e ./jaxsplat |
| 25 | +## Examples |
| 26 | + |
| 27 | +Render a `.ply` scene from one camera. |
| 28 | + |
| 29 | +```python |
| 30 | +import jax.numpy as jnp |
| 31 | +import splax |
| 32 | + |
| 33 | +means, scales, quats, colors, opacities = splax.load_ply("scene.ply") |
| 34 | +img = splax.inference.render( |
| 35 | + means, scales, quats, colors, opacities, |
| 36 | + viewmat=viewmat, background=jnp.ones(3), |
| 37 | + img_shape=(H, W), f=(fx, fy), c=(W // 2, H // 2), |
| 38 | + glob_scale=1.0, clip_thresh=0.01, |
| 39 | +) # (H, W, 3) |
| 40 | +``` |
| 41 | + |
| 42 | +Batch over a stack of camera poses with `jax.vmap`. One batched kernel launch, not a Python loop. |
| 43 | + |
| 44 | +```python |
| 45 | +import jax |
| 46 | + |
| 47 | +frames = jax.vmap(lambda vm: splax.inference.render( |
| 48 | + means, scales, quats, colors, opacities, |
| 49 | + viewmat=vm, background=jnp.ones(3), img_shape=(H, W), |
| 50 | + f=(fx, fy), c=(W // 2, H // 2), glob_scale=1.0, clip_thresh=0.01, |
| 51 | +))(viewmats) # (B, H, W, 3) |
| 52 | +``` |
| 53 | + |
| 54 | +Take gradients through the differentiable renderer with `jax.grad`. `splax.render` is `splax.training.render` and differentiates w.r.t. means, scales, quats, colors, opacities. |
| 55 | + |
| 56 | +```python |
| 57 | +import jax |
| 58 | + |
| 59 | +def loss(means, scales, quats, colors, opacities): |
| 60 | + img = splax.render( |
| 61 | + means, scales, quats, colors, opacities, |
| 62 | + viewmat=viewmat, background=jnp.ones(3), img_shape=(H, W), |
| 63 | + f=(fx, fy), c=(W // 2, H // 2), glob_scale=1.0, clip_thresh=0.01, |
| 64 | + ) |
| 65 | + return jnp.mean((img - target) ** 2) |
| 66 | + |
| 67 | +grads = jax.grad(loss, argnums=(0, 1, 2, 3, 4))(means, scales, quats, colors, opacities) |
27 | 68 | ``` |
28 | 69 |
|
| 70 | +## Documentation |
| 71 | + |
| 72 | +Full documentation lives at [amacati.github.io/splax](https://amacati.github.io/splax): installation, a quickstart, a user guide for rendering, training, batching, and IO, and the API reference. |
| 73 | + |
| 74 | +## Why |
| 75 | + |
| 76 | +Gaussian splatting lives mostly in PyTorch and hand-written CUDA. splax puts it inside JAX so splat rendering composes with `jax.vmap`, `jax.grad`, and `jax.jit` and drops into research pipelines that already run on JAX, without leaving the ecosystem for the render step. |
| 77 | + |
| 78 | +## Architecture |
| 79 | + |
| 80 | +The renderer is not pure JAX because the core of splatting does not map to XLA primitives. Rasterization is tile-binned with a data-dependent sort of gaussian-tile intersections, per-pixel early termination once transmittance saturates, and a memory-frugal backward that recomputes the blend instead of storing per-pixel state. splax implements the projection, rasterization, and their backward passes as Warp kernels and wires them into JAX through FFI custom calls under `jax.custom_vjp`. The kernels are batch-native: `jax.vmap` maps to a single batched launch (camera id folded into the sort key) rather than a sequential per-sample loop. |
| 81 | + |
| 82 | +## Relation to jaxsplat |
| 83 | + |
| 84 | +splax started from [jaxsplat](https://github.com/yklcs/jaxsplat) as the reference and parity baseline. The rasterizer was rewritten from CUDA to Warp to drop the system toolchain, then extended with the feature and performance work below. |
| 85 | + |
29 | 86 | ## Improvements |
30 | 87 |
|
31 | | -Ported from gsplat and the papers behind it (credit per item; details in `reports/`): |
| 88 | +Ported from [gsplat](https://github.com/nerfstudio-project/gsplat) and the papers behind it, which inspired most of the performance work (credit per item): |
| 89 | + |
| 90 | +- Native multi-camera batched rendering, one launch with the camera id folded into the sort key (gsplat) |
| 91 | +- Opacity-aware tight tile intersection (StopThePop, Speedy-Splat, gsplat #927) |
| 92 | +- Packed 32-bit sort keys with quantized depth |
| 93 | +- Persistent sort and bin scratch across frames (gsplat caching allocator design) |
| 94 | +- Cooperative shared-memory tile blending with block-vote early exit (3DGS, gsplat) |
| 95 | +- Fixed-budget MCMC training with static shapes, relocation plus covariance noise (gsplat MCMCStrategy, Kheradmand et al. 2024) |
| 96 | +- Opacity and scale regularizers (gsplat mcmc preset) |
| 97 | +- Progressive resolution fine-tuning (coarse-to-fine, 3DGS) |
| 98 | +- Per-parameter Adam learning-rate schedules (gsplat, 3DGS) |
| 99 | +- L1 plus D-SSIM photometric loss (3DGS, gsplat, via dm-pix) |
| 100 | +- Camera pose gradients via `render(..., diff_wrt=("viewmat",))` (gsplat projection backward) |
| 101 | +- Batch-native backward passes, `jax.vmap(jax.grad(render))` runs as one batched launch (gsplat) |
| 102 | +- Batched training steps with sqrt-batch learning-rate scaling (gsplat `batch_size` and `steps_scaler`) |
| 103 | +- Anti-aliased opacity compensation (Mip-Splatting, gsplat), depth regularization from COLMAP points (gsplat `depth_loss`), per-image exposure correction (gsplat appearance optimization), all opt-in |
32 | 104 |
|
33 | | -- Native multi-camera batched rendering, single launch with camera id in the sort key — up to 5× per-frame at batch 8, ~60k renders/s at 128² batch 1024 (gsplat) |
34 | | -- Opacity-aware tight tile intersection, −40–81% fewer intersections and −15–76% frame time on real trained scenes (lego 313k, drone 150k) (StopThePop §B.2; Speedy-Splat SNUGBOX/AccuTile, gsplat #927) |
35 | | -- Persistent signature-keyed sort/bin scratch, kills per-frame realloc (gsplat caching-allocator design) |
36 | | -- Cooperative shared-memory tile blending with block-vote early exit (3DGS; gsplat) |
37 | | -- Verified negatives, kept documented: 64-thread/4-px CTA blend (1.5–3.7× slower) and warp-level backward reduction (1.9× slower on dense drone; wins random 1.05–1.80×) — both occupancy/barrier-limited under warp-lang, re-confirmed on clean lego/drone scenes (gsplat designs; reports/phase5_o4.md, phase6b.md, phase6g_lego_reverification.md) |
38 | | -- Fixed-budget MCMC training in JAX (static shapes): dead-gaussian relocation + per-step covariance noise, +1.9 dB on the lego fit (gsplat MCMCStrategy; Kheradmand et al. 2024) |
39 | | -- Opacity/scale regularizers from the MCMC paper — opacity_reg is load-bearing for a fixed budget with no pruning, +8.9 dB on the lego fit (gsplat mcmc preset) |
40 | | -- Progressive 400²→800² resolution fine-tune, +5–6 dB by resolving fine detail at the eval resolution (coarse-to-fine, 3DGS) |
41 | | -- Per-parameter Adam LR schedules, means exponentially decayed ~100× over training (gsplat/3DGS create_splats_with_optimizers) |
42 | | -- L1 + 0.2·D-SSIM photometric loss mix, +0.7 dB on the lego fit (3DGS; gsplat, via dm-pix) |
43 | | -- Camera-pose (viewmat) gradients via `render(..., diff_wrt=("viewmat",))` — dedicated projection-backward kernels sharing all vjp math; recovers a 5°/0.2 pose perturbation to <1e-3°/5e-7 on lego. The 12-float v_viewmat reduction uses a block `tile_sum`, 22–113× faster than plain atomics at 50k–742k gaussians (gsplat ProjectionEWA3DGSFused bwd, reports/phase6e.md) |
44 | | -- Batch-native backward passes — `jax.vmap(jax.grad(render))` runs one batched backward launch (no per-sample loop) for every `diff_wrt`, up to 8.4× faster than the loop and ~2.1× for simultaneous multi-pose recovery (reports/phase8a_batched_grads.md) |
45 | | -- Batched training steps + √batch LR scaling — B views/step amortize per-step fixed cost, −12% wall at PSNR parity on the mid-N (150k) drone fit; scale-dependent, so B=1 stays default (gsplat `batch_size`/`steps_scaler`, reports/phase8l_batched_training.md) |
| 105 | +## Installation |
46 | 106 |
|
47 | | -## Benchmarks & reports |
| 107 | +Requires an NVIDIA GPU and a CUDA-enabled JAX (`jax[cuda12]`, pulled in as a dependency). |
48 | 108 |
|
49 | 109 | ```sh |
50 | | -.venv/bin/python scripts/benchmark.py <label> --backend {warp,cuda} [--mem] # real scene: lego.ply by default |
51 | | -.venv/bin/python scripts/benchmark_train.py <label> # training steps |
52 | | -.venv/bin/python scripts/plot_results.py <label> [<label2>] |
53 | | -.venv/bin/python scripts/render_lego.py # correctness gate vs ground truth |
| 110 | +uv pip install "git+https://github.com/amacati/splax" |
54 | 111 | ``` |
55 | 112 |
|
56 | | -Per-phase reports with figures live in `reports/` (phase1–4: setup, JAX FFI |
57 | | -migration, vmap, Warp port; phase5_*: one report per ported gsplat |
58 | | -optimization, including verified negatives; phase6*: training support). |
| 113 | +Developer setup with [pixi](https://pixi.sh/), which installs splax editable with the dev tooling: |
| 114 | + |
| 115 | +```sh |
| 116 | +git clone https://github.com/amacati/splax.git |
| 117 | +cd splax |
| 118 | +pixi shell |
| 119 | +``` |
59 | 120 |
|
60 | 121 | ## License |
61 | 122 |
|
62 | | -MIT (see [LICENSE](LICENSE)). |
63 | | -gsplat-derived portions under Apache-2.0 ([licenses/Apache-2.0.txt](licenses/Apache-2.0.txt)). |
64 | | -Started from [jaxsplat](https://github.com/yklcs/jaxsplat), rewritten in Warp to drop the CUDA toolchain, with optimizations ported from [gsplat](https://github.com/nerfstudio-project/gsplat) and the papers behind it (3DGS, StopThePop, Speedy-Splat). |
| 123 | +MIT (see [LICENSE](LICENSE)). gsplat-derived portions are under Apache-2.0 ([licenses/Apache-2.0.txt](licenses/Apache-2.0.txt)). |
0 commit comments