Real-time Streaming Support for DA3-Streaming

[Ga0512]

PR: Real-time Streaming Support for DA3-Streaming

Description

Refactors da3_streaming.py to support true streaming input (video files and webcam) alongside the original batch mode (image directories). Processes frames in fixed-size chunks with bounded memory, enabling online 3D reconstruction without accumulating the entire sequence in RAM.

Changes

• New streaming pipeline: reads chunk_size frames, infers depth/poses, aligns via Sim3, renders preview, discards old frames, repeats
• Batch mode preserved: image folder input behaves identically to original (with optional loop closure)
• Multi-source support: --video FILE (streaming), --video 0 (webcam), --image_dir PATH (batch)
• Real-time preview: side-by-side visualization with accumulated 3D point cloud during processing
• Identical output: final aligned PLYs, camera poses, and intrinsics identical to original pipeline

Architecture

Streaming (video/webcam)           Batch (images)
├─ Read chunk_size frames          ├─ Load all images
├─ DA3 inference                   ├─ Compute chunk_indices
├─ Sim3 alignment                  ├─ Process sequentially
├─ Preview PCD                     ├─ Optional loop closure
├─ Save .npy to disk               ├─ Preview PCD
├─ Discard old frames              └─ Save .npy to disk
└─ Repeat                          
        ↓
   Both: accumulate_sim3 → save final PLYs

Memory Usage

• Streaming: O(chunk_size) frames in RAM
• Batch: O(total_frames) in RAM

Example: 1000-frame 1080p sequence, chunk_size=60

• Streaming: ~200 MB
• Batch: ~2 GB

Testing

• Video file input (h264, h265, various codecs)
• Webcam input (cv2.VideoCapture(0))
• Image folder input (backward compatible)
• Sim3 alignment correctness (matches original)
• Final PCD output (identical to batch mode)
• Preview rendering (pyvirtualdisplay headless)
• Frame discard (no memory leaks over long sequences)

Usage

# True streaming (chunk-by-chunk processing)
python da3_live.py --video video.mp4 --headless

# Webcam
python da3_live.py --video 0 --headless

# Original batch mode
python da3_live.py --image_dir /path/to/images --headless

# Custom parameters
python da3_live.py --video video.mp4 --chunk_size 30 --overlap 15 --headless

Backward Compatibility

• Original da3_streaming.py unchanged
• --image_dir behavior identical
• Output format and PLY quality unchanged
• All original configs supported

Notes

• Loop closure only available in batch mode (--image_dir) — streaming mode disables it
• Webcam input runs indefinitely until Ctrl+C
• First chunk's viewpoint is set by Open3D auto-fit; subsequent chunks maintain that view for visual continuity
• Final PLYs are always globally aligned regardless of streaming vs batch

Demo:

reconstrucao_3d_final.mp4