ESIM: an Open Event Camera Simulator

This repository contains the source code for the improved ESIM event camera simulator used in Deblur e-NeRF and Robust e-NeRF . In particular, we incorporate the following changes:

1. Event simulation model
   • Improve the overall event simulation accuracy by accounting for additional edge cases
   • Improve the refractory period model by explicitly resetting the pixel reference intensity and timestamp at the end of the refractory period
   • Desynchronize initial event generation across pixels by randomly initializing pixel reference timestamps
   • Model junction leakage, which increases the rate of ON events and decreases the rate of OFF events
   • Modify the pixel-to-pixel contrast threshold variation model to be time-independent
   • Merge feature/color branch to support color event cameras
   • Model dark current
   • Accurately model the band-limiting behavior of the pixel with a 5th-order Non-Linear Time-Invariant (NLTI) Low-Pass Filter (LPF)
2. Rendering engine
   • Support Blender as a rendering engine
   • Support Unreal Engine 4.27.2 with a modified UnrealCV plugin
3. Camera trajectory
   • Circumvent singularities in interpolating quaternion orientations by supporting rotation vector/angle-axis orientation representation in the trajectory CSV for interpolation
4. Miscellaneous
   • Fix various bugs & installation errors

If you use this improved version of ESIM for your work, please cite:

@inproceedings{low2024_deblur-e-nerf,
  title = {Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions},
  author = {Low, Weng Fei and Lee, Gim Hee},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year = {2024}
}

@inproceedings{low2023_robust-e-nerf,
  title = {Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion},
  author = {Low, Weng Fei and Lee, Gim Hee},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year = {2023}
}

@inproceedings{rebecq18_esim,
  author = {Henri Rebecq and Daniel Gehrig and Davide Scaramuzza},
  title = {{ESIM}: an Open Event Camera Simulator},
  journal = {Conf. on Robotics Learning (CoRL)},
  year = 2018
}

Installation

The following installation steps were tested on Ubuntu 20.04 and 22.04 with GTX 1080 Ti, RTX 3090, RTX 4090, RTX A5000 and RTX 6000 GPUs.

ESIM

We recommend creating a new Catkin workspace specifically for ESIM. We name the workspace as esim_ws and place it under the home directory, as follows:

mkdir -p ~/esim_ws/src

Clone this repository into the source space of the workspace with:

cd ~/esim_ws/src
git clone https://github.com/wengflow/rpg_esim.git

We also recommend using Conda to set up an environment with the appropriate dependencies for running ESIM, as follows:

1. Install Mamba, an improved re-implementation of Conda, according to the official instructions
2. Create the esim environment with:

   mamba env create -f rpg_esim/environment.yml

3. Activate the environment and initialize rosdep with:

   conda activate esim
   rosdep init
   rosdep update

Initialize and configure the workspace with:

catkin init
catkin config --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo
vcs-import < rpg_esim/dependencies.yaml

cd ze_oss
touch imp_3rdparty_cuda_toolkit/CATKIN_IGNORE \
      imp_app_pangolin_example/CATKIN_IGNORE \
      imp_benchmark_aligned_allocator/CATKIN_IGNORE \
      imp_bridge_pangolin/CATKIN_IGNORE \
      imp_cu_core/CATKIN_IGNORE \
      imp_cu_correspondence/CATKIN_IGNORE \
      imp_cu_imgproc/CATKIN_IGNORE \
      imp_ros_rof_denoising/CATKIN_IGNORE \
      imp_tools_cmd/CATKIN_IGNORE \
      ze_data_provider/CATKIN_IGNORE \
      ze_geometry/CATKIN_IGNORE \
      ze_imu/CATKIN_IGNORE \
      ze_trajectory_analysis/CATKIN_IGNORE

We set -DCMAKE_BUILD_TYPE=RelWithDebInfo, instead of -DCMAKE_BUILD_TYPE=Release, to preserve the ease of debugging while still enabling code optimization during compilation.

Build the esim_ros package with:

catkin build esim_ros

Lastly, add the following alias to your .bashrc file for activating the esim Conda environment and sourcing the ROS environment setup files:

alias cae='conda activate esim; source ~/esim_ws/devel/setup.bash'

The installation steps described above were consolidated from the following sources, with some modifications:

1. https://github.com/uzh-rpg/rpg_esim/wiki/Installation
2. https://github.com/uzh-rpg/rpg_esim/wiki/Installation-(ROS-Melodic)
3. https://robostack.github.io/GettingStarted.html

Blender

To use Blender as the rendering engine, we require Blender as a Python module being installed in a separate blender Conda environment with a compatible Python version and PyZMQ also installed.

We provide two Python wheels for Blender 3.4.0, which were compiled with CUDA Toolkit 11.4/11.8 and NVIDIA OptiX 7.3.0 support (requires NVIDIA R465.84 driver or newer), for Linux and Python 3.10. They are available here (CUDA Toolkit 11.4) and here (CUDA Toolkit 11.8).

The blender environment may be created with all the necessary dependencies, including the wheel compiled with CUDA Toolkit 11.8, with the following:

conda env create -f rpg_esim/blender_environment.yml

Alternatively, build Blender as a Python module from source, according to the official instructions, and install it via pip in the blender environment.

UnrealEngine and UnrealCV

To use UnrealEngine as the rendering engine, first install UnrealEngine 4.27 (Windows/Mac, Linux). Then, build a compatible UnrealCV Plugin from this modified source code, according to the official instructions.

Using Blender as the Rendering Engine

Configuration Options

Configuration options for the Blender rendering engine are detailed at the top of blender_renderer.cpp of the imp_blender_renderer ROS package. Note that with N GPUs, the valid blender_render_device_type and blender_render_device_id combinations are as follows:

blender_render_device_type	blender_render_device_id
0 (CPU)	N
1 (CUDA)	0 to N-1
2 (OptiX)	N+1 to 2N

A sample usage is provided in cfg/blender.conf of the esim_ros package.

Running ESIM with Blender

First, start the Blender rendering server on a given port (e.g. 5555) with:

cae
roscd imp_blender_renderer
conda activate blender
python scripts/blender_bridge.py --port 5555

Then, run ESIM under a given configuration (e.g. cfg/blender.conf) in another terminal with:

cae
roscd esim_ros
roslaunch esim_ros esim.launch config:=cfg/blender.conf

To visualize the output of the simulator, you can open rviz (from a new terminal) as follows:

cae
roscd esim_visualization
rviz -d cfg/esim.rviz

You can also open rqt for more visualizations, as follows:

cae
roscd esim_visualization
rqt --perspective-file cfg/esim.perspective

Please refer to the ESIM wiki for further details on the usage of other rendering engines.