Structural changes for BundleSDF-PLL cyclic pipeline

[ebianchi]

Current work-in-progress for file reorganization efforts. The new file structure is documented here and will be able to keep track of:

• individual BundleSDF experiment runs for tracking,
• individual BundleSDF experiment runs for object reconstruction,
• incorporation of any PLL experiment results.

Cyclic BundleSDF-PLL experiments

The new organization will allow for more seamless execution of the cyclic BundleSDF-PLL pipeline. The above diagram shows how the pipeline can be used for multiple iterations starting from BundleSDF trajectories (above the grey dashed line), or how the pipeline can be run for one iteration with TagSLAM trajectories to train PLL (below the grey dashed line). The blue iteration numbers or purple TagSLAM labels indicate how the experiments will be sorted in the new file structure.

File structure

The file structure re-design focuses on the 4 data groups from the above flow diagram.

1. BundleSDF pose outputs

BundleSDF pose outputs are expressed:

• in camera frame,
• of the BundleSDF tracking origin.

These outputs are stored per BundleSDF experiment under bundlenets/results/{DATASET}/bundlesdf_iteration_{C}/bundlesdf_id_{B}/. This structure lets multiple BundleSDF experiment runs (indicated by a BundleSDF run ID B) store results for the same video dataset (indicated by DATASET, e.g. cube_2) and for multiple iterations of the BundleSDF-PLL pipeline (iteration number indicated by C).

The git submodule cnets-data-generation converts these outputs of BundleSDF poses to the expected input format and directory for PLL. This conversion can be run via:

python cnets-data-generation/conversion_bsdf_to_pll.py --vision-asset=DATASET --bundlesdf-id=B --cycle-iteration=C

2. PLL pose inputs

PLL expects pose inputs:

• in world frame,
• of the TagSLAM origin.*

*As long as the origin is consistent throughout all of the input trajectories for a given experiment, the choice of origin is inconsequential for PLL. We chose to use the TagSLAM origin so that the trajectories produced by BundleSDF are easily compared with TagSLAM trajectories.

These inputs are stored under bundlenets/dair_pll/assets/vision_{OBJECT}/{DATASET}/toss/bundlesdf_iteration_{C}/bundlesdf_id_{B}/. There is also a corresponding bundlenets/dair_pll/assets/vision_{OBJECT}/{DATASET}/toss/tagslam/ directory for poses generated with TagSLAM instead of BundleSDF for ease of comparison.

dair_pll is included as a git submodule. Running dair_pll performs the conversion from PLL pose inputs to PLL geometry outputs. This process can be run as a PLL experiment (with PLL run ID P) via:

python dair_pll/examples/contactnets_vision.py  --run-name=P --vision-asset=DATASET --cycle-iteration=C --bundlesdf-id=B

3. PLL geometry outputs

PLL geometry outputs are expressed:

• in meters,
• from the TagSLAM origin.

These outputs are stored per PLL experiment under bundlenets/dair_pll/results/vision_{OBJECT}/{DATASET}/bundlesdf_iteration_{C}/pll_id_{P}/. This structure lets multiple PLL experiment runs (indicated by PLL run ID P) store results for the same BundleSDF run (indicated by BundleSDF run ID B).

Again, the git submodule cnets-data-generation converts these outputs of PLL geometry information to the expected input format and directory for BundleSDF. This conversion can be run via:

python cnets-data-generation/conversion_pll_to_bsdf.py --vision-asset=DATASET --pll-id=P --cycle-iteration=C

4. BundleSDF geometry inputs

BundleSDF expects geometry inputs:

• in meters (which are later converted to normalized units),
• from the BundleSDF tracking origin (which is later converted to BundleSDF shape reconstruction origin).

These inputs are stored under bundlenets/geometry/{DATASET}/bundlesdf_iteration_{C}/pll_id_{P}/.

Running bundlenets performs the conversion from BundleSDF geometry inputs to BundleSDF pose outputs. This process can be run as a BundleSDF experiment (with BundleSDF run ID B2) via:

python bundlenets/run_custom.py --run-name=B2 --vision-asset=DATASET --mode=run_video --pll-id=P --cycle-iteration={C+1}

Testing performed

The following four use cases for the bundlenets code were tested in validation of this PR:

1. BundleSDF mode run_video without PLL inputs.
   • Tracking results in results/cube_2/bundlesdf_iteration_1/bundlesdf_id_pr00/.
   • Shape reconstruction results in results/cube_2/bundlesdf_iteration_1/bundlesdf_id_pr00/nerf_runs/bundlesdf_id_pr00/.
2. BundleSDF mode test_loss without PLL inputs.
   • Shape reconstruction results in results/cube_2/bundlesdf_iteration_1/bundlesdf_id_pr00/nerf_runs/bundlesdf_id_pr01/.
3. BundleSDF mode run_video with PLL inputs.
   • Tracking results in results/cube_2/bundlesdf_iteration_2/bundlesdf_id_pr02/.
   • Shape reconstruction results in results/cube_2/bundlesdf_iteration_2/bundlesdf_id_pr02/nerf_runs/bundlesdf_id_pr02/.
4. BundleSDF mode test_loss with PLL inputs.
   • Shape reconstruction results in results/cube_2/bundlesdf_iteration_2/bundlesdf_id_pr02/nerf_runs/bundlesdf_id_pr03/.

The BundleSDF mode run_video performs sequentially a tracking experiment, then an object reconstruction experiment within the tracking experiment. Mode test_loss performs just an object reconstruction experiment within a previously run tracking experiment.

Tests were performed on the DAIR Lab desktop cnets-vision account. The repository was cloned with this development branch checked out at ~/mengti_ws/structural_tests/bundlenets/, though running the code reused the environment/docker settings from the mengti_ws/BundleSDF/ folder. The following additional data was copied over:

• Video data: data/cube_2/ was copied from mengti_ws/BundleSDF/ without any contained .pt files.
• Geometry data: geometry/cube_2/bundlesdf_iteration_1/pll_id_t03/ was copied from Bibit's laptop where a test was performed locally.

Then tests for each of the four use cases above were run with:

# 1) Tracking and shape reconstruction without PLL inputs
python bundlenets/run_custom.py --run-name=pr00 --vision-asset=cube_2 --mode=run_video --use-segmenter=1 --use-gui=0 --debug-level=2

# 2) Shape reconstruction only without PLL inputs
python bundlenets/run_custom.py --run-name=pr01 --tracking-run-name=pr00 --vision-asset=cube_2 --mode=test_loss --use-segmenter=1 --use-gui=0 --debug-level=2

# 3) Tracking and shape reconstruction with PLL inputs
python bundlenets/run_custom.py --run-name=pr02 --vision-asset=cube_2 --mode=run_video --pll-id=t03 --use-segmenter=1 --use-gui=0 --debug-level=2

# 4) Shape reconstruction only with PLL inputs
python bundlenets/run_custom.py --run-name=pr03 --tracking-run-name=pr02 --vision-asset=cube_2 --mode=test_loss --pll-id=t03 --use-segmenter=1 --use-gui=0 --debug-level=2

Code review

The changes herein contain many formatting tweaks for easier viewing. There are also some changes that are just renaming of variables or functions. There are many files included in the changes, so below is a list to help bring attention to the functional changes.

• bundlesdf.py
  • Drops use of with_cnets arguments, instead interpreting if PLL information is to be used based on the configuration settings.
  • Uses the new file structure to save all NeRF-related results within a tracking experiment's nerf_runs folder.
• file_utils.py
  • A new file that manages file-related things.
• nerf_helpers.py
  • Documents the custom-added loss terms, though no changes in functionality.
  • Includes a placeholder with some implementation progress on a gradient loss supervised by PLL data.
• nerf_runner.py
  • Loads geometry results from new geometry/ directory based on configuration settings.
  • Includes PLL-related cost terms only if the configuration includes a geometry/ sub-directory.
  • Includes a placeholder with some implementation progress on a gradient loss supervised by PLL data.
  • Ensures all loss terms are logged in the metrics dictionary.
• run_custom.py
  • New command line interface.
  • Loads and uses 'toss' versions of the base configuration files.

[ebianchi]

This file shows up as new, though it's moved from {REPO}/run_custom.py to {REPO}/bundlenets/run_custom.py and the changes were significant enough to call this a new file rather than a moved and edited file. This makes seeing the changes a little more difficult.

[ebianchi]

Removed here because all conversions are done by the cnets-data-generation git submodule.

[ebianchi]

@minghanz Thanks for pointing out that the former dry_cnets setting had a use that we'd like to preserve. The old dry_cnets flag let the PLL-related losses to be computed without contributing to the SDF training process. Now, that is achieved with setting the use_pll_for_training to false, but still providing a PLL run ID from which to use PLL information.