DGD-NeRF: Depth-Guided Dynamic NeRF

1Technical University of Munich
DGD-NeRF Pipeline

DGD-NeRF consists of three MLPs: a deformation network mapping deformations to a shared canonical volume; a rigidity network masking deformations with rigidity scores; and a canonical volume regressing radiance and volume density from every camera ray.

Abstract

Current techniques for novel view synthesis using neural radiance fields (NeRF) in non-rigid 3D scenes produce a scene representation that matches the ground truth with a noticeable error.

Our approach DGD-NeRF reduces this error by improving and extending existing dynamic NeRF by depth using an RGB-D dataset. We constrain the optimization by depth supervision and depth-guided sampling. Our method outperforms current techniques in rendering novel views of the learned scene representation.

Data Capturing

We built a custom RGB-D dataset of challenging scenes. We used an iPad to record RGB-D videos and the corresponding camera poses.


Ray Sampling

Depth guided sampling ensures that most samples in spatial locations near the first surface in the direction of the ray.

Ray sampling

Animations

Bullet-Time Effect

With dynamic NeRF you can create fun visual effects like in this scene from "Matrix". In the movie they used over 100 cameras to capture a scene that can create with one.

Re-rendering the Input Video

Using DGD-NeRF, you can re-render a video from a novel viewpoint such as a stabilized camera by playing back the training deformations.

BibTeX

@article{kirmayr2022dgdnerf,
  author    = {Kirmayr, Johannes and Wulff, Philipp},
  title     = {Dynamic NeRF on RGB-D Data},
  year      = {2022},  
  month     = {Jul},
  url       = {https://philippwulff.github.io/DGD-NeRF/}
}