method.tex

\section{Method}

\subsection{Problem Formulation}

We define a scene as an underlying world \(S\). An RGB-D image is an observation \(\mathcal{O}\) sampled from this scene. Given an observation parameter \(c\) that specifies the sampling configuration, such as position, scale, and viewing angle, the observation operator produces:
\begin{equation}
\mathcal{O}_c = (I_c, D_c, M_c) = \mathcal{R}(S,c).
\end{equation}

Each observation is defined on a pixel domain \(\Omega_c=\{1, \ldots, H\} \times \{1, \ldots, W\}\), where \(I_c: \Omega_c \to \mathbb{R}^3\) is the RGB image, \(D_c: \Omega_c \to \mathbb{R}\) is the incomplete scene depth, and \(M_c: \Omega_c \to \{0,1\}\) is the depth-validity mask.
When \(c\) is fixed, we write \(\mathcal{O}=(I,D,M)\) on \(\Omega\). Let \(u \in \Omega\) denote a pixel.
\begin{equation}
\Omega_v = \{u \in \Omega : M(u)=1\}, \qquad
\Omega_h = \Omega \setminus \Omega_v .
\end{equation}

Only \(D|_{\Omega_v}\) is observed. For each observation, the goal of depth inpainting is to find a depth estimator \(f^*\), whose predicted depth \(\tilde D\) best approximates the complete depth map \(\hat D\). To simplify the derivation, we assume that \(f\) is a monocular depth estimator:
\begin{equation}
\tilde D = f(I), \qquad
f^* = \arg\min_f \mathcal{L}\bigl(f(I),\hat D\bigr).
\end{equation}

\subsection{Neural Depth Field}

Let \(f_{\theta}\) be a monocular depth estimator with parameters \(\theta\). For the input image \(I\), it gives a one-shot depth prediction \(f_{\theta}(I)\). Since \(I\) is fixed for a given observation, \(\theta\) defines a scene-level implicit depth field \(F_{\theta}: \Omega \to \mathbb{R}\), where
\begin{equation}
F_{\theta} = f_{\theta}(I)
\implies
F_{\theta}(u) = [f_{\theta}(I)](u).
\end{equation}

Therefore, the monocular depth model has two roles. As a predictor, \(f_\theta\) maps RGB to depth. As a field, \(F_{\theta}\) represents the depth surface of the current observation. Our objective is to minimize the discrepancy between the complete and the predicted depth:
\begin{equation}
\theta^* =
\arg\min_\theta \mathbb{E}_{c\sim\mu}[\ell_c(\theta)]
=
\arg\min_\theta \mathbb{E}_{c\sim\mu}
\left[
\int_{\Omega_c}
\rho(F_{\theta,c}(u),\hat D_{c}(u))\,d\nu_c(u)
\right],
\end{equation}
where \(\theta^*\) denotes the optimal parameters, \(\mu\) is the distribution of \(c\), \(\ell_c\) is the loss term, \(\rho: \mathbb{R} \times \mathbb{R} \to \mathbb{R}_+\) is a pointwise loss, and \(\nu_c\) is the normalized counting measure on \(\Omega_c\). The loss can be further decomposed into an observed-depth fitting term and a hidden-depth term:
\begin{equation}
\ell(\theta)
=
\int_{\Omega}
M(u)\rho(F_{\theta}(u),D(u))\,d\nu(u)
+
\int_{\Omega}
(1-M(u))\rho(F_{\theta}(u),\hat D(u))\,d\nu(u).
\end{equation}

Since \(\hat D\) is unknown on \(\Omega_h\), we can approximate it using a neural network \(g_{\lambda}\) based on the observation \(D\) and another available prior \(\Pi_{g_{\lambda}}\), where \(\lambda\) denotes the parameters. The optimization objective is:
\begin{equation}
\lambda^*
=
\arg\min_\lambda
\mathbb{E}_{c\sim\mu}
\left[
\int_{\Omega_c}
M_{c}(u)\rho(g_{\lambda}(u),D_c(u))\,d\nu_c(u)
+
\Pi_{g_{\lambda}}
\right].
\end{equation}

Notice that \(g_{\lambda}\) can be any optimizable neural network, including \(F_{\theta}\). In this case:
\begin{equation}
\ell_c(\theta)
\sim
\int_{\Omega_c}
M_c(u)\rho(F_{\theta,c}(u),D_c(u))\,d\nu_c(u)
+
\alpha\Pi_{F_{\theta,c}},
\end{equation}
where \(\alpha\) is a constant, and \(\Pi_{F_{\theta,c}}\) can be a pretrained depth prior, a smoothness regularizer, or any other constraint. Through the NDF parameterization \(F_{\theta,c}\), this single optimization objective unifies inpainting in the missing regions and reconstruction in the observed regions.