Abstract

---

In recent years, Monocular Depth Estimation (MDE) has evolved from predicting affine-invariant relative depth to estimating metric-scale (absolute) depth. However, local geometric inconsistencies in single-view depth maps and scale inconsistencies across different views still severely hinder their practical application in 3D matching and relative pose estimation. To address these challenges, we propose a geometry-guided depth correction framework for metric-scale relative pose estimation. Our approach first leverages pre-trained foundation models to extract initial metric depth, semi-dense correspondences, and high-dimensional semantic features from dual-view images. We then introduce a Local Depth Residual Learning (LDRL) module to correct geometric deviations. Finally, the corrected depth of stereo-matched pairs is integrated into a differentiable RANSAC framework to jointly optimize the relative pose with consistent scale. Experiments on ScanNet and 7-Scenes demonstrate that our method achieves superior performance and robustness across various challenging scenarios.

Depth Correction Comparison

---

The figure below illustrates the key difference between existing depth correction methods (based on affine-invariance assumptions) and our proposed geometry-guided approach. While global affine correction (scale $s$ and translation $t$) or relative scale normalization ($s_g$, $\alpha$, $\beta_1$, $\beta_2$) fails to capture local geometric distortions, our patch-wise depth correction produces smaller errors and stronger local consistency.

Framework Overview

---

Our framework leverages the comprehensive perception of multiple vision foundation models to address both intra-view geometric inconsistencies and inter-view scale misalignments in stereo depth estimation, consisting of three core stages:

• Multi-Prior Extraction: A monocular depth estimation foundation model (DepthPro) obtains initial metric depth maps; DINOv2 extracts high-dimensional semantic feature maps; and a semi-dense matcher (ELoFTR) establishes cross-view pixel correspondences.
• Local Depth Residual Learning (LDRL): A "Correction Transformer" with alternating Self-Attention and Windowed Cross-Attention (WCA) layers predicts local depth residuals $\Delta s$ for each matched pixel, enabling fine-grained depth correction at the pixel level.
• Differentiable Geometric Solving: The corrected 3D metric point correspondences are fed into a geometric solver equipped with differentiable RANSAC, enabling end-to-end joint optimization of pose loss and 3D geometric consistency loss.

Local Depth Residual Learning & Windowed Cross-Attention

---

To overcome geometric inconsistencies in stereo depth, we predict a local multiplicative residual $\Delta s$ for each matched patch instead of global affine parameters. We employ an exponential mapping in log-space to perform non-linear depth rectification, ensuring physical positivity:

$\hat{D} = D^{prior} \cdot \exp(\beta \cdot \Delta s),$

where $\Delta s \in [-1, 1]$ and the scaling constant $\beta = \ln(5)$ provide a versatile correction range of $[0.2, 5.0] \times D^{prior}$. This mechanism offers inherent scale symmetry and substantial local flexibility, enabling the network to selectively rectify geometric inconsistencies in challenging regions such as object boundaries.

For the Windowed Cross-Attention (WCA) mechanism, we explicitly constrain the search space of cross-attention using the matching prior. For a query coordinate $\mathbf{x}_i$ in image $I_0$, we define a local spatial window $\mathcal{W}(\mathbf{x}_i, \sigma)$ centered at the anchor pixel in the reference image $I_1$. This design ensures that feature interactions are explicitly focused on potential correspondence regions:

$\text{WCA}(f_0^i) = \text{Softmax}\left(\frac{f_0^i K_{win}^{i\top}}{\sqrt{d}}\right) V_{win}^i.$

Differentiable Pose Estimation

---

After obtaining the corrected depth of matched points, the 2D correspondences are lifted into 3D space using the known camera intrinsic matrix $\mathbf{A}$:

$\mathbf{X}_0^i = \hat{d}_0^i \cdot \mathbf{A}^{-1} \bar{\mathbf{x}}_0^i.$

Drawing on the probabilistic selection paradigm of DSAC, we employ a soft-selection mechanism over pose hypotheses, directly utilizing the confidence scores $\omega_i$ provided by the matcher as weights to solve for the optimal relative pose $(\mathbf{R}, \mathbf{t})$:

$\min_{\mathbf{R}, \mathbf{t}} \sum_{i=1}^{N} \omega_i \cdot \rho(\|(\mathbf{R}\mathbf{X}_0^i + \mathbf{t}) - \mathbf{X}_1^i\|_2).$

Experimental Results

---

1. Metric Relative Pose Estimation on ScanNet-1500

Table 1 reports Pose error AUC (%) at thresholds of $5^\circ$, $10^\circ$, and $20^\circ$. Our method (DepthPro + LDRL) significantly outperforms existing metric-scale methods, achieving 33.83% AUC@5 compared to 25.70% from Reloc3r-metric-pose. Notably, using ELoFTR with the original DepthPro depth map yields only 3.64% AUC@5, demonstrating that LDRL successfully corrects local geometric deviations.

2. Geometric Correction Strategy Analysis on ScanNet-1500

Table 2 compares our LDRL against standard feature matchers paired with different depth processing techniques. While global affine correction improves rotation estimation, its improvement on translation metrics remains limited. Our LDRL achieves the best results across all metrics.

3. Cumulative Depth Error Distribution

Figure 3 compares the percentage of patches satisfying different absolute error thresholds. LDRL (blue bars) outperforms Initial Depth (red bars) across all tolerances, achieving a fourfold improvement at the strict $\leq 0.1m$ threshold (0.44 vs. 0.11).

4. Metric Pose Estimation on 7-Scenes

Table 4 evaluates our method on the 7-Scenes visual localization task. Despite using only a single reference image, our method achieves average translation error of 0.05m and rotation error of 1.2°, significantly outperforming all single-pair methods and approaching state-of-the-art multi-view methods.

Ablation Study

---

Table 5 demonstrates LDRL's contribution across both ScanNet and 7-Scenes datasets. On 7-Scenes, LDRL improves AUC@10 from 20.76% to 64.57%, with median translation error reducing from 0.14m to 0.05m.

Conclusion

---

We present a geometry-guided depth correction framework that addresses local geometric inconsistency and cross-view scale misalignment in monocular depth estimation for metric-scale relative pose estimation. By introducing Local Depth Residual Learning with Windowed Cross-Attention and a differentiable RANSAC framework for joint optimization, our method achieves fine-grained depth correction beyond global affine alignment. Extensive experiments on ScanNet and 7-Scenes demonstrate superior accuracy and robustness in metric-scale pose estimation compared to both end-to-end regression and existing alignment methods, providing a practical solution for high-precision visual localization and 3D reconstruction tasks.