Enhancing Sparse-view 3D Reconstruction with LM-Gaussian: Leveraging Large Model Priors for High-Quality Scene Synthesis from Limited Images

Recent advancements in sparse-view 3D reconstruction have focused on novel view synthesis and scene representation techniques. Methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have shown significant success in accurately reconstructing complex real-world scenes. Researchers have proposed various enhancements to improve performance, speed, and quality. Sparse view scene reconstruction techniques employ regularization methods and generalizable reconstruction priors to address the challenges of limited input views. Recent approaches like SparseGS, pixelSplat, and MVSplat have further improved upon these foundations.

Unposed scene reconstruction remains a challenge, with many existing methods relying on known camera poses. Techniques such as iNeRF, NeRFmm, BARF, and GARF have explored strategies for estimating and optimizing camera poses alongside scene representation. However, these methods still face difficulties with complex camera trajectories. The introduction of LM-Gaussian represents a new direction in this field, incorporating large model priors to enhance reconstruction quality from limited images. This approach builds upon previous work while addressing persistent challenges in sparse-view 3D reconstruction.

LM-Gaussian addresses sparse-view 3D reconstruction challenges by generating high-quality outputs from limited input images. The method incorporates a robust initialization module utilizing stereo priors for camera pose recovery and reliable point cloud generation. An Iterative Gaussian Refinement Module employs diffusion-based techniques to enhance image details and preserve scene characteristics during 3D Gaussian Splatting optimization. Video diffusion priors further improve rendered images for realistic visual effects. This approach significantly reduces data acquisition requirements while maintaining high-quality 360-degree scene reconstruction. Experiments on public datasets validate the framework’s effectiveness in practical applications.

Previous 3D reconstruction methods like 3D Gaussian Splatting require numerous input images, making them impractical for real-world applications. These approaches struggle with sparse-view scenarios, leading to initialization failures, overfitting, and detail loss. Existing solutions employing frequency and depth regularization still produce cluttered results due to reliance on traditional Structure from Motion methods. LM-Gaussian addresses these limitations by integrating multiple large model priors. The method comprises four key modules: Background-Aware Depth-guided Initialization, Multi-Modal Regularized Gaussian Reconstruction, Iterative Gaussian Refinement Module, and Video Diffusion Priors.

LM-Gaussian’s initialization module utilizes stereo priors from DUSt3R for camera pose estimation and point cloud creation. The reconstruction process employs photometric loss and additional constraints to optimize 3D models. The iterative refinement module applies a diffusion-based Gaussian repair model to enhance image quality and incorporate high-frequency details. Validation experiments on public datasets demonstrate LM-Gaussian’s ability to produce high-quality 360-degree scene reconstructions with significantly reduced data acquisition requirements. This comprehensive methodology effectively addresses sparse-view 3D reconstruction challenges through innovative initialization, regularization, and refinement techniques.

LM-Gaussian demonstrates significant advancements in sparse-view 3D reconstruction, outperforming baseline methods like DNGaussian and SparseNerf. Quantitative metrics, including PSNR, SSIM, and LPIPS, show improved reconstruction quality and finer details in rendered images. The method excels with limited input data, achieving high-quality reconstructions from just 16 images. Multi-modal regularization techniques enhance performance, resulting in smoother surfaces and reduced artifacts. LM-Gaussian consistently outperforms the original 3DGS across varying numbers of input images, though its advantages diminish in denser setups.

The method’s effectiveness is particularly evident in sparse-view scenarios, where it preserves structures and details better than competitors. Visual quality improvements include smoother surfaces and fewer artifacts like black holes and sharp angles. LM-Gaussian significantly reduces data acquisition requirements compared to traditional 3DGS methods while maintaining high-quality results in 360-degree scenes. These achievements position LM-Gaussian as a robust solution for practical 3D reconstruction applications, effectively addressing the challenges of limited input data and demonstrating superior performance in sparse-view conditions.

In conclusion, LM-Gaussian presents a novel approach to sparse-view 3D reconstruction, leveraging priors from large vision models. The method incorporates a robust initialization module, multi-modal regularizations, and iterative diffusion refinement to enhance reconstruction quality and prevent overfitting. It significantly reduces data acquisition requirements while achieving high-quality results in complex 360-degree scenes. Although currently limited to static scenes, LM-Gaussian demonstrates substantial advancements in the field. Future work aims to incorporate dynamic 3DGS methods, potentially expanding the method’s applicability to dynamic modeling and further improving its effectiveness in various 3D reconstruction scenarios.

---

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter..

Don’t Forget to join our 50k+ ML SubReddit

⏩ ⏩ FREE AI WEBINAR: ‘SAM 2 for Video: How to Fine-tune On Your Data’ (Wed, Sep 25, 4:00 AM – 4:45 AM EST)