PowerPoint Presentation

[Audio] Hello, everyone! Today, we are excited to introduce you to our new work, WaveNeRF - Wavelet-based Generalizable Neural Radiance Fields. WaveNeRF is a novel neural radiance field model that brings frequency information to the process of novel view synthesis. But first, let's understand why this is so important..

[Audio] Neural Radiance Fields have demonstrated remarkable performance in novel view synthesis through implicit scene representation. However, they have faced a significant challenge—poor scalability. This is because they require densely sampled images for each new scene. To address this issue, previous studies attempted to integrate Multi-View Stereo techniques into NeRF. While this helped, it still involved a cumbersome fine-tuning process for new scenes. Collecting a substantial number of training images for each scene was often challenging..

[Audio] One key observation was that rendering errors primarily occurred in image regions with rich high-frequency information. This loss of high-frequency detail was largely attributed to down-sampling operations during feature extraction, like those used in CNN networks or the MVS module..

[Audio] In response to these challenges, we introduce WaveNeRF, a model that incorporates explicit high-frequency information into the training process. This eliminates the need for per-scene fine-tuning under the generalizable and few-shot setting. WaveNeRF proposes a new MVS module named Wavelet Multi-View Stereo (WMVS) to incorporate scene wavelet coefficients into the Multi-View Stereo process, achieving frequency-domain modeling. Unlike other frequency transformations, Wavelet Transform is coordinate invariant, preserving the relative spatial positions of pixels..

[Audio] With WMVS, we can build a frequency-based radiance field, allowing our designed Hybrid Neural Renderer (HNR) to use information from both the spatial and frequency domains. This significantly enhances rendering quality, especially in high-frequency regions..

[Audio] WaveNeRF also features a Frequency-guided Sampling Strategy (FSS) that directs the model to focus on regions with larger high-frequency coefficients. This results in improved rendering quality by denser point sampling around object surfaces..

[Audio] To demonstrate the performance of WaveNeRF, we conducted extensive performance evaluations. We compared it to existing generalizable NeRF models, such as PixelNeRF, MVSNeRF, PointNeRF, and GeoNeRF. We evaluated these models using three input views to ensure a fair comparison. Our quantitative assessments included metrics like PSNR, SSIM, and LPIPS. These metrics clearly indicate that WaveNeRF outperforms previous generalizable models, even under challenging conditions..

[Audio] Qualitatively, WaveNeRF shines as well. We compared it with existing methods on various datasets, and the results speak for themselves. Our model preserves scene details and produces images with fewer artifacts..