【读论文】3D Gaussian Splatting for Real-Time Radiance Field Rendering_3d guassian splatting 论文

1. What：

What kind of thing is this article going to do (from the abstract and conclusion, try to summarize it in one sentence)

To simultaneously satisfy the requirements of efficiency and quality, this article begins by establishing a foundation with sparse points using 3D Gaussian distributions to preserve desirable space. It then progresses to optimizing anisotropic covariance to achieve an accurate representation. Lastly, it introduces a cutting-edge, visibility-aware rendering algorithm designed for rapid processing, thereby achieving state-of-the-art results in the field.

2. Why：

Under what conditions or needs this research plan was proposed (Intro), what problems/deficiencies should be solved at the core, what others have done, and what are the innovation points? (From Introduction and related work)

Maybe contain Background, Question, Others, Innovation:

Three aspects of related work can explain this question.

1. Traditional reconstructions such as SfM and MVS need to re-project and
   blend the input images into the novel view camera, and use the
   geometry to guide this re-projection(From 2D to 3D).

   Sad: Cannot completely recover from unreconstructed regions, or from “over-reconstruction”, when MVS generates inexistent geometry.

2. Neural Rendering and Radiance Fields

   Neural rendering represents a broader category of techniques that leverage deep learning for image synthesis, while radiance field is a specific technique within neural rendering focused on the scene representation of light and color in 3D spaces.

• Deep Learning was mainly used on MVS-based geometry before, which is also its major drawback.

• Nerf is along the way of volumetric representation, which introduced positional encoding and importance sampling.

• Faster training methods focus on the use of spatial data structures to store (neural) features that are subsequently interpolated during volumetric ray-marching, different encodings, and MLP capacity.

• Today, notable works include InstantNGP and Plenoxels both rely on Spherical Harmonics.

  Understand Spherical Harmonics as a set of basic functions to fit a geometry in a 3D spherical coordinate system.

  球谐函数介绍（Spherical Harmonics） - 知乎 (zhihu.com)

3. Point-Based Rendering and Radiance Fields

• The methods in human performance capture inspired the choice of 3D Gaussians as scene representation.
• Point-based and spherical rendering is achieved before.

3. How：

Through the Gradient Flow in this paper’s pipeline, we are trying to connect Part4, 5, and 6 in this paper.

Firstly, start from the loss function, which is combined by a ${\mathcal{L}}_1$ loss and a $SSIM$ index, just as shown below:

$$\mathcal{L}=(1-\lambda ){\mathcal{L}}_1+\lambda {\mathcal{L}}_{\mathrm{D}-\mathrm{SSIM}}.\text{\hspace{1em}(1)}$$

It found a relation between the actual image and the rendering image. So to finish the optimization, we need to dive into the process of rendering. From the chapter on related work, we know Point-based $\alpha$-blending and NeRF-style volumetric rendering share essentially the same image formation model. That is

$$C=\sum _{i=1}^N{T}_i(1-\exp (-{\sigma }_i{\delta }_i))c_i\mathrm{with}{T}_i=\exp \left(-\sum _{j=1}^{i-1}{\sigma }_j{\delta }_j\right).\text{\hspace{1em}(2)}$$

And this paper actually uses a typical neural point-based approach just like (2), which can be represented as:

$$C=\sum _{i\in N}{c}_i{\alpha }_i\prod _{j=1}^{i-1}(1-{\alpha }_j)\text{\hspace{1em}(3)}$$

From this formulation, we can know what the representation of volume should contain the information of color $c$ and transparency $\alpha$. These are attached to the gaussian, where Spherical Harmonics was used to represent color, just like Plenoxels. The other attributes used are the position and covariance matrix. So, now we have introduced the four attributes to represent the scene, that is positions