【读论文】MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving

1. What

An autonomous driving simulator based on Nerf with three features: instance-aware, modular, and realistic.

2. Why

As for current autonomous driving algorithms, training the corner cases is helpful for their performance bottleneck.

Existing autonomous driving simulation methods have their own limitations, such as CARLA, AADS, and GeoSim.

Recently, Neural Scene Graph (NSG) decomposes dynamic scenes into learned scene graphs and learns latent representations for category-level objects. However, its multi-plane-based representation for background modeling cannot synthesize images under large viewpoint changes.

3. What

3.1 Inputs

The input to the system consists of a set of RGB-images { I i } N \{\mathcal{I}i\}^N {Ii}N, sensor poses { T i } N \{\mathcal{T}i\}^N {Ti}N (calculated using IMU/GPS signals), and object tracklets (including 3D bounding boxes { B i j } N × M \{\mathcal{B}_{ij}\}^{N\times M} {Bij​}N×M, categories { t y p e i j } N × M \{ \mathrm{type}_{ij}\} ^{N\times M} {typeij​}N×M, and instance IDs { i d x i j } N × M ) \{\mathrm{idx}_{ij}\}^{N\times M}) {idxij​}N×M). $N$ is the number of input frames and $M$ is the number of tracked instances { O j } M \{\mathcal{O}_j\}^M {Oj​}M across the whole sequence. An optional set of depth maps { D i } N \{\mathcal{D}_i\}^N {Di​}N and semantic segmentation masks.

3.2 Scene Representation

Architectures: It supports various NeRF backbones, which can be roughly categorized into two hyper-classes: MLP-based methods, or grid-based methods and this paper gives a formal exposition of grid-based methods(Instant-ngp).

Foreground Nodes: Similar to NSG, it exploits latent codes to encode
instance features and shared category-level decoders to encode class-wise priors.

3.3 Compositional Rendering

It uses the standard volume rendering process to render pixel-wise properties:

$$\begin{gathered} \hat{\mathbf{c}}(\mathbf{r})=\sum _{P_i}{T}_i{\alpha }_i{\mathbf{c}}_i+(1-\mathrm{accum})\cdot {\mathbf{c}}_{\mathrm{sky}},T_i=\exp (-\sum _{k=1}^{i-1}{\sigma }_k{\delta }_k) \\ \hat{d}(\mathbf{r})=\sum _{P_i}{T}_i{\alpha }_i{t}_i+(1-\mathrm{accum})\cdot \inf \\ \hat{\mathbf{s}}(\mathbf{r})=\sum _{P_i}{T}_i{\alpha }_i{\mathbf{s}}_i+(1-\mathrm{accum})\cdot {\mathbf{s}}_{\mathrm{sky}}. \end{gathered}$$

where P i ∈ s o r t e d ( { P i b g ,   o b j } ) P_{i}\in sorted(\{P_{i}^{\mathrm{bg,~obj}}\}) Pi​∈sorted({Pibg, obj​}), ${\alpha }_i=1-\exp (-{\sigma }_i{\delta }_i)$, ${\delta }_i=t_{i+1}-t_i$ , $accum={\sum }_{P_i}{T}_i{\alpha }_i$ , ${\mathbf{c}}_{\mathrm{sky}}$ is the rendered color from the Sky model, $inf$ is the upper bound distance, and $s_{\mathrm{sky}}$ is the one-hot semantic logits of the sky category.

As for the segment, we assign a one-hot vector to every object.

s k o b j − j [ l ] = { σ k o b j − j i f l = c a t e g o r y   o f   j ′ s   i n s t a n c e 0 o t h e r w i s e , for l in category . \left.\mathbf{s}_{k}^{\mathrm{obj-j}}[l]=\left\{

$$\begin{array}{cc}\sigma_{k}^{\mathrm{obj-j}}&\quad\mathrm{if} l=\mathrm{category~of~j's~instance} \\0&\quad\mathrm{otherwise}\end{array}$$ \right.\right.,\text{for l in category}. skobj−j​[l]={σkobj−j​0​ifl=category of j′s instanceotherwise​,for l in category.

3.4 Towards Realistic Rendering

1. Sky Model

   However, blending the sky color $c_{sky}$ with background and foreground
   rendering (Eq. 4) leads to potential inconsistency. Therefore, we introduce a BCE(Binary Cross Entropy) semantic regularization to alleviate this issue:

   $${\mathcal{L}}_{\mathrm{sky}}=\mathrm{BCE}(1-\mathrm{accum},{\mathcal{S}}_{\mathrm{sky}}).$$

2. Resolving Conflict Samples:

   Due to the fact that our background and foreground sampling are done independently, there is a chance that background samples fall within the foreground bounding box, causing incorrect classification of foreground samples as background samples.

   

   The ambiguity is NOT observed in NSG [21] as NSG only samples a few points on the ray-plane intersections, and is unlikely to have much background truncated samples.

   To address this issue, we devise a regularization term that minimizes the
   density sum of background truncated samples to minimize their influence during the rendering process as:

   $${\mathcal{L}}_{\mathrm{accum}}=\sum _{P_i^{(\mathrm{tr})}}{\sigma }_i,$$

   where { P i ( tr ) } \{P_i^{(\text{tr})}\} {Pi(tr)​} denotes background truncated samples.

3.5 Optimization

$$\mathcal{L}={\lambda }_1{\mathcal{L}}_{\mathrm{color}}+{\lambda }_2{\mathcal{L}}_{\mathrm{depth}}+{\lambda }_3{\mathcal{L}}_{\mathrm{sem}}+{\lambda }_4{\mathcal{L}}_{\mathrm{sky}}+{\lambda }_5{\mathcal{L}}_{\mathrm{accum}},$$

where ${\mathcal{L}}_{\mathrm{depth}}$ is from Depth-supervised NeRF and ${\mathcal{L}}_{\mathrm{sem}}$ is from SemanticNeRF.

4. Experiment

1. Photorealistic Rendering

   Dataset: KITTI+VKITTI
   

2. Instance-wise Editing

3. The blessing of modular design

4. Ablation Results

   Unlike prior works [26SUDS, 15 Panoptic Neural Fields, 21NSG] that evaluate their method on a short sequence of 90 images, we use the full sequence from the dataset for all evaluations.