One crucial limitation in this work is the large amount of compute and memory necessary to create, store, and process SRFs, especially at high-resolution voxel grids. This creates a bottleneck in training, developing, and building on SRFs.Developing efficient methods to work and learn from sparse voxel grids would be a viable plan moving forward in order to develop deep and large models in this space, as well as, leveraging popular (slow) generative models (diffusion models) on 3D radiance fields.


Detailed Formulations
Sparse Convolutions
Sparse convolutions are a variant of standard convolutions that are used in deep learning. In a sparse convolution, only a subset of the input elements is used in the computation, which allows for more efficient use of computation resources and can improve the performance of the convolutional neural network.
To perform a sparse convolution, we first define a set of indices that specify which input elements should be used in the convolution. This set of indices is called the "support" of the convolution. We then use these indices to select the relevant input elements and compute the convolution using these elements. This is typically done by applying a filter to the selected input elements and summing the results to produce the output of the convolution.
In the simplest 1 D case, let \(\mathbf {x}\)  be the input tensor, \(\mathbf {w}\)  be the convolutional filter, and \(c\)  be the support of the convolution (i.e. the set of indices specifying which elements of \(x\)  should be used in the convolution). The output of the sparse convolution, \(\mathbf {y}\) , can be computed as: \(\mathbf {y} = \mathbf {x}[c] * \mathbf {w}\) 
where \(*\)  denotes the convolution operation, and \(\mathbf {x}[c]\)  is the subset of elements from \(\mathbf {x}\)  specified by the support \(c\) . This equation applies the convolutional filter \(\mathbf {w}\)  to the selected input elements and sums the results to produce the output of the convolution. For more detailed formulation and implementation of the Sparse convolutions we used in our work, please refer to MinkowskiNetwork [1]}.

Detailed Setup
SPARF Dataset
All the rendered images are of \(400\times 400\)  resolution with 4 channels (RGB + alpha channel for background). SPARF has three main splits for every 3D shape: training views (400 views), test views (20 views), and an OOD “hard" views (10 views) as can be shown in Figure REF . Regarding the collected SRFs, Plenoxels [2]} is used as the base module. The spherical harmonics dimension of the whole SRFs is \(d=4\times 3= 12\) , while for partial SRFs, it is \(d=1\times 3 = 3\) . Most of the hyperparameters used in optimizing the SRFs are the default ones proposed in the Plenoxels paper [2]} (as can be seen in the attached code under Svox2/opt/opt-py). However, the following hyperparameters were engineered in order to scale up the optimization and maintain the quality of the SRFs ( as can be seen in Figure REF , and REF ). Running Plenoxels [2]} for fewer iterations (3\(\times \) 12K) reduces the time by 30% while maintaining the same PSNR. Using 400 views/shapes in SPARF to optimize the SRFs keep the time manageable in optimization (\(\sim 4\)  minutes for the 512 resolution) while maintaining high PSNR (\(\sim 30\) dB). When saving the SRFs, we only save the set of coordinates (integers) and float features (densities and radiance components). The upsampling iteration of Plenoxels is set to 1\(\times \) 12K for faster convergence. The distribution of the collected dataset is detailed in Table REF . More examples of the whole partial SRFs collected in SPARF can be found in Figure REF . The whole SRFs are easily convertible to high-quality meshes using Marching Cubes [5]} as shown in Figure REF .
<TABLE><FIGURE><FIGURE>
SuRFNet Training
We use a voxel resolution of \(128^3\)  of the SPARF dataset in most of the learning experiments and visualizations in this work, unless otherwise clearly stated. This choice is to reduce the computational cost of training the heavy pipeline and to facilitate the development of proper learning methods on SRFs. The input SRF is normalized with a fixed value of 10,0000 for the density and 10 for the colors, to ensure the distribution lies within -1 to 1. The Q-Gaussian std \(\sigma \)  is set to 0.444 (studied more in Section ). The strides for the SuRFNEt are all set 2, while the network depth \(l=3\)  modules. The batch size for training is 14 when A100 GPUs are used and 6 when V100 GPUs are used. The training saturates at 100 epochs. The optimizer used is AdamW [6]} with a learning rate of 0.01, a momentum of 0.9, a weight decay of \(1e-5\) , and a learning rate exponential decay rate of 0.99. The hyperparameters \(\lambda _{R}, \lambda _{\alpha }, \lambda _{\rho }\)  are all set independently to each class, where a different network is trained on each class separately. Most classes have \(\lambda _{\alpha } = 30.0 ,\lambda _{\rho }=1.0, \lambda _{R} \) . We did not prune the output sparse voxel as this leads to harming performance most of the time and increase the problem of vanishing gradients. The background color of the rendered images \(\mathcal {R}_{\phi } \left( \mathbf {F}(\mathcal {X})\right)\)  is masked out from the perceptual loss and the density component \(\alpha \)  of the output SRF is also not affected by the perceptual loss, as this can cause excessive densities around the object, leading to deteriorating the SRF output perceptuality. During training with the perceptual loss, three randomly selected images from three different \(\phi \)  as used as labels for the three rendered images from the output \(\mathcal {R}_{\phi } \left( \mathbf {F}(\mathcal {X})\right)\) . The SuRFNet is jointly predicting the density and radiance of spherical harmonics colors, but with different heads. More setup details can be found in the attached code and analyzed further in Section .
For a fair comparison to the baselines PixelNeRF and VisionNerf ( which use \(64 \times 64\)  resolution), we upsample their resolution at inference at test poses while using their pretrained weights of the NMR dataset. The upsampling is using the bicubic sampling of the Pytorch Transforms library. Retraining the methods from scratch on the high resolution \(400 \times 400\)  is computationally prohibitive. We train a separate model for each class, to maintain high-quality generation of 3D SRFs.

Additional Analysis
<TABLE>Shiny Objects
Some of the rendered objects have reflective materials, resulting in distorted optimized radiance fields for these shapes despite using all of the views. We separate these distorted SRFs (only 76 shapes in total) from the SPARF dataset (see Figure REF ).
<FIGURE>
Effect of Dataset Size
We study the effect of increasing the dataset size (Whole SRFs and Partial SRFs) on the generalization performance of SuRFNet in Figure REF ,REF . It shows that as the dataset size increase ( normalized the number of shapes in each class ), the generalization performance increase. This scalability effect underlines the importance of SPARF. However, as can be seen from these two figures, partial SRFs scalability is more important than increasing whole SRFs, which justifies collecting 4 variants per resolution ( as detailed in Table REF ).
<FIGURE><FIGURE>
Loss Ablation Study
For the density threshold \(\alpha _{\text{dense}}\)  defined in Eq 2 and 3, the validation accuracies of SuRFNet on car class are 13, 14.7, 67.8, 67, 67, 67.2, 62.5 for the values of -0.01, -0.001, 0, 0.001, 0.003, 0.01, 0.03 of \(\alpha _{\text{dense}}\)  respectively. The hyperparameter \(\sigma \)  which governs the spread of the Q-Gaussian loss is studied as follows. the validation accuracies of SuRFNet on airplane class are 52, 70.4, 71.7, 72.1, 72.2, 72.4, 72.3 for the values of 0.1, 0.2, 0.3, 0.4, 0.6, 0.8, and 1.0 of \(\sigma \)  respectively. The number of coordinates \(\mathbf {c}\)  sampled in the Q-Gaussian loss is proportional to the number of coordinates in the input SRFs with multiplier \(K=40\) . For different values of this multiplier 1, 5, 10, 20, 40, 80, 200, the validation accuracies of SuRFNet trained on airplane class are 72.2, 72.7, 72.9, 72.5, 71.8, 71.7, and 71.6 respectively.

Faulty Textures
In some rare instance of the shapes in ShapeNet [7]}, some objects have doubled textures in some areas. This occurs in less than 1% of the data and leads the renderer to render the background instead in these areas (highlighted with green). See Figure REF  for examples of these cases.
<FIGURE><FIGURE>
Additional Results
Additional results of normal test tracks benchmark of SPARF are presented in Table REF . Please see figures REF  and REF  for differences between the normal train/test track and the OOD hard track. More comparisons and generations are provided in Figures REF ,REF ,REF .
<FIGURE><FIGURE><FIGURE><FIGURE><FIGURE><FIGURE><FIGURE><FIGURE><FIGURE>