3DILG: Irregular Latent Grids for 3D Generative Modeling



We propose a new representation for encoding 3D shapes as neural fields. The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation. Existing works on neural fields are grid-based representations with latents defined on a regular grid. In contrast, we define latents on irregular grids, enabling our representation to be sparse and adaptive. In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid-based methods in terms of reconstruction accuracy. For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models. We show different applications that improve over the current state of the art. First, we show results for probabilistic shape reconstruction from a single higher resolution image. Second, we train a probabilistic model conditioned on very low resolution images. Third, we apply our model to category-conditioned generation. All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.

Shape reconstruction

Shape Reconstruction

Unidirectional Autoregressive Transformer

Category-conditioned generation


Low-resolution image conditioned generation


Mask conditioned generation


        title={3{DILG}: Irregular Latent Grids for 3D Generative Modeling},
        author={Biao Zhang and Matthias Nie{\ss}ner and Peter Wonka},
        booktitle={Thirty-Sixth Conference on Neural Information Processing Systems},