Giacomo Cristinelli

Giacomo Cristinelli

PhD candidate in applied math

A Taxonomy of Deep Learning Methods in Geometry Processing

Research note

A practical framework for reading deep-learning methods in geometry processing through four questions: what is predicted, how geometry is represented, which architecture is used, and how the model is trained.

Geometry processing · Deep learning · Neural fields

01

Prediction task

Analysis or synthesis
02

Representation

Points, meshes, images, grids, or fields
03

Architecture

MLPs, convolutions, or attention
04

Training objective

Data, priors, and optimization constraints

Most of the material is adapted from this talk.

1. Introduction

The characterization of deep learning methods in geometry processing can be organized around the following four fundamental questions:

  1. What prediction problem are we aiming to solve?
  2. How is the data/geometry represented?
  3. Which neural network architecture is employed?
  4. How is the network trained?

The central idea is to first identify the function we seek to approximate. Next, we analyze the structure of the underlying geometric objects and determine an appropriate representation. Finally, we select a suitable neural network architecture and specify the corresponding training and optimization strategy.

2. Prediction problems

Prediction problems in geometric processing can be broadly divided into two categories: analysis and synthesis.

In analysis tasks, the goal is to approximate a function that takes a geometric object as input and produces an analytical output (e.g., a scalar, vector field, or probability distribution). In synthesis tasks, the objective is to generate new geometric objects, possibly conditioned on geometric or non-geometric inputs.

2.1 Analysis tasks

Analysis problems can be further subdivided into global and local analysis.

Global analysis produces a quantity that characterizes the geometry of the object as a whole (typically a scalar, vector, or low-dimensional descriptor). Local analysis, in contrast, produces pointwise or region-wise quantities defined on the shape itself, such as scalar or vector fields.

Global analysis problems. Examples include:

  • Classification: Given a finite vocabulary of shape classes, the task is to approximate a function that maps a geometric object to a probability distribution over the set of classes.
  • Canonicalization: The goal is to learn a function that maps a shape to a canonical representation, for instance by predicting a rigid transformation (e.g., a rotation matrix) that aligns the shape with a standardized orientation.
  • Captioning: The task consists of mapping a geometric object to a textual description, typically represented as a sequence of tokens.

Local analysis problems. Examples include:

  • Segmentation: The objective is to assign a label or scalar value to each point (or element) of the shape, such that the induced partition decomposes the geometry into meaningful and coherent parts.
  • Normal estimation: The function to be approximated returns a vector field representing the surface normals at each point of the shape.
  • Geodesic distance estimation: Given a shape and a source point, the goal is to predict a scalar function representing the geodesic distance from that point. Although this problem admits analytical solutions, learning-based approaches are often employed to accelerate computation.
  • Shape correspondence: Given two shapes, the objective is to compute functions (or feature fields) on each shape such that points with similar feature values are in structural correspondence. Typically, learned feature embeddings are compared via linear operations (e.g., inner products) to establish correspondences.

2.2 Synthesis tasks

Synthesis tasks can be further subdivided into unconditional and conditional synthesis. The distinction depends on whether shapes are generated by sampling from an unconditional distribution over geometric objects, or from a distribution conditioned on additional input data.

In unconditional synthesis, the objective is to model and sample from a probability distribution \(p(\mathcal{S})\) over shapes \(\mathcal{S}\). In conditional synthesis, one instead models a conditional distribution \(p(\mathcal{S} \mid c)\), where \(c\) denotes auxiliary information such as partial geometry, semantic labels, images, text descriptions, or other constraints.

Conditional synthesis problems can further be characterized as low-entropy or high-entropy, depending on the amount of variability in the conditional distribution. Low-entropy problems correspond to settings in which the conditioning variable strongly constrains the output, leading to a narrow distribution of admissible shapes. High-entropy problems arise when the conditioning signal leaves substantial ambiguity, resulting in a broad and diverse set of plausible outputs.

Unconditional synthesis. Representative approaches include:

  • IM-NET / Implicit Field Decoders [1]: These models generate shapes via continuous implicit representations, typically by learning a neural function that approximates a signed distance function or occupancy field. Sampling in the latent space induces smooth deformations of the reconstructed geometry, enabling continuous shape interpolation.
  • Wavelet-domain diffusion [2]: This approach applies diffusion-based generative modeling in a wavelet domain corresponding to truncated signed distance functions. The multiscale structure of the wavelet representation improves stability and enables efficient synthesis of high-resolution geometry.
  • MeshGPT [3]: This method leverages autoregressive transformers to generate meshes sequentially, predicting vertices and faces token by token. The mesh is treated as a structured discrete sequence, allowing the model to learn geometric and combinatorial dependencies.

Conditional synthesis. Examples, ordered from relatively low-entropy to high-entropy settings, include:

  • Point-cloud conditioning: The objective is to reconstruct or complete a surface from a (possibly sparse or noisy) point cloud. In relatively low-entropy settings, the point cloud strongly constrains the geometry, and the task reduces to learning a conditional distribution concentrated around plausible surface completions. For example, [4] generates shapes conditioned on both a point cloud and a semantic class label. Alternatively, [5] focuses on local geometric structure, learning surface reconstruction operators from local neighborhoods of the point cloud.
  • Image conditioning: The goal is to infer a 3D shape from one or multiple 2D images. In seminal work such as [6], a model is trained on a large collection of aligned image–shape pairs to learn the structural priors of a specific object category; given a new image, the corresponding 3D geometry is inferred. This paradigm has been extended to multi-view and multi-category settings, for example via view-conditioned 2D diffusion models as in [7]. Compared to point-cloud conditioning, the ambiguity is higher due to occlusions and missing depth information.
  • Text conditioning: The objective is to generate a 3D shape from a textual description. Early approaches include [8], while more recent methods such as [9] leverage large-scale text–shape alignment and diffusion-based generative models. This setting is typically high-entropy, since textual descriptions underdetermine geometric detail and admit a wide range of plausible realizations.

3. Data representation

These are the main types of geometric data representations. There is no single canonical representation, as each serves different purposes depending on the application and the available data.

  • Point clouds: Point clouds consist of sets of points sampled from the surface of an object. They are relatively easy to acquire from sensors, such as LiDAR or depth cameras, and therefore arise frequently in real-world data. For this reason, it is important to develop neural network methods that can directly process point clouds. They are widely used in computer vision and robotics, where autonomous systems must interpret their surroundings from sensor measurements. In many applications, one also seeks to convert point clouds into richer representations, such as meshes, implicit surfaces, or CAD models (see point-cloud conditional synthesis).
  • Triangle meshes: Triangle meshes are one of the most common representations in computer graphics. They explicitly represent the surface geometry of a shape through vertices, edges, and triangular faces. This structure enables intrinsic geometric analysis using tools from discrete differential geometry.
  • Images / views: A geometric object can also be represented through a collection of two-dimensional images captured from different viewpoints. In this representation, one can leverage the extensive machinery developed in image processing and computer vision.
  • 3D grids / voxels: In a voxel representation, space is discretized into a three-dimensional grid. Each voxel stores information about the presence of a surface (e.g., occupancy) or a scalar value such as a discretized signed distance function. Learning methods developed for 2D images can often be extended naturally to this grid-based representation.
  • Implicit representations: In implicit representations, a surface is described as the level set of a function (for example, a signed distance function or occupancy field). Neural networks can be used to approximate this function, providing a continuous representation of geometry.
  • Procedural / CAD representations: CAD or procedural models describe geometry through parametric primitives, constructive operations, or procedural rules. These representations provide effectively infinite resolution and are particularly natural in engineering and manufacturing contexts.

4. Neural architectures

In very simplified terms, a neural network can be viewed as a composition of functions (or layers)

\[f(x) = g(h(j(x))),\]

some of which depend on parameters (also called weights). More generally, we write

\[f_{\theta}(x) = g_{\theta_1}(h(j_{\theta_2}(x))),\]

where \(\theta = (\theta_1,\theta_2)\) denotes the collection of all learnable parameters.

The functions in this composition that depend on parameters are called learnable layers. A fundamental property of such constructions is that gradients with respect to the parameters are well defined, which allows the model to be trained using gradient-based optimization:

\[\nabla_\theta f_\theta(x) = \nabla_{\{\theta_1,\theta_2\}} g_{\theta_1}\!\left(h(j_{\theta_2}(x))\right).\]

There are three primary types of learnable layers.

1. Fully connected (linear or affine) layers. These layers take the form

\[y = Wx + b,\]

where \(x\) is the input vector, \(W\) is a weight matrix, and \(b\) is a bias vector. Each output component depends linearly on all components of the input. When several such affine layers are composed with nonlinear activation functions between them, the resulting architecture is known as a multilayer perceptron (MLP).

2. Convolutional layers. These layers aggregate information from local neighborhoods:

\[y_i = \sum_{x_j \in \mathcal{N}(x_i)} f_\theta(x_i, x_j),\]

where \(\mathcal{N}(x_i)\) denotes the neighborhood of \(x_i\). In many practical implementations, \(f_\theta\) corresponds to a weighted combination of the neighboring features, for instance

\[f_\theta(x_i,x_j) = w_\theta(x_i,x_j)\,x_j.\]

This local aggregation mechanism is particularly effective when the data possess spatial or geometric structure.

3. Attention layers (Transformers). Attention mechanisms compute interactions between all pairs of elements:

\[y_i = \sum_j (Qx_i \cdot Kx_j)\, Vx_j.\]

Each input element \(x_i\) is linearly transformed into three vectors: a query \(Qx_i\), a key \(Kx_i\), and a value \(Vx_i\), where the matrices \(Q,K,V\) are learnable parameters. The output \(y_i\) is then obtained as a weighted combination of all value vectors, where the weights

\[w_{ij} = Qx_i \cdot Kx_j\]

measure the affinity between elements \(x_i\) and \(x_j\) in the learned feature space. Intuitively, these weights quantify how much attention element \(i\) pays to element \(j\).

Geometric data representations and network architectures are tightly coupled. Once a representation is chosen, the architecture must be adapted accordingly. For example, image representations naturally benefit from convolutional neural networks developed in computer vision. In geometry processing, the most prominent architectures are designed for point clouds, meshes, or implicit representations.

4.1 Point cloud architectures

One of the earliest and most influential architectures for point clouds is PointNet [10]. The input consists of \(n\) points. After an initial transformation, each point is processed independently by a multilayer perceptron. The resulting feature vectors undergo a second feature transformation and are processed by another MLP. A global descriptor is then computed using a symmetric aggregation operator (typically a maximum) across all point features. Finally, a third MLP maps this global feature vector to a classification score.

The key idea is to process each point locally in a high-dimensional feature space and then combine these features using an order-invariant aggregation function, like max, min, average, etc. Order invariance is crucial because point clouds are unordered sets.

Since the introduction of PointNet, many improvements have been proposed. For example, Dynamic Graph CNN (DGCNN) [11] introduces convolution-like operations on dynamically constructed graphs of neighboring points. More recently, the Point Transformer [12] incorporates attention mechanisms to model interactions between points.

4.2 Mesh architectures

For triangular meshes, one of the most influential works is MeshCNN [13], which defines convolutional neural networks directly on mesh structures. The key idea is to define the convolution operator on edges rather than faces or vertices. In a manifold triangular mesh without boundaries, every edge \(e\) has exactly four neighboring edges \(a,b,c,d\) (two from each adjacent triangle), which provides a consistent local neighborhood.

The winding order of the triangles is used to define an ordering of the neighboring edges. Each edge is associated with geometric features describing the local shape, including:

  • the dihedral angle between the two faces incident to the edge,
  • the two interior angles opposite the edge in the adjacent triangles,
  • the ratios between the edge length and the corresponding triangle heights.

The convolution operator is then defined as a weighted combination of edge feature vectors,

\[w_0 f_e + \sum_{i=1}^4 w_i \hat{f}_i,\]

where the transformed neighborhood features are given by

\[(\hat{f}_1,\hat{f}_2,\hat{f}_3,\hat{f}_4) = \bigl(\lvert f_a-f_c\rvert,\; f_a+f_c,\; \lvert f_b-f_d\rvert,\; f_b+f_d\bigr).\]

These transformed features are designed so that the convolution is invariant to the ordering of the two incident faces, using symmetric (order-invariant) operations.

Similarly to how PointNet aggregates local point features into global descriptors, MeshCNN progressively aggregates local mesh features through a pooling operation based on edge collapses. This reduces the mesh resolution while propagating feature information to coarser scales. The pooling can also be approximately inverted by reversing the sequence of edge collapses during the decoding stage.

Other noteworthy mesh-based architectures include:

  • DiffusionNet [14], which replaces discrete convolution operations with continuous diffusion processes on the surface (solutions of the heat equation). The diffusion operator propagates features smoothly across the surface and is largely discretization-agnostic, meaning that it generalizes well across meshes with different tessellations.
  • MeshGPT [3], which adopts a transformer-based architecture to generate meshes autoregressively as sequences of vertices and faces.

An insightful survey of deep learning on meshes is contained in [15].

4.3 Implicit neural representations (INRs)

Implicit neural representations are based on the classical description of surfaces via implicit functions. A surface is represented as the zero level set of a function \(f:\mathbb{R}^3 \to \mathbb{R}\), i.e.,

\[\mathcal{S} = \{ x \in \mathbb{R}^3 \mid f(x) = 0 \}.\]

Such representations date back to early work in computer graphics, e.g. [16], and are closely related to the framework of constructive solid geometry (CSG).

In practice, one typically considers well-behaved functions that are negative inside the object and positive outside. This sign structure makes it straightforward to determine whether a point lies inside or outside the surface. Moreover, Boolean operations between shapes—such as unions, intersections, and differences—can be expressed using pointwise \(\min\) and \(\max\) operations, which form the basis of CSG.

Several choices of implicit functions are possible, including indicator functions and signed distance functions (SDFs), where \(\lvert f(x)\rvert\) corresponds to the distance from \(x\) to the surface. SDFs are particularly attractive due to their geometric structure. For instance, if one moves from a point \(x_1\) away from the surface along the normal direction to a point \(x_2\), the function satisfies

\[f(x_2) = f(x_1) + \lVert x_2 - x_1\rVert,\]

which is equivalent to the Eikonal equation

\[\lvert\nabla f\rvert = 1.\]

SDFs encode additional geometric information: the gradient \(\nabla f\) yields the surface normal, while the Laplacian \(\Delta f\) is related to mean curvature.

Despite these advantages, SDFs are difficult to maintain under operations such as CSG, where the distance property is generally lost. To address this issue, conservative signed distance functions (CSDFs) are often used. These functions satisfy

\[\lvert f(x)\rvert \leq \mathrm{dist}(x, \mathcal{S}),\]

and preserve this inequality under common operations. They are particularly useful in rendering, where sphere tracing [17] provides an efficient ray-marching algorithm: starting from a point \(x\) and a direction \(v\), one iteratively advances along the ray by steps of size \(\lvert f(x)\rvert\). This procedure converges to the surface, with faster convergence when \(f\) is a true SDF.

Another important task is surface extraction, where one aims to recover a mesh representation from the implicit function. A classical method is marching cubes [18], which operates on a discretized grid and reconstructs local surface patches based on function values at the grid vertices. The quality of the resulting mesh depends on the grid resolution. More recent methods improve upon this by exploiting the structure of SDFs, either through geometric techniques [19] or learning-based approaches such as neural dual contouring and neural marching cubes [20, 21].

In practice, implicit representations are often discretized on grids (voxels), since analytical expressions are typically unavailable. However, such grid-based representations suffer from poor scalability with respect to resolution and dimension. To mitigate this issue, adaptive discretizations—such as octrees or multiresolution grids—are commonly used, allocating higher resolution near the surface and coarser resolution elsewhere.

Neural implicit representations provide an alternative by modeling the implicit function directly as a neural network. In this setting, one considers a parametric function \(f_\theta:\mathbb{R}^n \to \mathbb{R}\), typically implemented as a multilayer perceptron (MLP),

\[f_\theta(x) = A^N_\theta \,\varphi\bigl( A^{N-1}_\theta \,\varphi(\cdots \varphi(A^0_\theta x + b^0_\theta)\cdots) + b^{N-1}_\theta \bigr) + b^N_\theta,\]

where \(\varphi\) denotes a nonlinear activation function. Early and influential works include [22, 23, 1, 24].

The general learning paradigm consists of fitting the network to sampled data \((X, V)\), where \(X \subset \mathbb{R}^n\) are spatial points and \(V\) are target values (e.g. occupancy or signed distance values), by solving

\[\min_\theta \sum_{x \in X} \bigl(f_\theta(x) - v(x)\bigr)^2,\]

typically via stochastic gradient descent.

Neural implicit representations offer several advantages. First, by the universal approximation theorem, sufficiently wide MLPs can approximate any continuous function, providing high expressive power. Second, the parametric nature of the model allows one to encode geometric properties through suitable loss functions, enabling flexible adaptation to specific tasks. Third, neural fields are differentiable with respect to both inputs and parameters, which makes them well suited for downstream optimization tasks, such as topology optimization [25].

However, these representations also have limitations. Geometric properties are usually enforced only through soft constraints in the loss function, which can lead to inaccuracies, such as the loss of sharp features. Moreover, since the geometry is encoded implicitly in the network parameters, performing downstream operations (e.g. editing or local modifications) can be unintuitive and computationally challenging.

Addressing the representation of high-frequency details has been an active area of research. For example, [26] introduces Fourier feature mappings that transform input coordinates into a higher-dimensional space, enabling the network to capture fine-scale variations. Similarly, [27] proposes using sinusoidal activation functions instead of standard ReLU activations, which significantly improves the representation of high-frequency geometry.

Another important direction is the development of hybrid discrete/continuous representations [4, 28], which combine voxel-based structures with neural networks. These methods leverage discrete spatial encodings together with continuous function approximation, leading to substantial improvements in efficiency and scalability.

The sampling strategy for training data also plays a crucial role [29]. In particular, sampling points densely near the surface is essential for accurately capturing geometric detail.

If the Eikonal constraint \(\lvert\nabla f\rvert = 1\) is enforced during training [30], the learned function approximates a signed distance function, enabling the use of geometric properties such as normals and distances, as well as more robust CSG operations. Recent work [31] introduces additional losses, such as closest-point constraints, to better preserve SDF behavior under such operations.

Finally, recent advances have addressed the reliability of geometric queries on neural implicit surfaces. For instance, [32] employs range analysis techniques (e.g. affine arithmetic) to provide guarantees for operations such as ray intersection and closest-point queries, even for general neural implicit functions beyond SDFs.

References

[1] Chen, Z. & Zhang, H. (2019). Learning Implicit Fields for Generative Shape Modeling. CVPR.

[2] Hui, K.-H., Li, R., Hu, J. & Fu, C.-W. (2022). Neural Wavelet-domain Diffusion for 3D Shape Generation. SIGGRAPH Asia.

[3] Siddiqui, Y. et al. (2024). MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers. CVPR.

[4] Peng, S. et al. (2020). Convolutional Occupancy Networks. ECCV.

[5] Liu, S.-L. et al. (2021). Deep Implicit Moving Least-Squares Functions for 3D Reconstruction. CVPR.

[6] Sun, X. et al. (2018). Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling. CVPR.

[7] Liu, M. et al. (2024). One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. NeurIPS.

[8] Chen, K. et al. (2018). Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings. arXiv:1803.08495.

[9] Jun, H. & Nichol, A. (2023). Shap-E: Generating Conditional 3D Implicit Functions. arXiv:2305.02463.

[10] Qi, C. R. et al. (2016). PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. arXiv:1612.00593.

[11] Wang, Y. et al. (2019). Dynamic Graph CNN for Learning on Point Clouds. ACM TOG.

[12] Wu, X. et al. (2024). Point Transformer V3: Simpler, Faster, Stronger. CVPR.

[13] Hanocka, R. et al. (2019). MeshCNN: A Network with an Edge. ACM TOG, 38(4).

[14] Sharp, N. et al. (2022). DiffusionNet: Discretization Agnostic Learning on Surfaces. ACM TOG, 41(3).

[15] Chen, Z. (2023). A Review of Deep Learning-Powered Mesh Reconstruction Methods. arXiv:2303.02879.

[16] Ricci, A. (1973). A Constructive Geometry for Computer Graphics. The Computer Journal, 16(2).

[17] Hart, J. C. (1996). Sphere Tracing: A Geometric Method for the Antialiased Ray Tracing of Implicit Surfaces. IEEE Visualization.

[18] Lorensen, W. E. & Cline, H. E. (1987). Marching Cubes: A High Resolution 3D Surface Construction Algorithm. ACM SIGGRAPH.

[19] Sellán, S. et al. (2023). Tangency-Aware Surface Reconstruction of SDFs. ACM TOG, 42(4).

[20] Chen, Z. et al. (2022). Neural Dual Contouring. ACM TOG, 41(4).

[21] Chen, Z. & Zhang, H. (2021). Neural Marching Cubes. ACM TOG, 40(6).

[22] Mescheder, L. et al. (2019). Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR.

[23] Park, J. J. et al. (2019). DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. CVPR.

[24] Sitzmann, V. et al. (2019). Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations. NeurIPS.

[25] Zehnder, J. et al. (2021). Topology Optimization of Elastic Objects with Contact and Friction. ACM TOG, 40(4).

[26] Tancik, M. et al. (2020). Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS.

[27] Sitzmann, V. et al. (2020). Implicit Neural Representations with Periodic Activation Functions (SIREN). NeurIPS.

[28] Müller, T. et al. (2022). Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM TOG, 41(4).

[29] Davies, T. et al. (2021). On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes. NeurIPS.

[30] Gropp, A. et al. (2020). Implicit Geometric Regularization for Learning Shapes. ICML.

[31] Marschner, Z. et al. (2023). Constructive Solid Geometry on Neural Signed Distance Fields. ACM TOG, 42(6).

[32] Sharp, N. & Jacobson, A. (2022). Spelunking the Deep: Guaranteed Queries on General Neural Implicit Surfaces via Range Analysis. ACM TOG, 41(4).