Neural field

Last updated

In machine learning, a neural field (also known as implicit neural representation, neural implicit, or coordinate-based neural network), is a mathematical field that is fully or partially parametrized by a neural network. Initially developed to tackle visual computing tasks, such as rendering or reconstruction (e.g., neural radiance fields), neural fields emerged as a promising strategy to deal with a wider range of problems, including surrogate modelling of partial differential equations, such as in physics-informed neural networks. [1]

Contents

Differently from traditional machine learning algorithms, such as feed-forward neural networks, convolutional neural networks, or transformers, neural fields do not work with discrete data (e.g. sequences, images, tokens), but map continuous inputs (e.g., spatial coordinates, time) to continuous outputs (i.e., scalars, vectors, etc.). This makes neural fields not only discretization independent, but also easily differentiable. Moreover, dealing with continuous data allows for a significant reduction in space complexity, which translates to a much more lightweight network. [1]

Formulation and training

According to the universal approximation theorem, provided adequate learning, sufficient number of hidden units, and the presence of a deterministic relationship between the input and the output, a neural network can approximate any function to any degree of accuracy. [2]

Hence, in mathematical terms, given a field , with and , a neural field , with parameters , is such that: [1]

Training

For supervised tasks, given examples in the training dataset (i.e., ), the neural field parameters can be learned by minimizing a loss function (e.g., mean squared error). The parameters that satisfy the optimization problem are found as: [1] [3] [4] Notably, it is not necessary to know the analytical expression of , for the previously reported training procedure only requires input-output pairs. Indeed, a neural field is able to offer a continuous and differentiable surrogate of the true field, even from purely experimental data. [1]

Moreover, neural fields can be used in unsupervised settings, with training objectives that depend on the specific task. For example, physics-informed neural networks may be trained on just the residual. [4]

Spectral bias

As for any artificial neural network, neural fields may be characterized by a spectral bias (i.e., the tendency to preferably learn the low frequency content of a field), possibly leading to a poor representation of the ground truth. [5] In order to overcome this limitation, several strategies have been developed. For example, SIREN uses sinusoidal activations, [6] while the Fourier-features approach embeds the input through sines and cosines. [7]

Conditional neural fields

In many real-world cases, however, learning a single field is not enough. For example, when reconstructing 3D vehicle shapes from Lidar data, it is desirable to have a machine learning model that can work with arbitrary shapes (e.g., a car, a bicycle, a truck, etc.). The solution is to include additional parameters, the latent variables (or latent code) , to vary the field and adapt it to diverse tasks. [1]

Latent code production

Conditional neural field with encoder Conditional neural field with encoder.jpg
Conditional neural field with encoder
Auto-decoding conditional neural field Conditional neural field auto-decoder 01.jpg
Auto-decoding conditional neural field

When dealing with conditional neural fields, the first design choice is represented by the way in which the latent code is produced. Specifically, two main strategies can be identified: [1]

Since the latter strategy requires additional optimization steps at inference time, it sacrifices speed, but keeps the overall model smaller. Moreover, despite being simpler to implement, an encoder may harm the generalization capabilities of the model. [1] For example, when dealing with a physical scalar field (e.g., the pressure of a 2D fluid), an auto-decoder-based conditional neural field can map a single point to the corresponding value of the field, following a learned latent code . [10] However, if the latent variables were produced by an encoder, it would require access to the entire set of points and corresponding values (e.g. as a regular grid or a mesh graph), leading to a less robust model. [1]

Global and local conditioning

In a neural field with global conditioning, the latent code does not depend on the input and, hence, it offers a global representation (e.g., the overall shape of a vehicle). However, depending on the task, it may be more useful to divide the domain of in several subdomains, and learn different latent codes for each of them (e.g., splitting a large and complex scene in sub-scenes for a more efficient rendering). This is called local conditioning. [1]

Conditioning strategies

There are several strategies to include the conditioning information in the neural field. In the general mathematical framework, conditioning the neural field with the latent variables is equivalent to mapping them to a subset of the neural field parameters: [1] In practice, notable strategies are:

Meta-learning

Instead of relying on the latent code to adapt the neural field to a specific task, it is also possible to exploit gradient-based meta-learning. In this case, the neural field is seen as the specialization of an underlying meta-neural-field, whose parameters are modified to fit the specific task, through a few steps of gradient descent. [13] [14] An extension of this meta-learning framework is the CAVIA algorithm, that splits the trainable parameters in context-specific and shared groups, improving parallelization and interpretability, while reducing meta-overfitting. This strategy is similar to the auto-decoding conditional neural field, but the training procedure is substantially different. [15]

Applications

Thanks to the possibility of efficiently modelling diverse mathematical fields with neural networks, neural fields have been applied to a wide range of problems:

Neural radiance field (NeRF) pipeline Neural radiance fields nerf pipeline.jpg
Neural radiance field (NeRF) pipeline
CORAL architecture: operator learning with neural fields CORAL neural field architecture inference scheme.png
CORAL architecture: operator learning with neural fields

See also

References

  1. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Xie, Yiheng; Takikawa, Towaki; Saito, Shunsuke; Litany, Or; Yan, Shiqin; Khan, Numair; Tombari, Federico; Tompkin, James; Sitzmann, Vincent; Sridhar, Srinath (2022). "Neural Fields in Visual Computing and Beyond" . Computer Graphics Forum. 41 (2): 641–676. doi:10.1111/cgf.14505. ISSN   1467-8659.
  2. Hornik, Kurt; Stinchcombe, Maxwell; White, Halbert (1989-01-01). "Multilayer feedforward networks are universal approximators" . Neural Networks. 2 (5): 359–366. doi:10.1016/0893-6080(89)90020-8. ISSN   0893-6080.
  3. Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). Deep learning. Adaptive computation and machine learning. Cambridge, Mass: The MIT press. ISBN   978-0-262-03561-3.
  4. 1 2 3 Quarteroni, Alfio; Gervasio, Paola; Regazzoni, Francesco (2025). "Combining physics-based and data-driven models: Advancing the frontiers of research with scientific machine learning". Mathematical Models and Methods in Applied Sciences. 35 (4): 905–1071. arXiv: 2501.18708 . doi:10.1142/S0218202525500125.
  5. Rahaman, Nasim; Baratin, Aristide; Arpit, Devansh; Draxler, Felix; Lin, Min; Hamprecht, Fred A.; Bengio, Yoshua; Courville, Aaron (2018). "On the Spectral Bias of Neural Networks". arXiv: 1806.08734 [stat.ML].
  6. Sitzmann, Vincent; Martel, Julien N. P.; Bergman, Alexander W.; Lindell, David B.; Wetzstein, Gordon (2020). "Implicit Neural Representations with Periodic Activation Functions". arXiv: 2006.09661 [cs.CV].
  7. Tancik, Matthew; Srinivasan, Pratul P.; Mildenhall, Ben; Fridovich-Keil, Sara; Raghavan, Nithin; Singhal, Utkarsh; Ramamoorthi, Ravi; Barron, Jonathan T.; Ng, Ren (2020). "Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains". arXiv: 2006.10739 [cs.CV].
  8. Qi, Charles R.; Su, Hao; Mo, Kaichun; Guibas, Leonidas J. (2016). "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation". arXiv: 1612.00593 [cs.CV].
  9. 1 2 Jeong Joon Park; Florence, Peter; Straub, Julian; Newcombe, Richard; Lovegrove, Steven (2019). "DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation". arXiv: 1901.05103 [cs.CV].
  10. 1 2 Serrano, Louis; Lise Le Boudec; Armand Kassaï Koupaï; Thomas X Wang; Yin, Yuan; Vittaut, Jean-Noël; Gallinari, Patrick (2023). "Operator Learning with Neural Fields: Tackling PDEs on General Geometries". arXiv: 2306.07266 [cs.LG].
  11. Ha, David; Dai, Andrew; Le, Quoc V. (2016). "HyperNetworks". arXiv: 1609.09106 [cs.LG].
  12. Perez, Ethan; Strub, Florian; Harm de Vries; Dumoulin, Vincent; Courville, Aaron (2017). "FiLM: Visual Reasoning with a General Conditioning Layer". arXiv: 1709.07871 [cs.CV].
  13. Finn, Chelsea; Abbeel, Pieter; Levine, Sergey (2017). "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks". arXiv: 1703.03400 [cs.LG].
  14. Sitzmann, Vincent; Chan, Eric R.; Tucker, Richard; Snavely, Noah; Wetzstein, Gordon (2020). "MetaSDF: Meta-learning Signed Distance Functions". arXiv: 2006.09662 [cs.CV].
  15. Luisa M Zintgraf; Shiarlis, Kyriacos; Kurin, Vitaly; Hofmann, Katja; Whiteson, Shimon (2018). "Fast Context Adaptation via Meta-Learning". arXiv: 1810.03642 [cs.LG].
  16. Mescheder, Lars; Oechsle, Michael; Niemeyer, Michael; Nowozin, Sebastian; Geiger, Andreas (2018). "Occupancy Networks: Learning 3D Reconstruction in Function Space". arXiv: 1812.03828 [cs.CV].
  17. Mildenhall, Ben; Srinivasan, Pratul P.; Tancik, Matthew; Barron, Jonathan T.; Ramamoorthi, Ravi; Ng, Ren (2020). "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis". arXiv: 2003.08934 [cs.CV].
  18. Raissi, M.; Perdikaris, P.; Karniadakis, G. E. (2019-02-01). "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations". Journal of Computational Physics. 378: 686–707. Bibcode:2019JCoPh.378..686R. doi:10.1016/j.jcp.2018.10.045. ISSN   0021-9991.
  19. Yin, Yuan; Kirchmeyer, Matthieu; Franceschi, Jean-Yves; Rakotomamonjy, Alain; Gallinari, Patrick (2022). "Continuous PDE Dynamics Forecasting with Implicit Neural Representations". arXiv: 2209.14855 [cs.LG].