Gaussian splatting

Last updated

Gaussian Splatting is a volume rendering technique that deals with the direct rendering of volume data without converting the data into surface or line primitives. [1] The technique was originally introduced as splatting by Lee Westover in the early 1990s. [2] With advancements in computer graphics, newer methods such as 3D and 4D Gaussian splatting have been developed to offer real-time radiance field rendering and dynamic scene rendering respectively. [3] [4]

Contents

Development

Volume rendering focuses on the generation of images from discrete samples of volume data that are usually sampled in three dimensions. This data can originate from a variety of sources, such as CT scans or ozone density readings. Traditional methods transformed this volumetric data into lines and surface primitives to be displayed on computer graphics displays. However, this approach sometimes introduced artifacts and hampered interactive viewing. [1]

To address these issues, direct rendering techniques, which operate on the original volume data, were developed. These methods are classified into two primary categories: feed-backward methods and feed-forward methods. The splatting algorithm, being a feed-forward method, is directly concerned with the rendering of rectilinear volume meshes. Splatting can be customized to render volumes as either clouds or surfaces by altering the shading functions, providing flexibility in the rendering process. [1]

Since its inception, splatting has undergone various improvements. Some significant developments include textured splats, anti-aliasing, image-aligned sheet-based splatting, post-classified splatting, and the introduction of a splat primitive known as FastSplats. [2]

3D Gaussian Splatting

Recent advancements in novel-view synthesis have showcased the utility of Radiance Field methods. To enhance visual quality while ensuring real-time display rates, a new method was introduced that uses 3D Gaussians. This method integrates sparse points produced during camera calibration, representing scenes with 3D Gaussians which retain properties of continuous volumetric radiance fields. Additionally, an interleaved optimization/density control of the 3D Gaussians was introduced along with a rapid visibility-aware rendering algorithm supporting anisotropic splatting. [3] This technique has shown potential in synthesizing high-quality 3D scenes from 2D images in real-time. [5]

4D Gaussian Splatting

Extending the concept of 3D Gaussian Splatting, the 4D Gaussian Splatting incorporates a time component, allowing for dynamic scene rendering. It represents and renders dynamic scenes, with a focus on modeling complex motions while maintaining efficiency. [4] The method uses a HexPlane to connect different adjacent Gaussians, providing an accurate representation of position and shape deformations. By utilizing only a single set of canonical 3D Gaussians, and predictive analytics, the 4D Gaussian splatting method models how they move over different timestamps. [6]

Achievements of this technique include real-time rendering on dynamic scenes with high resolutions, while maintaining quality. It showcases potential applications for future developments in film and other media, although there are current limitations regarding the length of motion captured. [6]

See also

Related Research Articles

<span class="mw-page-title-main">Rendering (computer graphics)</span> Process of generating an image from a model

Rendering or image synthesis is the process of generating a photorealistic or non-photorealistic image from a 2D or 3D model by means of a computer program. The resulting image is referred to as a rendering. Multiple models can be defined in a scene file containing objects in a strictly defined language or data structure. The scene file contains geometry, viewpoint, textures, lighting, and shading information describing the virtual scene. The data contained in the scene file is then passed to a rendering program to be processed and output to a digital image or raster graphics image file. The term "rendering" is analogous to the concept of an artist's impression of a scene. The term "rendering" is also used to describe the process of calculating effects in a video editing program to produce the final video output.

<span class="mw-page-title-main">Global illumination</span> Group of rendering algorithms used in 3D computer graphics

Global illumination (GI), or indirect illumination, is a group of algorithms used in 3D computer graphics that are meant to add more realistic lighting to 3D scenes. Such algorithms take into account not only the light that comes directly from a light source, but also subsequent cases in which light rays from the same source are reflected by other surfaces in the scene, whether reflective or not.

<span class="mw-page-title-main">Texture mapping</span> Method of defining surface detail on a computer-generated graphic or 3D model

Texture mapping is a method for mapping a texture on a computer-generated graphic. Texture here can be high frequency detail, surface texture, or color.

<span class="mw-page-title-main">Ray casting</span> Methodological basis for 3D CAD/CAM solid modeling and image rendering

Ray casting is the methodological basis for 3D CAD/CAM solid modeling and image rendering. It is essentially the same as ray tracing for computer graphics where virtual light rays are "cast" or "traced" on their path from the focal point of a camera through each pixel in the camera sensor to determine what is visible along the ray in the 3D scene. The term "Ray Casting" was introduced by Scott Roth while at the General Motors Research Labs from 1978–1980. His paper, "Ray Casting for Modeling Solids", describes modeled solid objects by combining primitive solids, such as blocks and cylinders, using the set operators union (+), intersection (&), and difference (-). The general idea of using these binary operators for solid modeling is largely due to Voelcker and Requicha's geometric modelling group at the University of Rochester. See solid modeling for a broad overview of solid modeling methods. This figure on the right shows a U-Joint modeled from cylinders and blocks in a binary tree using Roth's ray casting system in 1979.

<span class="mw-page-title-main">Scientific visualization</span> Interdisciplinary branch of science concerned with presenting scientific data visually

Scientific visualization is an interdisciplinary branch of science concerned with the visualization of scientific phenomena. It is also considered a subset of computer graphics, a branch of computer science. The purpose of scientific visualization is to graphically illustrate scientific data to enable scientists to understand, illustrate, and glean insight from their data. Research into how people read and misread various types of visualizations is helping to determine what types and features of visualizations are most understandable and effective in conveying information.

<span class="mw-page-title-main">Volume rendering</span> Representing a 3D-modeled object or dataset as a 2D projection

In scientific visualization and computer graphics, volume rendering is a set of techniques used to display a 2D projection of a 3D discretely sampled data set, typically a 3D scalar field.

A light field, or lightfield, is a vector function that describes the amount of light flowing in every direction through every point in a space. The space of all possible light rays is given by the five-dimensional plenoptic function, and the magnitude of each ray is given by its radiance. Michael Faraday was the first to propose that light should be interpreted as a field, much like the magnetic fields on which he had been working. The term light field was coined by Andrey Gershun in a classic 1936 paper on the radiometric properties of light in three-dimensional space.

In computer graphics, level of detail (LOD) refers to the complexity of a 3D model representation. LOD can be decreased as the model moves away from the viewer or according to other metrics such as object importance, viewpoint-relative speed or position. LOD techniques increase the efficiency of rendering by decreasing the workload on graphics pipeline stages, usually vertex transformations. The reduced visual quality of the model is often unnoticed because of the small effect on object appearance when distant or moving fast.

<span class="mw-page-title-main">Surfel</span>

Surfel is an abbreviated term for a "surface element," analogous to a "voxel" or a "pixel". In 3D computer graphics, the use of surfels is an alternative to polygonal modeling. An object is represented by a dense set of points or viewer-facing discs holding lighting information. Surfels are well suited to modeling dynamic geometry, because there is no need to compute topology information such as adjacency lists. Common applications are medical scanner data representation, real time rendering of particle systems, and more generally, rendering surfaces of volumetric data by first extracting the isosurface.

Volume ray casting, sometimes called volumetric ray casting, volumetric ray tracing, or volume ray marching, is an image-based volume rendering technique. It computes 2D images from 3D volumetric data sets. Volume ray casting, which processes volume data, must not be mistaken with ray casting in the sense used in ray tracing, which processes surface data. In the volumetric variant, the computation doesn't stop at the surface but "pushes through" the object, sampling the object along the ray. Unlike ray tracing, volume ray casting does not spawn secondary rays. When the context/application is clear, some authors simply call it ray casting. Because ray marching does not necessarily require an exact solution to ray intersection and collisions, it is suitable for real time computing for many applications for which ray tracing is unsuitable.

<span class="mw-page-title-main">3D rendering</span> Process of converting 3D scenes into 2D images

3D rendering is the 3D computer graphics process of converting 3D models into 2D images on a computer. 3D renders may include photorealistic effects or non-photorealistic styles.

<span class="mw-page-title-main">Volumetric lighting</span> Effect in computer graphics

Volumetric lighting, also known as "God rays", is a technique used in 3D computer graphics to add lighting effects to a rendered scene. It allows the viewer to see beams of light shining across the environment. Examples of volumetric lighting are seeing sunbeams shining through a window and seeing sunbeams radiating when the Sun is below the horizon, also known as crepuscular rays. The term seems to have been introduced from cinematography and is now widely applied to 3D modeling and rendering, especially in the development of 3D video games.

Self-Shadowing is a computer graphics lighting effect, used in 3D rendering applications such as computer animation and video games. Self-shadowing allows non-static objects in the environment, such as game characters and interactive objects, to cast shadows on themselves and each other. For example, without self-shadowing, if a character puts their right arm over the left, the right arm will not cast a shadow over the left arm. If that same character places a hand over a ball, that hand will cast a shadow over the ball.

<span class="mw-page-title-main">Computer graphics</span> Graphics created using computers

Computer graphics deals with generating images and art with the aid of computers. Computer graphics is a core technology in digital photography, film, video games, digital art, cell phone and computer displays, and many specialized applications. A great deal of specialized hardware and software has been developed, with the displays of most devices being driven by computer graphics hardware. It is a vast and recently developed area of computer science. The phrase was coined in 1960 by computer graphics researchers Verne Hudson and William Fetter of Boeing. It is often abbreviated as CG, or typically in the context of film as computer generated imagery (CGI). The non-artistic aspects of computer graphics are the subject of computer science research.

<span class="mw-page-title-main">3D modeling</span> Form of computer-aided engineering

In 3D computer graphics, 3D modeling is the process of developing a mathematical coordinate-based representation of a surface of an object in three dimensions via specialized software by manipulating edges, vertices, and polygons in a simulated 3D space.

In computer vision and computer graphics, 4D reconstruction is the process of capturing the shape and appearance of real objects along a temporal dimension. This process can be accomplished by methods such as depth camera imaging, photometric stereo, or structure from motion, and is also referred to as spatio-temporal reconstruction.

Ray marching is a class of rendering methods for 3D computer graphics where rays are traversed iteratively, effectively dividing each ray into smaller ray segments, sampling some function at each step. For example, in volume ray casting the function would access data points from a 3D scan. In Sphere tracing, the function estimates a distance to step next. Ray marching is also used in physics simulations as an alternative to ray tracing where analytic solutions of the trajectories of light or sound waves are solved. Ray marching for computer graphics often takes advantage of SDFs to determine a maximum safe step-size, while this is less common in physics simulations a similar adaptive step method can be achieved using adaptive Runge-Kutta methods.

Volumetric path tracing is a method for rendering images in computer graphics which was first introduced by Lafortune and Willems. This method enhances the rendering of the lighting in a scene by extending the path tracing method with the effect of light scattering. It is used for photorealistic effects of participating media like fire, explosions, smoke, clouds, fog or soft shadows.

This is a glossary of terms relating to computer graphics.

A neural radiance field (NeRF) is a method based on deep learning for reconstructing a three-dimensional representation of a scene from sparse two-dimensional images. The NeRF model enables learning of novel view synthesis, scene geometry, and the reflectance properties of the scene. Additional scene properties such as camera poses may also be jointly learned. NeRF enables rendering of photorealistic views from novel viewpoints. First introduced in 2020, it has since gained significant attention for its potential applications in computer graphics and content creation.

References

  1. 1 2 3 Westover, Lee Alan (July 1991). "SPLATTING: A Parallel, Feed-Forward Volume Rendering Algorithm" (PDF). Retrieved October 18, 2023.
  2. 1 2 Huang, Jian (Spring 2002). "Splatting" (PPT). Retrieved 5 August 2011.
  3. 1 2 Bernhard Kerbl; Georgios Kopanas; Thomas Leimkühler; George Drettakis (8 Aug 2023). "3D Gaussian Splatting for Real-Time Radiance Field Rendering". arXiv: 2308.04079 [cs.GR].
  4. 1 2 Guanjun Wu; Taoran Yi; Jiemin Fang; Lingxi Xie; Xiaopeng Zhang; Wei Wei; Wenyu Liu; Qi Tian; Xinggang Wang (12 Oct 2023). "4D Gaussian Splatting for Real-Time Dynamic Scene Rendering". arXiv: 2310.08528 [cs.CV].
  5. Papp, Donald. "HIGH QUALITY 3D SCENE GENERATION FROM 2D SOURCE, IN REALTIME". hackaday.com. Hackaday . Retrieved October 18, 2023.
  6. 1 2 Franzen, Carl. "Actors' worst fears come true? New 4D Gaussian splatting method captures human motion". venturebeat.com. VentureBeat . Retrieved October 18, 2023.