Gaussian splatting

Last updated

Gaussian Splatting is a volume rendering technique that deals with the direct rendering of volume data without converting the data into surface or line primitives. [1] The technique was originally introduced as splatting by Lee Westover in the early 1990s. [2] With advancements in computer graphics, newer methods such as 3D and 4D Gaussian splatting have been developed to offer real-time radiance field rendering and dynamic scene rendering respectively. [3] [4]

Contents

Development

Volume rendering focuses on the generation of images from discrete samples of volume data that are usually sampled in three dimensions. This data can originate from a variety of sources, such as CT scans or ozone density readings. Traditional methods transformed this volumetric data into lines and surface primitives to be displayed on computer graphics displays. However, this approach sometimes introduced artifacts and hampered interactive viewing. [1]

To address these issues, direct rendering techniques, which operate on the original volume data, were developed. These methods are classified into two primary categories: feed-backward methods and feed-forward methods. The splatting algorithm, being a feed-forward method, is directly concerned with the rendering of rectilinear volume meshes. Splatting can be customized to render volumes as either clouds or surfaces by altering the shading functions, providing flexibility in the rendering process. [1]

Since its inception, splatting has undergone various improvements. Some significant developments include textured splats, anti-aliasing, image-aligned sheet-based splatting, post-classified splatting, and the introduction of a splat primitive known as FastSplats. [2]

3D Gaussian Splatting

Recent advancements in novel-view synthesis have showcased the utility of Radiance Field methods. To enhance visual quality while ensuring real-time display rates, a new method was introduced that uses 3D Gaussians. This method integrates sparse points produced during camera calibration, representing scenes with 3D Gaussians which retain properties of continuous volumetric radiance fields. Additionally, an interleaved optimization/density control of the 3D Gaussians was introduced along with a rapid visibility-aware rendering algorithm supporting anisotropic splatting. [3] This technique has shown potential in synthesizing high-quality 3D scenes from 2D images in real-time. [5]

4D Gaussian Splatting

Extending the concept of 3D Gaussian Splatting, the 4D Gaussian Splatting incorporates a time component, allowing for dynamic scene rendering. It represents and renders dynamic scenes, with a focus on modeling complex motions while maintaining efficiency. [4] The method uses a HexPlane to connect different adjacent Gaussians, providing an accurate representation of position and shape deformations. By utilizing only a single set of canonical 3D Gaussians, and predictive analytics, the 4D Gaussian splatting method models how they move over different timestamps. [6]

Achievements of this technique include real-time rendering on dynamic scenes with high resolutions, while maintaining quality. It showcases potential applications for future developments in film and other media, although there are current limitations regarding the length of motion captured. [6]

See also

Related Research Articles

<span class="mw-page-title-main">Rendering (computer graphics)</span> Process of generating an image from a model

Rendering or image synthesis is the process of generating a photorealistic or non-photorealistic image from a 2D or 3D model by means of a computer program. The resulting image is referred to as the render. Multiple models can be defined in a scene file containing objects in a strictly defined language or data structure. The scene file contains geometry, viewpoint, textures, lighting, and shading information describing the virtual scene. The data contained in the scene file is then passed to a rendering program to be processed and output to a digital image or raster graphics image file. The term "rendering" is analogous to the concept of an artist's impression of a scene. The term "rendering" is also used to describe the process of calculating effects in a video editing program to produce the final video output.

<span class="mw-page-title-main">Global illumination</span> Group of rendering algorithms used in 3D computer graphics

Global illumination (GI), or indirect illumination, is a group of algorithms used in 3D computer graphics that are meant to add more realistic lighting to 3D scenes. Such algorithms take into account not only the light that comes directly from a light source, but also subsequent cases in which light rays from the same source are reflected by other surfaces in the scene, whether reflective or not.

<span class="mw-page-title-main">Ray casting</span> Methodological basis for 3D CAD/CAM solid modeling and image rendering

Ray casting is the methodological basis for 3D CAD/CAM solid modeling and image rendering. It is essentially the same as ray tracing for computer graphics where virtual light rays are "cast" or "traced" on their path from the focal point of a camera through each pixel in the camera sensor to determine what is visible along the ray in the 3D scene. The term "Ray Casting" was introduced by Scott Roth while at the General Motors Research Labs from 1978–1980. His paper, "Ray Casting for Modeling Solids", describes modeled solid objects by combining primitive solids, such as blocks and cylinders, using the set operators union (+), intersection (&), and difference (-). The general idea of using these binary operators for solid modeling is largely due to Voelcker and Requicha's geometric modelling group at the University of Rochester. See solid modeling for a broad overview of solid modeling methods. This figure on the right shows a U-Joint modeled from cylinders and blocks in a binary tree using Roth's ray casting system in 1979.

<span class="mw-page-title-main">Scientific visualization</span> Interdisciplinary branch of science concerned with presenting scientific data visually

Scientific visualization is an interdisciplinary branch of science concerned with the visualization of scientific phenomena. It is also considered a subset of computer graphics, a branch of computer science. The purpose of scientific visualization is to graphically illustrate scientific data to enable scientists to understand, illustrate, and glean insight from their data. Research into how people read and misread various types of visualizations is helping to determine what types and features of visualizations are most understandable and effective in conveying information.

<span class="mw-page-title-main">Volume rendering</span> Representing a 3D-modeled object or dataset as a 2D projection

In scientific visualization and computer graphics, volume rendering is a set of techniques used to display a 2D projection of a 3D discretely sampled data set, typically a 3D scalar field.

In computer graphics, level of detail (LOD) refers to the complexity of a 3D model representation. LOD can be decreased as the model moves away from the viewer or according to other metrics such as object importance, viewpoint-relative speed or position. LOD techniques increase the efficiency of rendering by decreasing the workload on graphics pipeline stages, usually vertex transformations. The reduced visual quality of the model is often unnoticed because of the small effect on object appearance when distant or moving fast.

<span class="mw-page-title-main">Real-time computer graphics</span> Sub-field of computer graphics

Real-time computer graphics or real-time rendering is the sub-field of computer graphics focused on producing and analyzing images in real time. The term can refer to anything from rendering an application's graphical user interface (GUI) to real-time image analysis, but is most often used in reference to interactive 3D computer graphics, typically using a graphics processing unit (GPU). One example of this concept is a video game that rapidly renders changing 3D environments to produce an illusion of motion.

<span class="mw-page-title-main">Surfel</span>

Surfel is an abbreviated term for a "surface element," analogous to a "voxel" or a "pixel". In 3D computer graphics, the use of surfels is an alternative to polygonal modeling. An object is represented by a dense set of points or viewer-facing discs holding lighting information. Surfels are well suited to modeling dynamic geometry, because there is no need to compute topology information such as adjacency lists. Common applications are medical scanner data representation, real time rendering of particle systems, and more generally, rendering surfaces of volumetric data by first extracting the isosurface.

<span class="mw-page-title-main">Path tracing</span> Computer graphics method

Path tracing is a computer graphics Monte Carlo method of rendering images of three-dimensional scenes such that the global illumination is faithful to reality. Fundamentally, the algorithm is integrating over all the illuminance arriving to a single point on the surface of an object. This illuminance is then reduced by a surface reflectance function (BRDF) to determine how much of it will go towards the viewpoint camera. This integration procedure is repeated for every pixel in the output image. When combined with physically accurate models of surfaces, accurate models of real light sources, and optically correct cameras, path tracing can produce still images that are indistinguishable from photographs.

Volume ray casting, sometimes called volumetric ray casting, volumetric ray tracing, or volume ray marching, is an image-based volume rendering technique. It computes 2D images from 3D volumetric data sets. Volume ray casting, which processes volume data, must not be mistaken with ray casting in the sense used in ray tracing, which processes surface data. In the volumetric variant, the computation doesn't stop at the surface but "pushes through" the object, sampling the object along the ray. Unlike ray tracing, volume ray casting does not spawn secondary rays. When the context/application is clear, some authors simply call it ray casting. Because ray marching does not necessarily require an exact solution to ray intersection and collisions, it is suitable for real time computing for many applications for which ray tracing is unsuitable.

<span class="mw-page-title-main">3D rendering</span> Process of converting 3D scenes into 2D images

3D rendering is the 3D computer graphics process of converting 3D models into 2D images on a computer. 3D renders may include photorealistic effects or non-photorealistic styles.

<span class="mw-page-title-main">Volumetric lighting</span> Effect in computer graphics

Volumetric lighting, also known as "God rays", is a technique used in 3D computer graphics to add lighting effects to a rendered scene. It allows the viewer to see beams of light shining across the environment. Examples of volumetric lighting are seeing sunbeams shining through a window and seeing sunbeams radiating when the Sun is below the horizon, also known as crepuscular rays. The term seems to have been introduced from cinematography and is now widely applied to 3D modeling and rendering, especially in the development of 3D video games.

Self-Shadowing is a computer graphics lighting effect, used in 3D rendering applications such as computer animation and video games. Self-shadowing allows non-static objects in the environment, such as game characters and interactive objects, to cast shadows on themselves and each other. For example, without self-shadowing, if a character puts their right arm over the left, the right arm will not cast a shadow over the left arm. If that same character places a hand over a ball, that hand will cast a shadow over the ball.

Screen space ambient occlusion (SSAO) is a computer graphics technique for efficiently approximating the ambient occlusion effect in real time. It was developed by Vladimir Kajalin while working at Crytek and was used for the first time in 2007 by the video game Crysis, also developed by Crytek.

<span class="mw-page-title-main">3D modeling</span> Form of computer-aided engineering

In 3D computer graphics, 3D modeling is the process of developing a mathematical coordinate-based representation of a surface of an object in three dimensions via specialized software by manipulating edges, vertices, and polygons in a simulated 3D space.

In computer vision and computer graphics, 4D reconstruction is the process of capturing the shape and appearance of real objects along a temporal dimension. This process can be accomplished by methods such as depth camera imaging, photometric stereo, or structure from motion, and is also referred to as spatio-temporal reconstruction.

Ray marching is a class of rendering methods for 3D computer graphics where rays are traversed iteratively, effectively dividing each ray into smaller ray segments, sampling some function at each step. For example, in volume ray casting the function would access data points from a 3D scan. In Sphere tracing, the function estimates a distance to step next.

Volumetric path tracing is a method for rendering images in computer graphics which was first introduced by Lafortune and Willems. This method enhances the rendering of the lighting in a scene by extending the path tracing method with the effect of light scattering. It is used for photorealistic effects of participating media like fire, explosions, smoke, clouds, fog or soft shadows.

This is a glossary of terms relating to computer graphics.

A neural radiance field (NeRF) is a method based on deep learning for reconstructing a three-dimensional representation of a scene from sparse two-dimensional images. The NeRF model enables learning of novel view synthesis, scene geometry, and the reflectance properties of the scene. Additional scene properties such as camera poses may also be jointly learned. NeRF enables rendering of photorealistic views from novel viewpoints. First introduced in 2020, it has since gained significant attention for its potential applications in computer graphics and content creation.

References

  1. 1 2 3 Westover, Lee Alan (July 1991). "SPLATTING: A Parallel, Feed-Forward Volume Rendering Algorithm" (PDF). Retrieved October 18, 2023.
  2. 1 2 Huang, Jian (Spring 2002). "Splatting" (PPT). Retrieved 5 August 2011.
  3. 1 2 Bernhard Kerbl; Georgios Kopanas; Thomas Leimkühler; George Drettakis (8 Aug 2023). "3D Gaussian Splatting for Real-Time Radiance Field Rendering". arXiv: 2308.04079 [cs.GR].
  4. 1 2 Guanjun Wu; Taoran Yi; Jiemin Fang; Lingxi Xie; Xiaopeng Zhang; Wei Wei; Wenyu Liu; Qi Tian; Xinggang Wang (12 Oct 2023). "4D Gaussian Splatting for Real-Time Dynamic Scene Rendering". arXiv: 2310.08528 [cs.CV].
  5. Papp, Donald. "HIGH QUALITY 3D SCENE GENERATION FROM 2D SOURCE, IN REALTIME". hackaday.com. Hackaday . Retrieved October 18, 2023.
  6. 1 2 Franzen, Carl. "Actors' worst fears come true? New 4D Gaussian splatting method captures human motion". venturebeat.com. VentureBeat . Retrieved October 18, 2023.