Computational photography

Last updated
Computational photography provides many new capabilities. This example combines HDR (High Dynamic Range) imaging with panoramics (image-stitching), by optimally combining information from multiple differently exposed pictures of overlapping subject matter. Process nocomparam.png
Computational photography provides many new capabilities. This example combines HDR (High Dynamic Range) imaging with panoramics (image-stitching), by optimally combining information from multiple differently exposed pictures of overlapping subject matter.

Computational photography refers to digital image capture and processing techniques that use digital computation instead of optical processes. Computational photography can improve the capabilities of a camera, or introduce features that were not possible at all with film-based photography, or reduce the cost or size of camera elements. Examples of computational photography include in-camera computation of digital panoramas, [6] high-dynamic-range images, and light field cameras. Light field cameras use novel optical elements to capture three dimensional scene information which can then be used to produce 3D images, enhanced depth-of-field, and selective de-focusing (or "post focus"). Enhanced depth-of-field reduces the need for mechanical focusing systems. All of these features use computational imaging techniques.

Contents

The definition of computational photography has evolved to cover a number of subject areas in computer graphics, computer vision, and applied optics. These areas are given below, organized according to a taxonomy proposed by Shree K. Nayar [ citation needed ]. Within each area is a list of techniques, and for each technique one or two representative papers or books are cited. Deliberately omitted from the taxonomy are image processing (see also digital image processing) techniques applied to traditionally captured images in order to produce better images. Examples of such techniques are image scaling, dynamic range compression (i.e. tone mapping), color management, image completion (a.k.a. inpainting or hole filling), image compression, digital watermarking, and artistic image effects. Also omitted are techniques that produce range data, volume data, 3D models, 4D light fields, 4D, 6D, or 8D BRDFs, or other high-dimensional image-based representations. Epsilon photography is a sub-field of computational photography.

Effect on photography

Photos taken using computational photography can allow amateurs to produce photographs rivalling the quality of professional photographers, but as of 2019 do not outperform the use of professional-level equipment. [7]

Computational illumination

This is controlling photographic illumination in a structured fashion, then processing the captured images, to create new images. The applications include image-based relighting, image enhancement, image deblurring, geometry/material recovery and so forth.

High-dynamic-range imaging uses differently exposed pictures of the same scene to extend dynamic range. [8] Other examples include processing and merging differently illuminated images of the same subject matter ("lightspace").

Computational optics

This is capture of optically coded images, followed by computational decoding to produce new images. Coded aperture imaging was mainly applied in astronomy or X-ray imaging to boost the image quality. Instead of a single pin-hole, a pinhole pattern is applied in imaging, and deconvolution is performed to recover the image. [9] In coded exposure imaging, the on/off state of the shutter is coded to modify the kernel of motion blur. [10] In this way motion deblurring becomes a well-conditioned problem. Similarly, in a lens based coded aperture, the aperture can be modified by inserting a broadband mask. [11] Thus, out of focus deblurring becomes a well-conditioned problem. The coded aperture can also improve the quality in light field acquisition using Hadamard transform optics.

Coded aperture patterns can also be designed using color filters, in order to apply different codes at different wavelengths. [12] [13] This allows to increase the amount of light that reaches the camera sensor, compared to binary masks.

Computational imaging

Computational imaging is a set of imaging techniques that combine data acquisition and data processing to create the image of an object through indirect means to yield enhanced resolution, additional information such as optical phase or 3D reconstruction. The information is often recorded without using a conventional optical microscope configuration or with limited datasets.

Computational imaging allows to go beyond physical limitations of optical systems, such as numerical aperture, [14] or even obliterates the need for optical elements. [15]

For parts of the optical spectrum where imaging elements such as objectives are difficult to manufacture or image sensors cannot be miniaturized, computational imaging provides useful alternatives, in fields such as X-ray [16] and THz radiations.

Common techniques

Among common computational imaging techniques are lensless imaging, computational speckle imaging, [17] ptychography and Fourier ptychography.

Computational imaging technique often draws on compressive sensing or phase retrieval techniques, where the angular spectrum of the object is being reconstructed. Other techniques are related to the field of computational imaging, such as digital holography, computer vision and inverse problems such as tomography.

Computational processing

This is processing of non-optically-coded images to produce new images.

Computational sensors

These are detectors that combine sensing and processing, typically in hardware, like the oversampled binary image sensor.

Early work in computer vision

Although computational photography is a currently popular buzzword in computer graphics, many of its techniques first appeared in the computer vision literature, either under other names or within papers aimed at 3D shape analysis.

A 1981 wearable computational photography apparatus Lightspace-wearcomp-lightcomb.jpg
A 1981 wearable computational photography apparatus

Art history

Wearable Computational Photography originated in the 1970s and early 1980s, and has evolved into a more recent art form. This picture was used on the cover of the John Wiley and Sons textbook on the subject. Computational-photography-lightvectoring-as-art-form.jpg
Wearable Computational Photography originated in the 1970s and early 1980s, and has evolved into a more recent art form. This picture was used on the cover of the John Wiley and Sons textbook on the subject.

Computational photography, as an art form, has been practiced by capture of differently exposed pictures of the same subject matter, and combining them together. This was the inspiration for the development of the wearable computer in the 1970s and early 1980s. Computational photography was inspired by the work of Charles Wyckoff, and thus computational photography datasets (e.g. differently exposed pictures of the same subject matter that are taken in order to make a single composite image) are sometimes referred to as Wyckoff Sets, in his honor.

Early work in this area (joint estimation of image projection and exposure value) was undertaken by Mann and Candoccia.

Charles Wyckoff devoted much of his life to creating special kinds of 3-layer photographic films that captured different exposures of the same subject matter. A picture of a nuclear explosion, taken on Wyckoff's film, appeared on the cover of Life Magazine and showed the dynamic range from dark outer areas to inner core.

See also

Related Research Articles

Computer vision tasks include methods for acquiring, processing, analyzing and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the forms of decisions. Understanding in this context means the transformation of visual images into descriptions of the world that make sense to thought processes and can elicit appropriate action. This image understanding can be seen as the disentangling of symbolic information from image data using models constructed with the aid of geometry, physics, statistics, and learning theory.

<span class="mw-page-title-main">Depth of field</span> Distance between the nearest and the furthest objects that are in focus in an image

The depth of field (DOF) is the distance between the nearest and the furthest objects that are in acceptably sharp focus in an image captured with a camera.

<span class="mw-page-title-main">Multi-exposure HDR capture</span> Technique to capture HDR images and videos

In photography and videography, multi-exposure HDR capture is a technique that creates high dynamic range (HDR) images by taking and combining multiple exposures of the same subject matter at different exposure levels. Combining multiple images in this way results in an image with a greater dynamic range than what would be possible by taking one single image. The technique can also be used to capture video by taking and combining multiple exposures for each frame of the video. The term "HDR" is used frequently to refer to the process of creating HDR images from multiple exposures. Many smartphones have an automated HDR feature that relies on computational imaging techniques to capture and combine multiple exposures.

<span class="mw-page-title-main">Imaging</span> Representation or reproduction of an objects form

Imaging is the representation or reproduction of an object's form; especially a visual representation.

The light field is a vector function that describes the amount of light flowing in every direction through every point in space. The space of all possible light rays is given by the five-dimensional plenoptic function, and the magnitude of each ray is given by its radiance. Michael Faraday was the first to propose that light should be interpreted as a field, much like the magnetic fields on which he had been working. The phrase light field was coined by Andrey Gershun in a classic 1936 paper on the radiometric properties of light in three-dimensional space.

<span class="mw-page-title-main">Vignetting</span> Reduction of an images brightness or saturation toward the periphery compared to the image center

In photography and optics, vignetting is a reduction of an image's brightness or saturation toward the periphery compared to the image center. The word vignette, from the same root as vine, originally referred to a decorative border in a book. Later, the word came to be used for a photographic portrait that is clear at the center and fades off toward the edges. A similar effect is visible in photographs of projected images or videos off a projection screen, resulting in a so-called "hotspot" effect.

<span class="mw-page-title-main">Light field camera</span> Type of camera that can also capture the direction of travel of light rays

A light field camera, also known as a plenoptic camera, is a camera that captures information about the light field emanating from a scene; that is, the intensity of light in a scene, and also the precise direction that the light rays are traveling in space. This contrasts with conventional cameras, which record only light intensity at various wavelengths.

<span class="mw-page-title-main">Digital photography</span> Photography with a digital camera

Digital photography uses cameras containing arrays of electronic photodetectors interfaced to an analog-to-digital converter (ADC) to produce images focused by a lens, as opposed to an exposure on photographic film. The digitized image is stored as a computer file ready for further digital processing, viewing, electronic publishing, or digital printing. It is a form of digital imaging based on gathering visible light.

The following are common definitions related to the machine vision field.

In optics and signal processing, wavefront coding refers to the use of a phase modulating element in conjunction with deconvolution to extend the depth of field of a digital imaging system such as a video camera.

The following outline is provided as an overview of and topical guide to photography:

<span class="mw-page-title-main">Coded aperture</span>

Coded apertures or coded-aperture masks are grids, gratings, or other patterns of materials opaque to various wavelengths of electromagnetic radiation. The wavelengths are usually high-energy radiation such as X-rays and gamma rays. A coded "shadow" is cast upon a plane by blocking radiation in a known pattern. The properties of the original radiation sources can then be mathematically reconstructed from this shadow. Coded apertures are used in X- and gamma ray imaging systems, because these high-energy rays cannot be focused with lenses or mirrors that work for visible light.

<span class="mw-page-title-main">Focus stacking</span> Digital image processing technique

Focus stacking is a digital image processing technique which combines multiple images taken at different focus distances to give a resulting image with a greater depth of field (DOF) than any of the individual source images. Focus stacking can be used in any situation where individual images have a very shallow depth of field; macro photography and optical microscopy are two typical examples. Focus stacking can also be useful in landscape photography.

Range imaging is the name for a collection of techniques that are used to produce a 2D image showing the distance to points in a scene from a specific point, normally associated with some type of sensor device.

<span class="mw-page-title-main">3D reconstruction</span> Process of capturing the shape and appearance of real objects

In computer vision and computer graphics, 3D reconstruction is the process of capturing the shape and appearance of real objects. This process can be accomplished either by active or passive methods. If the model is allowed to change its shape in time, this is referred to as non-rigid or spatio-temporal reconstruction.

<span class="mw-page-title-main">Femto-photography</span>

Femto-photography is a technique for recording the propagation of ultrashort pulses of light through a scene at a very high speed (up to 1013 frames per second). A femto-photograph is equivalent to an optical impulse response of a scene and has also been denoted by terms such as a light-in-flight recording or transient image. Femto-photography of macroscopic objects was first demonstrated using a holographic process in the 1970s by Nils Abramsson at the Royal Institute of Technology (Sweden). A research team at the MIT Media Lab led by Ramesh Raskar, together with contributors from the Graphics and Imaging Lab at the Universidad de Zaragoza, Spain, more recently achieved a significant increase in image quality using a streak camera synchronized to a pulsed laser and modified to obtain 2D images instead of just a single scanline.

Gradient domain image processing, also called Poisson image editing, is a type of digital image processing that operates directly on the differences between neighboring pixels, rather than on the pixel values. Mathematically, an image gradient represents the derivative of an image, so the goal of gradient domain processing is to construct a new image by integrating the gradient, which requires solving Poisson's equation.

Epsilon photography is a form of computational photography wherein multiple images are captured with slightly varying camera parameters such as aperture, exposure, focus, film speed and viewpoint for the purpose of enhanced post-capture flexibility. The term was coined by Prof. Ramesh Raskar. The technique has been developed as an alternative to light field photography that requires no specialized equipment. Examples of epsilon photography include focal stack photography, High dynamic range (HDR) photography, lucky imaging, multi-image panorama stitching and confocal stereo. The common thread for all the aforementioned imaging techniques is that multiple images are captured in order to produce a composite image of higher quality, such as richer color information, wider-field of view, more accurate depth map, less noise/blur and greater resolution.

Computational imaging is the process of indirectly forming images from measurements using algorithms that rely on a significant amount of computing. In contrast to traditional imaging, computational imaging systems involve a tight integration of the sensing system and the computation in order to form the images of interest. The ubiquitous availability of fast computing platforms, the advances in algorithms and modern sensing hardware is resulting in imaging systems with significantly enhanced capabilities. Computational Imaging systems cover a broad range of applications include computational microscopy, tomographic imaging, MRI, ultrasound imaging, computational photography, Synthetic Aperture Radar (SAR), seismic imaging etc. The integration of the sensing and the computation in computational imaging systems allows for accessing information which was otherwise not possible. For example:

Todor G. Georgiev is a Bulgarian American research scientist and inventor, best known for his work on plenoptic cameras. He is the author of the Healing Brush tool in Adobe Photoshop, and, as of 2020, is a principal scientist at Adobe in San Jose, California. Georgiev's work has been cited 7700 times as of 2020. As an inventor, he has at least 89 patents to his name.

References

  1. Steve Mann. "Compositing Multiple Pictures of the Same Scene", Proceedings of the 46th Annual Imaging Science & Technology Conference, May 9–14, Cambridge, Massachusetts, 1993
  2. S. Mann, C. Manders, and J. Fung, "The Lightspace Change Constraint Equation (LCCE) with practical application to estimation of the projectivity+gain transformation between multiple pictures of the same subject matter" IEEE International Conference on Acoustics, Speech, and Signal Processing, 6–10 April 2003, pp III - 481-4 vol. 3.
  3. joint parameter estimation in both domain and range of functions in same orbit of the projective-Wyckoff group" ", IEEE International Conference on Image Processing, Vol. 3, 16-19, pp. 193-196 September 1996
  4. Frank M. Candocia: Jointly registering images in domain and range by piecewise linear comparametric analysis. IEEE Transactions on Image Processing 12(4): 409-419 (2003)
  5. Frank M. Candocia: Simultaneous homographic and comparametric alignment of multiple exposure-adjusted pictures of the same scene. IEEE Transactions on Image Processing 12(12): 1485-1494 (2003)
  6. Steve Mann and R. W. Picard. "Virtual bellows: constructing high-quality stills from video.", In Proceedings of the IEEE First International Conference on Image ProcessingAustin, Texas, November 13–16, 1994
  7. "The Edge of Computational Photography".
  8. ON BEING `UNDIGITAL' WITH DIGITAL CAMERAS: EXTENDING DYNAMIC RANGE BY COMBINING DIFFERENTLY EXPOSED PICTURES, IS&T's (Society for Imaging Science and Technology's) 48th annual conference, Cambridge, Massachusetts, May 1995, pages 422-428
  9. Martinello, Manuel. "Coded Aperture Imaging" (PDF).
  10. Raskar, Ramesh; Agrawal, Amit; Tumblin, Jack (2006). "Coded Exposure Photography: Motion Deblurring using Fluttered Shutter" . Retrieved November 29, 2010.
  11. Veeraraghavan, Ashok; Raskar, Ramesh; Agrawal, Amit; Mohan, Ankit; Tumblin, Jack (2007). "Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing" . Retrieved November 29, 2010.
  12. Martinello, Manuel; Wajs, Andrew; Quan, Shuxue; Lee, Hank; Lim, Chien; Woo, Taekun; Lee, Wonho; Kim, Sang-Sik; Lee, David (2015). "Dual Aperture Photography: Image and Depth from a Mobile Camera" (PDF). International Conference on Computational Photography.
  13. Chakrabarti, A.; Zickler, T. (2012). "Depth and deblurring from a spectrally-varying depth-of-field". IEEE European Conference on Computer Vision. 7576: 648–666.
  14. Ou et al., "High numerical aperture Fourier ptychography: principle, implementation and characterization" Optics Express 23, 3 (2015)
  15. Boominathan et al., "Lensless Imaging: A Computational Renaissance" (2016)
  16. Miyakawa et al., "Coded aperture detector : an image sensor with sub 20-nm pixel resolution", Optics Express 22, 16 (2014)
  17. Katz et al., "Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations", Nature Photonics 8, 784–790 (2014)