Computational imaging

Last updated September 03, 2023

Computational imaging is the process of indirectly forming images from measurements using algorithms that rely on a significant amount of computing. In contrast to traditional imaging, computational imaging systems involve a tight integration of the sensing system and the computation in order to form the images of interest. The ubiquitous availability of fast computing platforms (such as multi-core CPUs and GPUs), the advances in algorithms and modern sensing hardware is resulting in imaging systems with significantly enhanced capabilities. Computational Imaging systems cover a broad range of applications include computational microscopy,^[1] tomographic imaging, MRI, ultrasound imaging, computational photography, Synthetic Aperture Radar (SAR), seismic imaging etc. The integration of the sensing and the computation in computational imaging systems allows for accessing information which was otherwise not possible. For example:

A single X-ray image does not reveal the precise location of fracture, but a CT scan which works by combining multiple X-ray images can determine the precise location of one in 3D
A typical camera image cannot image around corners. However, by designing a set-up that involves sending fast pulses of light, recording the received signal and using an algorithm, researchers have demonstrated the first steps in building such a system.^[2]

Computational imaging systems also enable system designers to overcome some hardware limitations of optics and sensors (resolution, noise etc.) by overcoming challenges in the computing domain. Some examples of such systems include coherent diffractive imaging, coded-aperture imaging and image super-resolution.

Computational imaging differs from image processing in a sense that the primary goal of the former is to reconstruct human-recognizable images from measured data via algorithms while the latter is to process already-recognizable images (that may be not sufficient in quality) to improve the quality or derive some information from them.

History

Computational imaging systems span a broad range of applications. While applications such as SAR, computed tomography, seismic inversion are well known, they have undergone significant improvements (faster, higher-resolution, lower dose exposures^[3]) driven by advances in signal and image processing algorithms (including compressed sensing techniques), and faster computing platforms. Photography has evolved from purely chemical processing to now being able to capture and computationally fuse multiple digital images (computational photography)^[4] making techniques such as HDR and panoramic imaging available to most cell-phone users. Computational imaging has also seen an emergence of techniques that modify the light source incident on an object using known structure/patterns and then reconstructing an image from what is received (For example: coded-aperture imaging, super-resolution microscopy, Fourier ptychography). Advances in the development of powerful parallel computing platforms has played a vital role in being able to make advances in computational imaging.

Techniques

Coded aperture imaging

Imaging is usually made at optical wavelengths by lenses and mirrors. However, for X-rays and Gamma-rays, lenses and mirrors are impractical, therefore modulating apertures are often used instead. The pinhole camera is the most basic form of such a modulation imager, but its disadvantage is low throughput, as its small aperture allows through little radiation. Since only a tiny fraction of the light passes through the pinhole, which causes a low signal-to-noise ratio, imaging through pinholes involves unacceptable long exposures. This problem can be overcome to some degree by making the hole larger, which unfortunately leads to a decrease in resolution. Pinhole cameras have a couple of advantages over lenses - they have infinite depth of field, and they don't suffer from chromatic aberration, which can be cured in a refractive system only by using a multiple element lens. The smallest feature which can be resolved by a pinhole is approximately the same size as the pinhole itself. The larger the hole, the more blurred the image becomes. Using multiple, small pinholes might seem to offer a way around this problem, but this gives rise to a confusing montage of overlapping images. Nonetheless, if the pattern of holes is carefully chosen, it is possible to reconstruct the original image with a resolution equal to that of a single hole.

In recent years much work has been done using patterns of holes of clear and opaque regions, constituting what is called a coded aperture. The motivation for using coded aperture imaging techniques is to increase the photon collection efficiency whilst maintaining the high angular resolution of a single pinhole. Coded aperture imaging (CAI) is a two-stage imaging process. The coded image is obtained by the convolution of the object with the intensity point spread function (PSF) of the coded aperture. Once the coded picture is formed it has to be decoded to yield the image. This decoding can be performed in three ways, namely correlation, Fresnel diffraction or deconvolution. An estimation of the original image is attained by convolving the coded image with the original coded aperture. In general, the recovered image will be the convolution of the object with the autocorrelation of the coded aperture and will contain artifacts unless its autocorrelation is a delta function.

Some examples of coded apertures include the Fresnel zone plate (FZP), random arrays (RA), non-redundant arrays (NRA), uniformly redundant arrays (URA), modified uniformly redundant arrays (MURA), among others. Fresnel zone plates, called after Augustin-Jean Fresnel, may not be considered coded apertures at all since they consist of a set of radially symmetric rings, known as Fresnel zones, which alternate between opaque and transparent. They use diffraction instead of refraction or reflection to focus the light. Light hitting the FZP will diffract around the opaque zones, therefore an image will be created when constructive interference occurs. The opaque and transparent zones can be spaced so that imaging occurs at different focuses.

In the early work on coded-apertures, pinholes were randomly distributed on the mask and placed in front of a source to be analyzed. Random patterns, however, pose difficulties with image reconstruction due to a lack of uniformity in pinholes distribution. An inherent noise appears as a result of small terms present in the Fourier transform of large size random binary arrays. This problem was addressed by the development of uniformly redundant arrays (URAs). If the distribution of the transparent and opaque elements of the aperture can be represented as a binary encoding array A and the decoding array as G, then A and G can be chosen such that the reconstructed image (correlation of A and G with an addition of some noise signal N) approximates a delta function. It has experimentally been shown that URAs offer significant improvements to SNR in comparison with randomly distributed arrays, however, the algorithm used for the construction of URAs restricts the shape of the aperture to a rectangle. Therefore, Modified Uniformly Redundant Array (MURA), was introduced with a change to URA's encoding algorithm, enabling new arrays to be created in linear, hexagonal and square configurations. The design method for URAs was modified so that the new arrays were based on quadratic residues rather than pseudo-noise (PN) sequences.

Compressive spectral imaging

Conventional spectral imaging techniques typically scan adjacent zones of the underlying spectral scene and then merge the results to construct a spectral data cube. In contrast, compressive spectral imaging (CSI), which naturally embodies the principles of compressed sensing (CS), involves the acquisition of the spatial-spectral information in 2-dimensional sets of multiplexed projections. The remarkable advantage of compressive spectral imaging is that the entire data cube is sensed with just a few measurements and in some cases with as little as a single FPA snapshot such that the entire data set can be obtained during a single detector integration period.

In general, compressive spectral imaging systems exploit different optical phenomena such as spatial, spectral, or spatial-spectral coding and dispersion, to acquire the compressive measurements. The significant advantage behind CSI is that it is possible to design sensing protocols that capture the essential information from sparse signals with a reduced amount of measurements. Because the amount of captured projections is less than the number of voxels in the spectral data cube, the reconstruction process is performed by numerical optimization algorithms. This is the step where computational imaging plays a key role because the power of computational algorithms and mathematics is exploited to recover the underlying data cube.

In the CSI literature, different strategies can be encountered to attain the coded projections.^[5]^[6]^[7] The coded aperture snapshot spectral imager (CASSI) was the first spectral imager designed to take advantage of compressive sensing theory.^[8] CASSI employs binary coded apertures that create a transmission pattern at each column, such that these patterns are orthogonal with respect to all other columns. The spatial-spectral projection at the detector array is modulated by the binary mask in such a way that each wavelength of the data cube is affected by a shifted modulation code. More recent CSI systems include the CASSI using colored coded apertures (C-CASSI) instead of the black and white masks; a compact version of the colored CASSI, called snapshot colored compressive spectral imager (SCCSI), and a variation of the latter that uses a black-and-white coded aperture in the convolutional plane, known as the spatial–spectral encoded hyperspectral imager (SSCSI). Common characteristics of this kind of CSI systems include the use of a dispersive element to decouple the spectral information, and a coding element to encode the incoming data.

Algorithms

While computational imaging covers a broad range of applications, the algorithms used in computational imaging systems are often related to solving a mathematical inverse problem. The algorithms are generally divided into direct inversion techniques which are often "fast" and iterative reconstruction techniques that are computationally expensive but are able to model more complex physical processes. The typical steps to design algorithms for computational imaging systems are:

Formulating a relationship between the measurements and the quantity to be estimated. This process requires a mathematical model for how the measurements are related to the unknown. For example: In high-dynamic range imaging, the measurements are a sequence of known exposures of the underlying area to be imaged. In an X-ray CT scan, the measurements are X-ray images of the patient obtained from several known positions of the X-ray source and detector camera with a well-established relationship for X-ray propagation.
Choosing a metric to "invert" the measurements and reconstruct the quantity of interest. This could be a simple metric such as a least-squares difference between the measurements and the model or a more sophisticated metric based on precisely modeling the noise statistics of the detector and a model for the object of interest. This choice can be related to choosing a statistical estimator for the quantity to be reconstructed.
Designing fast and robust algorithms that compute the solution to Step 2. These algorithms often use techniques from mathematical optimization and mapping such methods to fast computing platforms to build practical systems.

Related Research Articles

Ray casting is the methodological basis for 3D CAD/CAM solid modeling and image rendering. It is essentially the same as ray tracing for computer graphics where virtual light rays are "cast" or "traced" on their path from the focal point of a camera through each pixel in the camera sensor to determine what is visible along the ray in the 3D scene. The term "Ray Casting" was introduced by Scott Roth while at the General Motors Research Labs from 1978–1980. His paper, "Ray Casting for Modeling Solids", describes modeled solid objects by combining primitive solids, such as blocks and cylinders, using the set operators union (+), intersection (&), and difference (-). The general idea of using these binary operators for solid modeling is largely due to Voelcker and Requicha's geometric modelling group at the University of Rochester. See solid modeling for a broad overview of solid modeling methods. This figure on the right shows a U-Joint modeled from cylinders and blocks in a binary tree using Roth's ray casting system in 1979.

<span class="mw-page-title-main">Synthetic-aperture radar</span> Form of radar used to create images of landscapes

Synthetic-aperture radar (SAR) is a form of radar that is used to create two-dimensional images or three-dimensional reconstructions of objects, such as landscapes. SAR uses the motion of the radar antenna over a target region to provide finer spatial resolution than conventional stationary beam-scanning radars. SAR is typically mounted on a moving platform, such as an aircraft or spacecraft, and has its origins in an advanced form of side looking airborne radar (SLAR). The distance the SAR device travels over a target during the period when the target scene is illuminated creates the large synthetic antenna aperture. Typically, the larger the aperture, the higher the image resolution will be, regardless of whether the aperture is physical or synthetic – this allows SAR to create high-resolution images with comparatively small physical antennas. For a fixed antenna size and orientation, objects which are further away remain illuminated longer - therefore SAR has the property of creating larger synthetic apertures for more distant objects, which results in a consistent spatial resolution over a range of viewing distances.

The light field is a vector function that describes the amount of light flowing in every direction through every point in space. The space of all possible light rays is given by the five-dimensional plenoptic function, and the magnitude of each ray is given by its radiance. Michael Faraday was the first to propose that light should be interpreted as a field, much like the magnetic fields on which he had been working. The phrase light field was coined by Andrey Gershun in a classic 1936 paper on the radiometric properties of light in three-dimensional space.

$<span class="mw-page-title-main">Zone plate</span> Device used to focus light using diffraction$

A zone plate is a device used to focus light or other things exhibiting wave character. Unlike lenses or curved mirrors, zone plates use diffraction instead of refraction or reflection. Based on analysis by French physicist Augustin-Jean Fresnel, they are sometimes called Fresnel zone plates in his honor. The zone plate's focusing ability is an extension of the Arago spot phenomenon caused by diffraction from an opaque disc.

Iterative reconstruction refers to iterative algorithms used to reconstruct 2D and 3D images in certain imaging techniques. For example, in computed tomography an image must be reconstructed from projections of an object. Here, iterative reconstruction techniques are usually a better, but computationally more expensive alternative to the common filtered back projection (FBP) method, which directly calculates the image in a single reconstruction step. In recent research works, scientists have shown that extremely fast computations and massive parallelism is possible for iterative reconstruction, which makes iterative reconstruction practical for commercialization.

Computational photography refers to digital image capture and processing techniques that use digital computation instead of optical processes. Computational photography can improve the capabilities of a camera, or introduce features that were not possible at all with film based photography, or reduce the cost or size of camera elements. Examples of computational photography include in-camera computation of digital panoramas, high-dynamic-range images, and light field cameras. Light field cameras use novel optical elements to capture three dimensional scene information which can then be used to produce 3D images, enhanced depth-of-field, and selective de-focusing. Enhanced depth-of-field reduces the need for mechanical focusing systems. All of these features use computational imaging techniques.

In physics, the wavefront of a time-varying wave field is the set (locus) of all points having the same phase. The term is generally meaningful only for fields that, at each point, vary sinusoidally in time with a single temporal frequency.

A demosaicing algorithm is a digital image process used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA). It is also known as CFA interpolation or color reconstruction.

<span class="mw-page-title-main">Hyperspectral imaging</span> Multi-wavelength imaging method

Hyperspectral imaging collects and processes information from across the electromagnetic spectrum. The goal of hyperspectral imaging is to obtain the spectrum for each pixel in the image of a scene, with the purpose of finding objects, identifying materials, or detecting processes. There are three general types of spectral imagers. There are push broom scanners and the related whisk broom scanners, which read images over time, band sequential scanners, which acquire images of an area at different wavelengths, and snapshot hyperspectral imagers, which uses a staring array to generate an image in an instant.

An imaging spectrometer is an instrument used in hyperspectral imaging and imaging spectroscopy to acquire a spectrally-resolved image of an object or scene, often referred to as a datacube due to the three-dimensional representation of the data. Two axes of the image correspond to vertical and horizontal distance and the third to wavelength. The principle of operation is the same as that of the simple spectrometer, but special care is taken to avoid optical aberrations for better image quality.

Terahertz tomography is a class of tomography where sectional imaging is done by terahertz radiation. Terahertz radiation is electromagnetic radiation with a frequency between 0.1 and 10 THz; it falls between radio waves and light waves on the spectrum; it encompasses portions of the millimeter waves and infrared wavelengths. Because of its high frequency and short wavelength, terahertz wave has a high signal-to-noise ratio in the time domain spectrum. Tomography using terahertz radiation can image samples that are opaque in the visible and near-infrared regions of the spectrum. Terahertz wave three-dimensional (3D) imaging technology has developed rapidly since its first successful application in 1997, and a series of new 3D imaging technologies have been proposed successively.

Coded apertures or coded-aperture masks are grids, gratings, or other patterns of materials opaque to various wavelengths of electromagnetic radiation. The wavelengths are usually high-energy radiation such as X-rays and gamma rays. A coded "shadow" is cast upon a plane by blocking radiation in a known pattern. The properties of the original radiation sources can then be mathematically reconstructed from this shadow. Coded apertures are used in X- and gamma ray imaging systems, because these high-energy rays cannot be focused with lenses or mirrors that work for visible light.

Computer-generated holography (CGH) is the method of digitally generating holographic interference patterns. A holographic image can be generated e.g., by digitally computing a holographic interference pattern and printing it onto a mask or film for subsequent illumination by suitable coherent light source.

Compressed sensing is a signal processing technique for efficiently acquiring and reconstructing a signal, by finding solutions to underdetermined linear systems. This is based on the principle that, through optimization, the sparsity of a signal can be exploited to recover it from far fewer samples than required by the Nyquist–Shannon sampling theorem. There are two conditions under which recovery is possible. The first one is sparsity, which requires the signal to be sparse in some domain. The second one is incoherence, which is applied through the isometric property, which is sufficient for sparse signals.

In optical astronomy, interferometry is used to combine signals from two or more telescopes to obtain measurements with higher resolution than could be obtained with either telescopes individually. This technique is the basis for astronomical interferometer arrays, which can make measurements of very small astronomical objects if the telescopes are spread out over a wide area. If a large number of telescopes are used a picture can be produced which has resolution similar to a single telescope with the diameter of the combined spread of telescopes. These include radio telescope arrays such as VLA, VLBI, SMA, LOFAR and SKA, and more recently astronomical optical interferometer arrays such as COAST, NPOI and IOTA, resulting in the highest resolution optical images ever achieved in astronomy. The VLT Interferometer is expected to produce its first images using aperture synthesis soon, followed by other interferometers such as the CHARA array and the Magdalena Ridge Observatory Interferometer which may consist of up to 10 optical telescopes. If outrigger telescopes are built at the Keck Interferometer, it will also become capable of interferometric imaging.

Ptychography is a computational method of microscopic imaging. It generates images by processing many coherent interference patterns that have been scattered from an object of interest. Its defining characteristic is translational invariance, which means that the interference patterns are generated by one constant function moving laterally by a known amount with respect to another constant function. The interference patterns occur some distance away from these two components, so that the scattered waves spread out and "fold" into one another as shown in the figure.

Snapshot hyperspectral imaging is a method for capturing hyperspectral images during a single integration time of a detector array. No scanning is involved with this method, in contrast to push broom and whisk broom scanning techniques. The lack of moving parts means that motion artifacts should be avoided. This instrument typically features detector arrays with a high number of pixels.

The computed tomography imaging spectrometer (CTIS) is a snapshot imaging spectrometer which can produce in fine the three-dimensional hyperspectral datacube of a scene.

Preclinical or small-animal Single Photon Emission Computed Tomography (SPECT) is a radionuclide based molecular imaging modality for small laboratory animals. Although SPECT is a well-established imaging technique that is already for decades in use for clinical application, the limited resolution of clinical SPECT (~10 mm) stimulated the development of dedicated small animal SPECT systems with sub-mm resolution. Unlike in clinics, preclinical SPECT outperforms preclinical coincidence PET in terms of resolution and, at the same time, allows to perform fast dynamic imaging of animals.

<span class="mw-page-title-main">Operation of computed tomography</span>

X-ray computed tomography operates by using an X-ray generator that rotates around the object; X-ray detectors are positioned on the opposite side of the circle from the X-ray source.

References

↑ CITRIS (2017-03-08), Computational Microscopy , retrieved 2017-09-04
↑ March, G. (20 March 2012). "How to see around corners". Nature News. doi:10.1038/nature.2012.10258. S2CID 171886549.
↑ Ju, Yun Hye; Lee, Geewon; Lee, Ji Won; Hong, Seung Baek; Suh, Young Ju; Jeong, Yeon Joo (8 August 2017). "Ultra-low-dose lung screening CT with model-based iterative reconstruction: an assessment of image quality and lesion conspicuity". Acta Radiologica. 59 (5): 553–559. doi:10.1177/0284185117726099. PMID 28786301. S2CID 4428506.
↑ "Plenary talk (Peyman Milanfar)" (PDF).
↑ Hagen, Nathan (13 June 2012). "Snapshot advantage: a review of the light collection improvement for parallel high-dimensional measurement systems" (PDF). Optical Engineering. 51 (11): 111702. Bibcode:2012OptEn..51k1702H. doi:10.1117/1.OE.51.11.111702. PMC 3393130 . PMID 22791926.
↑ Hagen, Nathan; Kudenov, Michael W. (23 September 2013). "Review of snapshot spectral imaging technologies" (PDF). Optical Engineering. 52 (9): 090901. Bibcode:2013OptEn..52i0901H. doi: 10.1117/1.OE.52.9.090901 .
↑ Arce, Gonzalo R.; Rueda, Hoover; Correa, Claudia V.; Ramirez, Ana; Arguello, Henry (2017-02-15). Snapshot Compressive Multispectral Cameras. Wiley Encyclopedia of Electrical and Electronics Engineering. pp. 1–22. doi:10.1002/047134608X.W8345. ISBN 9780471346081.
↑ Wagadarikar, Ashwin; John, Renu; Willett, Rebecca; Brady, David (8 February 2008). "Single disperser design for coded aperture snapshot spectral imaging". Applied Optics. 47 (10): B44-51. Bibcode:2008ApOpt..47B..44W. doi:10.1364/AO.47.000B44. PMID 18382550.