Demosaicing

Last updated

Demosaicing (or de-mosaicing, demosaicking), also known as color reconstruction, is a digital image processing algorithm used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA) such as a Bayer filter. It is also known as CFA interpolation or debayering.

Contents

Most modern digital cameras acquire images using a single image sensor overlaid with a CFA, so demosaicing is part of the processing pipeline required to render these images into a viewable format.

Many modern digital cameras can save images in a raw format allowing the user to demosaic them using software, rather than using the camera's built-in firmware.

Goal

The aim of a demosaicing algorithm is to reconstruct a full color image (i.e. a full set of color triples) from the spatially undersampled color channels output from the CFA. The algorithm should have the following traits:

Background: color filter array

The Bayer arrangement of color filters on the pixel array of an image sensor. Each two-by-two cell contains two green, one blue, and one red filter. Bayer pattern on sensor.svg
The Bayer arrangement of color filters on the pixel array of an image sensor. Each two-by-two cell contains two green, one blue, and one red filter.

A color filter array is a mosaic of color filters in front of the image sensor. Commercially, the most commonly used CFA configuration is the Bayer filter illustrated here. This has alternating red (R) and green (G) filters for odd rows and alternating green (G) and blue (B) filters for even rows. There are twice as many green filters as red or blue ones, catering to the human eye's higher sensitivity to green light.

Since the color subsampling of a CFA by its nature results in aliasing, an optical anti-aliasing filter is typically placed in the optical path between the image sensor and the lens to reduce the false color artifacts (chromatic aliases) introduced by interpolation. [1]

Since each pixel of the sensor is behind a color filter, the output is an array of pixel values, each indicating a raw intensity of one of the three filter colors. Thus, an algorithm is needed to estimate for each pixel the color levels for all color components, rather than a single component.

Illustration

To reconstruct a full color image from the data collected by the color filtering array, a form of interpolation is needed to fill in the blanks. The mathematics here is subject to individual implementation, and is called demosaicing.

In this example, we use Adobe Photoshop's bicubic interpolation to simulate the circuitry of a Bayer filter device such as a digital camera.

The image below simulates the output from a Bayer filtered image sensor; each pixel has only a red, green or blue component. The corresponding original image is shown alongside the demosaiced reconstruction at the end of this section.

Shades framed bayer.png
Bayer filter samples
Shades bayer r.png Shades bayer g.png Shades bayer b.png
RedGreenBlue

A digital camera typically has means to reconstruct a whole RGB image using the above information. The resulting image could be something like this:

Shades framed aa.jpg Shades framed dc.jpg
OriginalReconstructed

The reconstructed image is typically accurate in uniform-colored areas, but has a loss of resolution (detail and sharpness) and has edge artifacts (for example, the edges of letters have visible color fringes and some roughness).

Algorithms

Simple interpolation

These algorithms are examples of multivariate interpolation on a uniform grid, using relatively straightforward mathematical operations on nearby instances of the same color component. The simplest method is nearest-neighbor interpolation which simply copies an adjacent pixel of the same color channel. It is unsuitable for any application where quality matters, but can be useful for generating previews given limited computational resources. Another simple method is bilinear interpolation, whereby the red value of a non-red pixel is computed as the average of the two or four adjacent red pixels, and similarly for blue and green. More complex methods that interpolate independently within each color plane include bicubic interpolation, spline interpolation, and Lanczos resampling.

Although these methods can obtain good results in homogeneous image regions, they are prone to severe demosaicing artifacts in regions with edges and details when used with pure-color CFAs. [2] However, linear interpolation can obtain very good results when combined with a spatio-spectral (panchromatic) CFA. [3] One could exploit simple formation models of images for demosaicing. In natural images within the same segment, the ratio of colors should be preserved. This fact was exploited in an image sensitive interpolation for demosaicing. [4]

Pixel correlation within an image

More sophisticated demosaicing algorithms exploit the spatial and/or spectral correlation of pixels within a color image. [5] Spatial correlation is the tendency of pixels to assume similar color values within a small homogeneous region of an image. Spectral correlation is the dependency between the pixel values of different color planes in a small image region.

These algorithms include:

Video super-resolution/demosaicing

It has been shown that super-resolution and demosaicing are two faces of the same problem and it is reasonable to address them in a unified context. [10] Note that both these problems face the aliasing issue. Therefore, especially in the case of video (multi-frame) reconstruction, a joint super-resolution and demosaicing approach provides the optimal solution.

Trade-offs

Some methods may produce better results for natural scenes, and some for printed material, for instance. This reflects the inherent problem of estimating pixels that are not definitively known. Naturally, there is also the ubiquitous trade-off of speed versus quality of estimation.

Use in computer image processing software

When one has access to the raw image data from a digital camera, one can use computer software with a variety of different demosaicing algorithms instead of being limited to the one built into the camera. A few raw development programs, such as RawTherapee and darktable, give the user an option to choose which algorithm should be used. Most programs, however, are coded to use one particular method. The differences in rendering the finest detail (and grain texture) that come from the choice of demosaicing algorithm are among the main differences between various raw developers; often photographers will prefer a particular program for aesthetic reasons related to this effect.

The color artifacts due to demosaicing provide important clues for identifying photo forgeries. [11]

See also

Related Research Articles

<span class="mw-page-title-main">Digital camera</span> Camera that captures photographs or video in digital format

A digital camera, also called a digicam, is a camera that captures photographs in digital memory. Most cameras produced today are digital, largely replacing those that capture images on photographic film or film stock. Digital cameras are now widely incorporated into mobile devices like smartphones with the same or more capabilities and features of dedicated cameras. High-end, high-definition dedicated cameras are still commonly used by professionals and those who desire to take higher-quality photographs.

In digital signal processing, spatial anti-aliasing is a technique for minimizing the distortion artifacts (aliasing) when representing a high-resolution image at a lower resolution. Anti-aliasing is used in digital photography, computer graphics, digital audio, and many other applications.

The Foveon X3 sensor is a digital camera image sensor designed by Foveon, Inc., and manufactured by Dongbu Electronics. It uses an array of photosites that consist of three vertically stacked photodiodes. Each of the three stacked photodiodes has a different spectral sensitivity, allowing it to respond differently to different wavelengths. The signals from the three photodiodes are then processed as additive color data that are transformed to a standard RGB color space. In the late 1970s, a similar color sensor having three stacked photo detectors at each pixel location, with different spectral responses due to the differential absorption of light by the semiconductor, had been developed and patented Kodak.

<span class="mw-page-title-main">Bayer filter</span> Color filter array

A Bayer filter mosaic is a color filter array (CFA) for arranging RGB color filters on a square grid of photosensors. Its particular arrangement of color filters is used in most single-chip digital image sensors used in digital cameras, and camcorders to create a color image. The filter pattern is half green, one quarter red and one quarter blue, hence is also called BGGR, RGBG, GRBG, or RGGB.

<span class="mw-page-title-main">Color balance</span> Adjustment of color intensities in photography

In photography and image processing, color balance is the global adjustment of the intensities of the colors. An important goal of this adjustment is to render specific colors – particularly neutral colors like white or grey – correctly. Hence, the general method is sometimes called gray balance, neutral balance, or white balance. Color balance changes the overall mixture of colors in an image and is used for color correction. Generalized versions of color balance are used to correct colors other than neutrals or to deliberately change them for effect. White balance is one of the most common kinds of balancing, and is when colors are adjusted to make a white object appear white and not a shade of any other colour.

<span class="mw-page-title-main">Image scaling</span> Changing the resolution of a digital image

In computer graphics and digital imaging, imagescaling refers to the resizing of a digital image. In video technology, the magnification of digital material is known as upscaling or resolution enhancement.

A camera raw image file contains unprocessed or minimally processed data from the image sensor of either a digital camera, a motion picture film scanner, or other image scanner. Raw files are so named because they are not yet processed, and contain large amounts of potentially redundant data. Normally, the image is processed by a raw converter, in a wide-gamut internal color space where precise adjustments can be made before conversion to a viewable file format such as JPEG or PNG for storage, printing, or further manipulation. There are dozens of raw formats in use by different manufacturers of digital image capture equipment.

<span class="mw-page-title-main">Three-CCD camera</span> Camera whose imaging system uses three separate charge-coupled devices

A three-CCD (3CCD) camera is a camera whose imaging system uses three separate charge-coupled devices (CCDs), each one receiving filtered red, green, or blue color ranges. Light coming in from the lens is split by a beam-splitter prism into three beams, which are then filtered to produce colored light in three color ranges or "bands". The system is employed by high quality still cameras, telecine systems, professional video cameras and some prosumer video cameras.

<span class="mw-page-title-main">Image sensor</span> Device that converts images into electronic signals

An image sensor or imager is a sensor that detects and conveys information used to form an image. It does so by converting the variable attenuation of light waves into signals, small bursts of current that convey the information. The waves can be light or other electromagnetic radiation. Image sensors are used in electronic imaging devices of both analog and digital types, which include digital cameras, camera modules, camera phones, optical mouse devices, medical imaging equipment, night vision equipment such as thermal imaging devices, radar, sonar, and others. As technology changes, electronic and digital imaging tends to replace chemical and analog imaging.

<span class="mw-page-title-main">CYGM filter</span>

In digital photography, the CYGM filter is an alternative color filter array to the Bayer filter (GRGB). It similarly uses a mosaic of pixel filters, of cyan, yellow, green and magenta, and so also requires demosaicing to produce a full-color image.

<span class="mw-page-title-main">Color filter array</span>

In digital imaging, a color filter array (CFA), or color filter mosaic (CFM), is a mosaic of tiny color filters placed over the pixel sensors of an image sensor to capture color information.

<span class="mw-page-title-main">Image processor</span> Specialized digital signal processor used for image processing

An image processor, also known as an image processing engine, image processing unit (IPU), or image signal processor (ISP), is a type of media processor or specialized digital signal processor (DSP) used for image processing, in digital cameras or other devices. Image processors often employ parallel computing even with SIMD or MIMD technologies to increase speed and efficiency. The digital image processing engine can perform a range of tasks. To increase the system integration on embedded devices, often it is a system on a chip with multi-core processor architecture.

Image quality can refer to the level of accuracy with which different imaging systems capture, process, store, compress, transmit and display the signals that form an image. Another definition refers to image quality as "the weighted combination of all of the visually significant attributes of an image". The difference between the two definitions is that one focuses on the characteristics of signal processing in different imaging systems and the latter on the perceptual assessments that make an image pleasant for human viewers.

<span class="mw-page-title-main">RawTherapee</span> Raw photo processing software

RawTherapee is application software for processing photographs in raw image formats, as created by many digital cameras. It comprises a subset of image editing operations specifically aimed at non-destructive post-production of raw photos and is primarily focused on improving a photographer's workflow by facilitating the handling of large numbers of images. It is notable for the advanced control it gives the user over the demosaicing and developing process. It is cross-platform, with versions for Microsoft Windows, macOS and Linux.

<span class="mw-page-title-main">Colour co-site sampling</span> System of photographic colour sensing

Colour co-site sampling is a system of photographic colour sensing, wherein 4, 16 or 36 images are collected from the sensor and merged to form a single image. Each subsequent image physically moves the sensor by exactly one pixel, in order to collect R, G and B data for each pixel, known as microscanning. This is a viable alternative to the typical Bayer filter array of pixels which returns a lower quality images with interpolated pixel colours.

<span class="mw-page-title-main">Fujifilm X-Trans sensor</span> Type of photosensor

The Fujifilm X-Trans is a sensor developed by Fujifilm and used in its Fujifilm X series cameras. Unlike most sensors featuring a conventional Bayer filter array, X-Trans sensors have a unique 6 by 6 pattern of photosites. Fujifilm claims that this layout can minimise moiré effects, and in turn increase resolution by eliminating the need for a low-pass filter.

JPEG XS is an interoperable, visually lossless, low-latency and lightweight image and video coding system used in professional applications. Applications of the standard include streaming high quality content for virtual reality, drones, autonomous vehicles using cameras, gaming, and broadcasting. It was the first ISO codec ever designed for this specific purpose. JPEG XS, built on core technology from both intoPIX and Fraunhofer IIS, is formally standardized as ISO/IEC 21122 by the Joint Photographic Experts Group with the first edition published in 2019. Although not official, the XS acronym was chosen to highlight the eXtra Small and eXtra Speed characteristics of the codec. Today, the JPEG committee is still actively working on further improvements to XS, with the second edition scheduled for publication and initial efforts being launched towards a third edition.

A pixel format refers to the format in which the image data output by a digital camera is represented. In comparison to the raw pixel information captured by the image sensor, the output pixels could be formatted differently based on the active pixel format. For several digital cameras, this format is a user-configurable feature; the available pixel formats on a particular camera depends on the type and model of the camera.

<span class="mw-page-title-main">Albert Brault</span> American chemist and inventor

Albert Thomas Brault is an American chemist who invented the fabrication process used for the first integral color image sensors. The curator of the Technology Collection at the George Eastman Museum, Todd Gustavson, has stated that "the color sensor technology developed by Albert Brault has revolutionized all forms of color photography. These color sensors are now ubiquitous in products such as smart phone cameras, digital cameras and camcorders, digital cinema cameras, medical cameras, automobile cameras, and drones".

<span class="mw-page-title-main">Peter L. P. Dillon</span> American physicist

Peter L. P. Dillon is an American physicist, and the inventor of integral color image sensors and single-chip color video cameras. The curator of the Technology Collection at the George Eastman Museum, Todd Gustavson, has stated that "the color sensor technology developed by Peter Dillon has revolutionized all forms of color photography. These color sensors are now ubiquitous in products such as smart phone cameras, digital cameras and camcorders, digital cinema cameras, medical cameras, automobile cameras, and drones". Dillon joined Kodak Research Labs in 1959 and retired from Kodak in 1991. He lives in Pittsford, New York.

References

  1. Adrian Davies; Phil Fennessy (2001). Digital imaging for photographers (Fourth ed.). Focal Press. ISBN   978-0-240-51590-8.
  2. Lanlan Chang; Yap-Peng Tan (2006). "Hybrid color filter array demosaicking for effective artifact suppression" (PDF). Journal of Electronic Imaging. 15: 2. Bibcode:2006JEI....15a3003C. doi:10.1117/1.2183325. Archived from the original (PDF) on 2009-12-29.
  3. Hirakawa, K., & Wolfe, P. J. (2007, September). Color Filter Array Design for Enhanced Image Fidelity. In 2007 IEEE International Conference on Image Processing (Vol. 2, pp. II-81). IEEE.
  4. Kimmel, R. (1999). Demosaicing: Image reconstruction from color CCD samples. IEEE Transactions on image processing, 8(9), 1221-1228.
  5. Lanlan Chang; Yap-Peng Tan (2006). "Hybrid color filter array demosaicking for effective artifact suppression" (PDF). Journal of Electronic Imaging. 15: 013003. Bibcode:2006JEI....15a3003C. doi:10.1117/1.2183325. Archived from the original (PDF) on 2009-12-29.
  6. Ting Chen. "Interpolation using a Threshold-based variable number of gradients". Archived from the original on 2012-04-22.
  7. Chuan-kai Lin, Portland State University (2004). "Pixel Grouping for Color Filter Array Demosaicing". Archived from the original on 2016-09-23.
  8. Kiego Hirakawa; Thomas W. Parks (2005). "Adaptive homogeneity-directed demosaicing algorithm" (PDF). IEEE Transactions on Image Processing. 14 (3): 360–369. Bibcode:2005ITIP...14..360H. doi:10.1109/TIP.2004.838691. PMID   15762333. S2CID   37217924.
  9. Decoding raw digital photos in Linux Archived 2016-10-19 at the Wayback Machine , Dave Coffin.
  10. Sina Farsiu; Michael Elad; Peyman Milanfar (2006). "Multi-Frame Demosaicing and Super-Resolution of Color Images" (PDF). IEEE Transactions on Image Processing. 15 (1): 141–159. Bibcode:2006ITIP...15..141F. CiteSeerX   10.1.1.132.7607 . doi:10.1109/TIP.2005.860336. PMID   16435545. S2CID   2989394.
  11. YiZhen Huang; YangJing Long (2008). "Demosaicking recognition with applications in digital photo authentication based on a quadratic pixel correlation model" (PDF). Proc. IEEE Conference on Computer Vision and Pattern Recognition: 1–8. Archived from the original (PDF) on 2010-06-17.