Demosaicing (or de-mosaicing, demosaicking), also known as color reconstruction, is a digital image processing algorithm used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA) such as a Bayer filter. It is also known as CFA interpolation or debayering.
Most modern digital cameras acquire images using a single image sensor overlaid with a CFA, so demosaicing is part of the processing pipeline required to render these images into a viewable format.
Many modern digital cameras can save images in a raw format allowing the user to demosaic them using software, rather than using the camera's built-in firmware.
The aim of a demosaicing algorithm is to reconstruct a full color image (i.e. a full set of color triples) from the spatially undersampled color channels output from the CFA. The algorithm should have the following traits:
A color filter array is a mosaic of color filters in front of the image sensor. Commercially, the most commonly used CFA configuration is the Bayer filter illustrated here. This has alternating red (R) and green (G) filters for odd rows and alternating green (G) and blue (B) filters for even rows. There are twice as many green filters as red or blue ones, catering to the human eye's higher sensitivity to green light.
Since the color subsampling of a CFA by its nature results in aliasing, an optical anti-aliasing filter is typically placed in the optical path between the image sensor and the lens to reduce the false color artifacts (chromatic aliases) introduced by interpolation. [1]
Since each pixel of the sensor is behind a color filter, the output is an array of pixel values, each indicating a raw intensity of one of the three filter colors. Thus, an algorithm is needed to estimate for each pixel the color levels for all color components, rather than a single component.
To reconstruct a full color image from the data collected by the color filtering array, a form of interpolation is needed to fill in the blanks. The mathematics here is subject to individual implementation, and is called demosaicing.
In this example, we use Adobe Photoshop's bicubic interpolation to simulate the circuitry of a Bayer filter device such as a digital camera.
The image below simulates the output from a Bayer filtered image sensor; each pixel has only a red, green or blue component. The corresponding original image is shown alongside the demosaiced reconstruction at the end of this section.
Bayer filter samples | ||
Red | Green | Blue |
A digital camera typically has means to reconstruct a whole RGB image using the above information. The resulting image could be something like this:
Original | Reconstructed |
The reconstructed image is typically accurate in uniform-colored areas, but has a loss of resolution (detail and sharpness) and has edge artifacts (for example, the edges of letters have visible color fringes and some roughness).
These algorithms are examples of multivariate interpolation on a uniform grid, using relatively straightforward mathematical operations on nearby instances of the same color component. The simplest method is nearest-neighbor interpolation which simply copies an adjacent pixel of the same color channel. It is unsuitable for any application where quality matters, but can be useful for generating previews given limited computational resources. Another simple method is bilinear interpolation, whereby the red value of a non-red pixel is computed as the average of the two or four adjacent red pixels, and similarly for blue and green. More complex methods that interpolate independently within each color plane include bicubic interpolation, spline interpolation, and Lanczos resampling.
Although these methods can obtain good results in homogeneous image regions, they are prone to severe demosaicing artifacts in regions with edges and details when used with pure-color CFAs. [2] However, linear interpolation can obtain very good results when combined with a spatio-spectral (panchromatic) CFA. [3] One could exploit simple formation models of images for demosaicing. In natural images within the same segment, the ratio of colors should be preserved. This fact was exploited in an image sensitive interpolation for demosaicing. [4]
More sophisticated demosaicing algorithms exploit the spatial and/or spectral correlation of pixels within a color image. [5] Spatial correlation is the tendency of pixels to assume similar color values within a small homogeneous region of an image. Spectral correlation is the dependency between the pixel values of different color planes in a small image region.
These algorithms include:
It has been shown that super-resolution and demosaicing are two faces of the same problem and it is reasonable to address them in a unified context. [10] Note that both these problems face the aliasing issue. Therefore, especially in the case of video (multi-frame) reconstruction, a joint super-resolution and demosaicing approach provides the optimal solution.
Some methods may produce better results for natural scenes, and some for printed material, for instance. This reflects the inherent problem of estimating pixels that are not definitively known. Naturally, there is also the ubiquitous trade-off of speed versus quality of estimation.
When one has access to the raw image data from a digital camera, one can use computer software with a variety of different demosaicing algorithms instead of being limited to the one built into the camera. A few raw development programs, such as RawTherapee and darktable, give the user an option to choose which algorithm should be used. Most programs, however, are coded to use one particular method. The differences in rendering the finest detail (and grain texture) that come from the choice of demosaicing algorithm are among the main differences between various raw developers; often photographers will prefer a particular program for aesthetic reasons related to this effect.
The color artifacts due to demosaicing provide important clues for identifying photo forgeries. [11]
A digital camera, also called a digicam, is a camera that captures photographs in digital memory. Most cameras produced today are digital, largely replacing those that capture images on photographic film or film stock. Digital cameras are now widely incorporated into mobile devices like smartphones with the same or more capabilities and features of dedicated cameras. High-end, high-definition dedicated cameras are still commonly used by professionals and those who desire to take higher-quality photographs.
In digital signal processing, spatial anti-aliasing is a technique for minimizing the distortion artifacts (aliasing) when representing a high-resolution image at a lower resolution. Anti-aliasing is used in digital photography, computer graphics, digital audio, and many other applications.
The Foveon X3 sensor is a digital camera image sensor designed by Foveon, Inc., and manufactured by Dongbu Electronics. It uses an array of photosites that consist of three vertically stacked photodiodes. Each of the three stacked photodiodes has a different spectral sensitivity, allowing it to respond differently to different wavelengths. The signals from the three photodiodes are then processed as additive color data that are transformed to a standard RGB color space. In the late 1970s, a similar color sensor having three stacked photo detectors at each pixel location, with different spectral responses due to the differential absorption of light by the semiconductor, had been developed and patented Kodak.
A Bayer filter mosaic is a color filter array (CFA) for arranging RGB color filters on a square grid of photosensors. Its particular arrangement of color filters is used in most single-chip digital image sensors used in digital cameras, and camcorders to create a color image. The filter pattern is half green, one quarter red and one quarter blue, hence is also called BGGR, RGBG, GRBG, or RGGB.
In photography and image processing, color balance is the global adjustment of the intensities of the colors. An important goal of this adjustment is to render specific colors – particularly neutral colors like white or grey – correctly. Hence, the general method is sometimes called gray balance, neutral balance, or white balance. Color balance changes the overall mixture of colors in an image and is used for color correction. Generalized versions of color balance are used to correct colors other than neutrals or to deliberately change them for effect. White balance is one of the most common kinds of balancing, and is when colors are adjusted to make a white object appear white and not a shade of any other colour.
In computer graphics and digital imaging, imagescaling refers to the resizing of a digital image. In video technology, the magnification of digital material is known as upscaling or resolution enhancement.
A camera raw image file contains unprocessed or minimally processed data from the image sensor of either a digital camera, a motion picture film scanner, or other image scanner. Raw files are so named because they are not yet processed, and contain large amounts of potentially redundant data. Normally, the image is processed by a raw converter, in a wide-gamut internal color space where precise adjustments can be made before conversion to a viewable file format such as JPEG or PNG for storage, printing, or further manipulation. There are dozens of raw formats in use by different manufacturers of digital image capture equipment.
A three-CCD (3CCD) camera is a camera whose imaging system uses three separate charge-coupled devices (CCDs), each one receiving filtered red, green, or blue color ranges. Light coming in from the lens is split by a beam-splitter prism into three beams, which are then filtered to produce colored light in three color ranges or "bands". The system is employed by high quality still cameras, telecine systems, professional video cameras and some prosumer video cameras.
An image sensor or imager is a sensor that detects and conveys information used to form an image. It does so by converting the variable attenuation of light waves into signals, small bursts of current that convey the information. The waves can be light or other electromagnetic radiation. Image sensors are used in electronic imaging devices of both analog and digital types, which include digital cameras, camera modules, camera phones, optical mouse devices, medical imaging equipment, night vision equipment such as thermal imaging devices, radar, sonar, and others. As technology changes, electronic and digital imaging tends to replace chemical and analog imaging.
In digital photography, the CYGM filter is an alternative color filter array to the Bayer filter (GRGB). It similarly uses a mosaic of pixel filters, of cyan, yellow, green and magenta, and so also requires demosaicing to produce a full-color image.
In digital imaging, a color filter array (CFA), or color filter mosaic (CFM), is a mosaic of tiny color filters placed over the pixel sensors of an image sensor to capture color information.
An image processor, also known as an image processing engine, image processing unit (IPU), or image signal processor (ISP), is a type of media processor or specialized digital signal processor (DSP) used for image processing, in digital cameras or other devices. Image processors often employ parallel computing even with SIMD or MIMD technologies to increase speed and efficiency. The digital image processing engine can perform a range of tasks. To increase the system integration on embedded devices, often it is a system on a chip with multi-core processor architecture.
Image quality can refer to the level of accuracy with which different imaging systems capture, process, store, compress, transmit and display the signals that form an image. Another definition refers to image quality as "the weighted combination of all of the visually significant attributes of an image". The difference between the two definitions is that one focuses on the characteristics of signal processing in different imaging systems and the latter on the perceptual assessments that make an image pleasant for human viewers.
RawTherapee is application software for processing photographs in raw image formats, as created by many digital cameras. It comprises a subset of image editing operations specifically aimed at non-destructive post-production of raw photos and is primarily focused on improving a photographer's workflow by facilitating the handling of large numbers of images. It is notable for the advanced control it gives the user over the demosaicing and developing process. It is cross-platform, with versions for Microsoft Windows, macOS and Linux.
Colour co-site sampling is a system of photographic colour sensing, wherein 4, 16 or 36 images are collected from the sensor and merged to form a single image. Each subsequent image physically moves the sensor by exactly one pixel, in order to collect R, G and B data for each pixel, known as microscanning. This is a viable alternative to the typical Bayer filter array of pixels which returns a lower quality images with interpolated pixel colours.
The Fujifilm X-Trans is a sensor developed by Fujifilm and used in its Fujifilm X series cameras. Unlike most sensors featuring a conventional Bayer filter array, X-Trans sensors have a unique 6 by 6 pattern of photosites. Fujifilm claims that this layout can minimise moiré effects, and in turn increase resolution by eliminating the need for a low-pass filter.
JPEG XS is an interoperable, visually lossless, low-latency and lightweight image and video coding system used in professional applications. Applications of the standard include streaming high quality content for virtual reality, drones, autonomous vehicles using cameras, gaming, and broadcasting. It was the first ISO codec ever designed for this specific purpose. JPEG XS, built on core technology from both intoPIX and Fraunhofer IIS, is formally standardized as ISO/IEC 21122 by the Joint Photographic Experts Group with the first edition published in 2019. Although not official, the XS acronym was chosen to highlight the eXtra Small and eXtra Speed characteristics of the codec. Today, the JPEG committee is still actively working on further improvements to XS, with the second edition scheduled for publication and initial efforts being launched towards a third edition.
A pixel format refers to the format in which the image data output by a digital camera is represented. In comparison to the raw pixel information captured by the image sensor, the output pixels could be formatted differently based on the active pixel format. For several digital cameras, this format is a user-configurable feature; the available pixel formats on a particular camera depends on the type and model of the camera.
Albert Thomas Brault is an American chemist who invented the fabrication process used for the first integral color image sensors. The curator of the Technology Collection at the George Eastman Museum, Todd Gustavson, has stated that "the color sensor technology developed by Albert Brault has revolutionized all forms of color photography. These color sensors are now ubiquitous in products such as smart phone cameras, digital cameras and camcorders, digital cinema cameras, medical cameras, automobile cameras, and drones".
Peter L. P. Dillon is an American physicist, and the inventor of integral color image sensors and single-chip color video cameras. The curator of the Technology Collection at the George Eastman Museum, Todd Gustavson, has stated that "the color sensor technology developed by Peter Dillon has revolutionized all forms of color photography. These color sensors are now ubiquitous in products such as smart phone cameras, digital cameras and camcorders, digital cinema cameras, medical cameras, automobile cameras, and drones". Dillon joined Kodak Research Labs in 1959 and retired from Kodak in 1991. He lives in Pittsford, New York.