Bayer filter

Last updated
The Bayer arrangement of color filters on the pixel array of an image sensor Bayer pattern on sensor.svg
The Bayer arrangement of color filters on the pixel array of an image sensor
Profile/cross-section of sensor Bayer pattern on sensor profile.svg
Profile/cross-section of sensor

A Bayer filter mosaic is a color filter array (CFA) for arranging RGB color filters on a square grid of photosensors. Its particular arrangement of color filters is used in most single-chip digital image sensors used in digital cameras, and camcorders to create a color image. The filter pattern is half green, one quarter red and one quarter blue, hence is also called BGGR, RGBG, [1] [2] GRBG, [3] or RGGB. [4]

Contents

It is named after its inventor, Bryce Bayer of Eastman Kodak. Bayer is also known for his recursively defined matrix used in ordered dithering.

Alternatives to the Bayer filter include both various modifications of colors and arrangement and completely different technologies, such as color co-site sampling, the Foveon X3 sensor, the dichroic mirrors or a transparent diffractive-filter array. [5]

Explanation

Original scene
Output of a 120x80-pixel sensor with a Bayer filter
Output color-coded with Bayer filter colors
Reconstructed image after interpolating missing color information
Full RGB version at 120x80-pixels for comparison (e.g. as a film scan, Foveon or pixel shift image might appear) Colorful spring garden Bayer + RGB.png
  1. Original scene
  2. Output of a 120×80-pixel sensor with a Bayer filter
  3. Output color-coded with Bayer filter colors
  4. Reconstructed image after interpolating missing color information
  5. Full RGB version at 120×80-pixels for comparison (e.g. as a film scan, Foveon or pixel shift image might appear)

Bryce Bayer's patent (U.S. Patent No. 3,971,065 [6] ) in 1976 called the green photosensors luminance-sensitive elements and the red and blue ones chrominance-sensitive elements. He used twice as many green elements as red or blue to mimic the physiology of the human eye. The luminance perception of the human retina uses M and L cone cells combined, during daylight vision, which are most sensitive to green light. These elements are referred to as sensor elements, sensels , pixel sensors, or simply pixels; sample values sensed by them, after interpolation, become image pixels. At the time Bayer registered his patent, he also proposed to use a cyan-magenta-yellow combination, that is another set of opposite colors. This arrangement was impractical at the time because the necessary dyes did not exist, but is used in some new digital cameras. The big advantage of the new CMY dyes is that they have an improved light absorption characteristic; that is, their quantum efficiency is higher.

The raw output of Bayer-filter cameras is referred to as a Bayer pattern image. Since each pixel is filtered to record only one of three colors, the data from each pixel cannot fully specify each of the red, green, and blue values on its own. To obtain a full-color image, various demosaicing algorithms can be used to interpolate a set of complete red, green, and blue values for each pixel. These algorithms make use of the surrounding pixels of the corresponding colors to estimate the values for a particular pixel.

Different algorithms requiring various amounts of computing power result in varying-quality final images. This can be done in-camera, producing a JPEG or TIFF image, or outside the camera using the raw data directly from the sensor. Since the processing power of the camera processor is limited, many photographers prefer to do these operations manually on a personal computer. The cheaper the camera, the fewer opportunities to influence these functions. In professional cameras, image correction functions are completely absent, or they can be turned off. Recording in Raw-format provides the ability to manually select demosaicing algorithm and control the transformation parameters, which is used not only in consumer photography but also in solving various technical and photometric problems. [7]

Demosaicing

Demosaicing can be performed in different ways. Simple methods interpolate the color value of the pixels of the same color in the neighborhood. For example, once the chip has been exposed to an image, each pixel can be read. A pixel with a green filter provides an exact measurement of the green component. The red and blue components for this pixel are obtained from the neighbors. For a green pixel, two red neighbors can be interpolated to yield the red value, also two blue pixels can be interpolated to yield the blue value.

This simple approach works well in areas with constant color or smooth gradients, but it can cause artifacts such as color bleeding in areas where there are abrupt changes in color or brightness especially noticeable along sharp edges in the image. Because of this, other demosaicing methods attempt to identify high-contrast edges and only interpolate along these edges, but not across them.

Other algorithms are based on the assumption that the color of an area in the image is relatively constant even under changing light conditions, so that the color channels are highly correlated with each other. Therefore, the green channel is interpolated at first then the red and afterwards the blue channel, so that the color ratio red-green respective blue-green are constant. There are other methods that make different assumptions about the image content and starting from this attempt to calculate the missing color values.

Artifacts

Images with small-scale detail close to the resolution limit of the digital sensor can be a problem to the demosaicing algorithm, producing a result which does not look like the model. The most frequent artifact is Moiré, which may appear as repeating patterns, color artifacts or pixels arranged in an unrealistic maze-like pattern.

False color artifact

A common and unfortunate artifact of Color Filter Array (CFA) interpolation or demosaicing is what is known and seen as false coloring. Typically this artifact manifests itself along edges, where abrupt or unnatural shifts in color occur as a result of misinterpolating across, rather than along, an edge. Various methods exist for preventing and removing this false coloring. Smooth hue transition interpolation is used during the demosaicing to prevent false colors from manifesting themselves in the final image. However, there are other algorithms that can remove false colors after demosaicing. These have the benefit of removing false coloring artifacts from the image while using a more robust demosaicing algorithm for interpolating the red and blue color planes.

Three images depicting the false color demosaicing artifact. False colour artifact.JPG
Three images depicting the false color demosaicing artifact.

Zippering artifact

The zippering artifact is another side effect of CFA demosaicing, which also occurs primarily along edges, is known as the zipper effect. Simply put, zippering is another name for edge blurring that occurs in an on/off pattern along an edge. This effect occurs when the demosaicing algorithm averages pixel values over an edge, especially in the red and blue planes, resulting in its characteristic blur. As mentioned before, the best methods for preventing this effect are the various algorithms which interpolate along, rather than across image edges. Pattern recognition interpolation, adaptive color plane interpolation, and directionally weighted interpolation all attempt to prevent zippering by interpolating along edges detected in the image.

Three images depicting the zippering artifact of CFA demosaicing Zippering artifact.JPG
Three images depicting the zippering artifact of CFA demosaicing

However, even with a theoretically perfect sensor that could capture and distinguish all colors at each photosite, Moiré and other artifacts could still appear. This is an unavoidable consequence of any system that samples an otherwise continuous signal at discrete intervals or locations. For this reason, most photographic digital sensors incorporate something called an optical low-pass filter (OLPF) or an anti-aliasing (AA) filter. This is typically a thin layer directly in front of the sensor, and works by effectively blurring any potentially problematic details that are finer than the resolution of the sensor.

Modifications

The Bayer filter is almost universal on consumer digital cameras. Alternatives include the CYGM filter (cyan, yellow, green, magenta) and RGBE filter (red, green, blue, emerald), which require similar demosaicing. The Foveon X3 sensor (which layers red, green, and blue sensors vertically rather than using a mosaic) and arrangements of three separate CCDs (one for each color) doesn't need demosaicing.

Panchromatic cells

Three new Kodak RGBW filter patterns Kodak RGBW patterns.svg
Three new Kodak RGBW filter patterns

On June 14, 2007, Eastman Kodak announced an alternative to the Bayer filter: a colour-filter pattern that increases the sensitivity to light of the image sensor in a digital camera by using some panchromatic cells that are sensitive to all wavelengths of visible light and collect a larger amount of light striking the sensor. [8] They present several patterns, but none with a repeating unit as small as the Bayer pattern's 2×2 unit.

Earlier RGBW filter pattern RGBW Bayer.svg
Earlier RGBW filter pattern

Another 2007 U.S. patent filing, by Edward T. Chang, claims a sensor where "the color filter has a pattern comprising 2×2 blocks of pixels composed of one red, one blue, one green and one transparent pixel," in a configuration intended to include infrared sensitivity for higher overall sensitivity. [9] The Kodak patent filing was earlier. [10]

Such cells have previously been used in "CMYW" (cyan, magenta, yellow, and white) [11] "RGBW" (red, green, blue, white) [12] sensors, but Kodak has not compared the new filter pattern to them yet.

Fujifilm "EXR" color filter array

EXR sensor EXR sensor.svg
EXR sensor

Fujifilm's EXR color filter array are manufactured in both CCD (SuperCCD) and CMOS (BSI CMOS). As with the SuperCCD, the filter itself is rotated 45 degrees. Unlike conventional Bayer filter designs, there are always two adjacent photosites detecting the same color. The main reason for this type of array is to contribute to pixel "binning", where two adjacent photosites can be merged, making the sensor itself more "sensitive" to light. Another reason is for the sensor to record two different exposures, which is then merged to produce an image with greater dynamic range. The underlying circuitry has two read-out channels that take their information from alternate rows of the sensor. The result is that it can act like two interleaved sensors, with different exposure times for each half of the photosites. Half of the photosites can be intentionally underexposed so that they fully capture the brighter areas of the scene. This retained highlight information can then be blended in with the output from the other half of the sensor that is recording a 'full' exposure, again making use of the close spacing of similarly colored photosites.

Fujifilm "X-Trans" filter

The repeating 6x6 grid used in the x-trans sensor Xtranscolourfilter.svg
The repeating 6×6 grid used in the x-trans sensor

The Fujifilm X-Trans CMOS sensor used in many Fujifilm X-series cameras is claimed [13] to provide better resistance to color moiré than the Bayer filter, and as such they can be made without an anti-aliasing filter. This in turn allows cameras using the sensor to achieve a higher resolution with the same megapixel count. Also, the new design is claimed to reduce the incidence of false colors, by having red, blue and green pixels in each line. The arrangement of these pixels is also said to provide grain more like film.

One of main drawbacks for custom patterns is that they may lack full support in third party raw processing software like Adobe Photoshop Lightroom [14] where adding improvements took multiple years. [15]

Quad Bayer

Sony introduced Quad Bayer color filter array, which first featured in the iPhone 6's front camera released in 2014. Quad Bayer is similar to Bayer filter, however adjacent 2x2 pixels are the same color, the 4x4 pattern features 4x blue, 4x red, and 8x green. [16] For darker scenes, signal processing can combine data from each 2x2 group, essentially like a larger pixel. For brighter scenes, signal processing can convert the Quad Bayer into a conventional Bayer filter to achieve higher resolution. [17] The pixels in Quad Bayer can be operated in long-time integration and short-time integration to achieve single shot HDR, reducing blending issues. [18] Quad Bayer is also known as Tetracell by Samsung, 4-cell by OmniVision, [17] [19] and Quad CFA (QCFA) by Qualcomm. [20]

On March 26, 2019, the Huawei P30 series were announced featuring RYYB Quad Bayer, with the 4x4 pattern featuring 4x blue, 4x red, and 8x yellow. [21]

Nonacell

On February 12, 2020, the Samsung Galaxy S20 Ultra was announced featuring Nonacell CFA. Nonacell CFA is similar to Bayer filter, however adjacent 3x3 pixels are the same color, the 6x6 pattern features 9x blue, 9x red, and 18x green. [22]

See also

Related Research Articles

<span class="mw-page-title-main">Charge-coupled device</span> Device for the movement of electrical charge

A charge-coupled device (CCD) is an integrated circuit containing an array of linked, or coupled, capacitors. Under the control of an external circuit, each capacitor can transfer its electric charge to a neighboring capacitor. CCD sensors are a major technology used in digital imaging.

<span class="mw-page-title-main">Pixel</span> Physical point in a raster image

In digital imaging, a pixel, pel, or picture element is the smallest addressable element in a raster image, or the smallest addressable element in a dot matrix display device. In most digital display devices, pixels are the smallest element that can be manipulated through software.

<span class="mw-page-title-main">RGB color model</span> Color model based on red, green, and blue

The RGB color model is an additive color model in which the red, green and blue primary colors of light are added together in various ways to reproduce a broad array of colors. The name of the model comes from the initials of the three additive primary colors, red, green, and blue.

<span class="mw-page-title-main">Digital camera</span> Camera that captures photographs or video in digital format

A digital camera, also called a digicam, is a camera that captures photographs in digital memory. Most cameras produced today are digital, largely replacing those that capture images on photographic film. Digital cameras are now widely incorporated into mobile devices like smartphones with the same or more capabilities and features of dedicated cameras. High-end, high-definition dedicated cameras are still commonly used by professionals and those who desire to take higher-quality photographs.

The Foveon X3 sensor is a digital camera image sensor designed by Foveon, Inc., and manufactured by Dongbu Electronics. It uses an array of photosites that consist of three vertically stacked photodiodes. Each of the three stacked photodiodes has a different spectral sensitivity, allowing it to respond differently to different wavelengths. The signals from the three photodiodes are then processed as additive color data that are transformed to a standard RGB color space.

<span class="mw-page-title-main">Dichroic prism</span>

A dichroic prism is a prism that splits light into two beams of differing wavelength (colour). A trichroic prism assembly combines two dichroic prisms to split an image into 3 colours, typically as red, green and blue of the RGB colour model. They are usually constructed of one or more glass prisms with dichroic optical coatings that selectively reflect or transmit light depending on the light's wavelength. That is, certain surfaces within the prism act as dichroic filters. These are used as beam splitters in many optical instruments.

<span class="mw-page-title-main">Super CCD</span>

Super CCD is a proprietary charge-coupled device that has been developed by Fujifilm since 1999. The Super CCD uses octagonal, rather than rectangular, pixels. This allows a higher horizontal and vertical resolution to be achieved than a traditional sensor of an equivalent pixel count.

<span class="mw-page-title-main">Image noise</span> Visible interference in an image

Image noise is random variation of brightness or color information in images, and is usually an aspect of electronic noise. It can be produced by the image sensor and circuitry of a scanner or digital camera. Image noise can also originate in film grain and in the unavoidable shot noise of an ideal photon detector. Image noise is an undesirable by-product of image capture that obscures the desired information. Typically the term “image noise” is used to refer to noise in 2D images, not 3D images.

<span class="mw-page-title-main">Fujifilm FinePix S7000</span> Camera model

The FinePix S7000 is a 6.3 megapixel digital camera manufactured by Fujifilm. It was announced in July 2003, and has since been discontinued. It features 6x optical zoom.

Bryce Edward Bayer was an American scientist who invented the Bayer filter pattern, which is used in most modern color digital cameras. He has been called "the maestro without whom photography as we know wouldn't have been the same."

A camera raw image file contains unprocessed or minimally processed data from the image sensor of either a digital camera, a motion picture film scanner, or other image scanner. Raw files are so named because they are not yet processed, and contain large amounts of potentially redundant data. Normally, the image is processed by a raw converter, in a wide-gamut internal color space where precise adjustments can be made before conversion to a viewable file format such as JPEG or PNG for storage, printing, or further manipulation. There are dozens of raw formats in use by different manufacturers of digital image capture equipment.

Demosaicing, also known as color reconstruction, is a digital image processing algorithm used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA) such as a Bayer filter. It is also known as CFA interpolation or debayering.

<span class="mw-page-title-main">Image sensor</span> Device that converts images into electronic signals

An image sensor or imager is a sensor that detects and conveys information used to form an image. It does so by converting the variable attenuation of light waves into signals, small bursts of current that convey the information. The waves can be light or other electromagnetic radiation. Image sensors are used in electronic imaging devices of both analog and digital types, which include digital cameras, camera modules, camera phones, optical mouse devices, medical imaging equipment, night vision equipment such as thermal imaging devices, radar, sonar, and others. As technology changes, electronic and digital imaging tends to replace chemical and analog imaging.

<span class="mw-page-title-main">CYGM filter</span>

In digital photography, the CYGM filter is an alternative color filter array to the Bayer filter (GRGB). It similarly uses a mosaic of pixel filters, of cyan, yellow, green and magenta, and so also requires demosaicing to produce a full-color image.

<span class="mw-page-title-main">Color filter array</span>

In digital imaging, a color filter array (CFA), or color filter mosaic (CFM), is a mosaic of tiny color filters placed over the pixel sensors of an image sensor to capture color information.

<span class="mw-page-title-main">RGBE filter</span>

In digital photography, the RGBE filter is an alternative color filter array to the Bayer filter (GRGB). It similarly uses a mosaic of pixel filters, of red, green, blue and "emerald", and so also requires demosaicing to produce a full-color image. It was developed by Sony and so far is used only in the ICX456 8-megapixel CCD and in the Sony Cyber-shot DSC-F828 camera.

<span class="mw-page-title-main">Image processor</span> Specialized digital signal processor used for image processing

An image processor, also known as an image processing engine, image processing unit (IPU), or image signal processor (ISP), is a type of media processor or specialized digital signal processor (DSP) used for image processing, in digital cameras or other devices. Image processors often employ parallel computing even with SIMD or MIMD technologies to increase speed and efficiency. The digital image processing engine can perform a range of tasks. To increase the system integration on embedded devices, often it is a system on a chip with multi-core processor architecture.

<span class="mw-page-title-main">Colour co-site sampling</span> System of photographic colour sensing

Colour co-site sampling is a system of photographic colour sensing, wherein 4, 16 or 36 images are collected from the sensor and merged to form a single image. Each subsequent image physically moves the sensor by exactly one pixel, in order to collect R, G and B data for each pixel, known as microscanning. This is a viable alternative to the typical Bayer filter array of pixels which returns a lower quality images with interpolated pixel colours.

<span class="mw-page-title-main">Fujifilm X-Trans sensor</span> Type of photosensor

The Fujifilm X-Trans is a sensor developed by Fujifilm and used in its Fujifilm X series cameras. Unlike most sensors featuring a conventional Bayer filter array, X-Trans sensors have a unique 6 by 6 pattern of photosites. Fujifilm claims that this layout can minimise moiré effects, and in turn increase resolution by eliminating the need for a low-pass filter.

<span class="mw-page-title-main">Peter L. P. Dillon</span> American physicist

Peter L. P. Dillon is an American physicist, and the inventor of integral color image sensors.. and single-chip color video cameras. The curator of the Technology Collection at the George Eastman Museum, Todd Gustavson, has stated that "the color sensor technology developed by Peter Dillon has revolutionized all forms of color photography. These color sensors are now ubiquitous in products such as smart phone cameras, digital cameras and camcorders, digital cinema cameras, medical cameras, automobile cameras, and drones". Dillon joined Kodak Research Labs in 1959 and retired from Kodak in 1991. He lives in Pittsford, New York.

References

Front page of Bryce Bayer's 1976 patent on the Bayer pattern filter mosaic, showing his terminology of luminance-sensitive and chrominance-sensitive elements US03971065 Bayer Front.png
Front page of Bryce Bayer's 1976 patent on the Bayer pattern filter mosaic, showing his terminology of luminance-sensitive and chrominance-sensitive elements

Notes

  1. Jeff Mather (2008). "Adding L* to RGBG". Archived from the original on 2011-07-13. Retrieved 2011-02-18.
  2. dpreview.com (2000). "Sony announce 3 new digital cameras". Archived from the original on 2011-07-21.
  3. Margaret Brown (2004). Advanced Digital Photography. Media Publishing. ISBN   0-9581888-5-8.
  4. Thomas Maschke (2004). Digitale Kameratechnik: Technik digitaler Kameras in Theorie und Praxis. Springer. ISBN   3-540-40243-8. Archived from the original on 2019-01-09. Retrieved 2016-09-23.
  5. Wang, Peng; Menon, Rajesh (29 October 2015). "Ultra-high-sensitivity color imaging via a transparent diffractive-filter array and computational optics". Optica. 2 (11): 933. Bibcode:2015Optic...2..933W. doi: 10.1364/optica.2.000933 .
  6. "Patent US3971065 - Color imaging array - Google Patents". Archived from the original on 2013-08-11. Retrieved 2013-04-23.
  7. Cheremkhin, P. A., Lesnichii, V. V. & Petrov, N. V. (2014). "Use of spectral characteristics of DSLR cameras with Bayer filter sensors". Journal of Physics: Conference Series. 536 (1): 012021. Bibcode:2014JPhCS.536a2021C. doi: 10.1088/1742-6596/536/1/012021 .{{cite journal}}: CS1 maint: multiple names: authors list (link)
  8. John Compton and John Hamilton (2007-06-14). "Color Filter Array 2.0". A Thousand Nerds: A Kodak blog. Archived from the original on 2007-07-20. Retrieved 2011-02-25.
  9. "US patent publication 20070145273 "High-sensitivity infrared color camera"". Archived from the original on 2017-02-22.
  10. "US Patent Application 20070024879 "Processing color and panchromatic pixels"". Archived from the original on 2016-12-21.
  11. L. J. d'Luna; et al. (1989). "A digital video signal post-processor for color image sensors". 1989 Proceedings of the IEEE Custom Integrated Circuits Conference. Vol. 1989. pp. 24.2/1–24.2/4. doi:10.1109/CICC.1989.56823. S2CID   61954103. A variety of CFA patterns can be used, with various arrangements of red, green, and blue (RGB) or of cyan, magenta, yellow, and white (CMYW) colors.
  12. Sugiyama, Toshinobu, US patent application 20050231618, "Image-capturing apparatus" Archived 2017-02-22 at the Wayback Machine , filed March 30, 2005
  13. "Fujifilm X-Trans sensor technology". Archived from the original on 2012-04-09. Retrieved 2012-03-15.
  14. Diallo, Amadou. "Adobe's Fujifilm X-Trans sensor processing tested". dpreview.com. Archived from the original on 21 October 2016. Retrieved 20 October 2016.
  15. "Adobe Improves X-Trans Processing in Lightroom CC Update: Promises More to Come". Thomas Fitzgerald Photography Blog. 17 June 2015. Archived from the original on 21 October 2016. Retrieved 20 October 2016.
  16. "Sony Releases Stacked CMOS Image Sensor for Smartphones with Industry's Highest 48 Effective Megapixels". Sony Global - Sony Global Headquarters. Archived from the original on 2019-09-05. Retrieved 2019-08-16.
  17. 1 2 "How Tetracell delivers crystal clear photos day and night | Samsung Semiconductor Global Website". www.samsung.com. Archived from the original on 2019-08-16. Retrieved 2019-08-16.
  18. "IMX294CJK | Sony Semiconductor Solutions". Sony Semiconductor Solutions Corporation. Archived from the original on 2019-08-16. Retrieved 2019-08-16.
  19. "Product Releases | News & Events | OmniVision". www.ovt.com. Archived from the original on 2019-08-16. Retrieved 2019-08-16.
  20. USpending 20200280659,"Quad color filter array camera sensor configurations"
  21. "Part 4: Non-Bayer CFA, Phase Detection Autofocus (PDAF) | TechInsights". techinsights.com. Archived from the original on 2019-08-16. Retrieved 2019-08-16.
  22. "Samsung's 108Mp ISOCELL Bright HM1 Delivers Brighter Ultra-High-Res Images with Industry-First Nonacell Technology". news.samsung.com. Archived from the original on 2020-02-12. Retrieved 2020-02-14.