Image processor

Last updated
Nikon EXPEED, a system on a chip including an image processor, video processor, digital signal processor (DSP) and a 32-bit microcontroller controlling the chip Nikon D90 - board 0 - Nikon Expeed EI-149-1769.jpg
Nikon EXPEED, a system on a chip including an image processor, video processor, digital signal processor (DSP) and a 32-bit microcontroller controlling the chip

An image processor, also known as an image processing engine, image processing unit (IPU), or image signal processor (ISP), is a type of media processor or specialized digital signal processor (DSP) used for image processing, in digital cameras or other devices. [1] [2] Image processors often employ parallel computing even with SIMD or MIMD technologies to increase speed and efficiency. [3] The digital image processing engine can perform a range of tasks. To increase the system integration on embedded devices, often it is a system on a chip with multi-core processor architecture.

Contents

Function

Bayer transformation

The photodiodes employed in an image sensor are color-blind by nature: they can only record shades of grey. To get color into the picture, they are covered with different color filters: red, green and blue (RGB) according to the pattern designated by the Bayer filter - named after its inventor. As each photodiode records the color information for exactly one pixel of the image, without an image processor there would be a green pixel next to each red and blue pixel.

This process, however, is quite complex, and involves a number of different operations. Its quality depends largely on the effectiveness of the algorithms applied to the raw data coming from the sensor. The mathematically manipulated data becomes the photo file recorded.

Demosaicing

As stated above, the image processor evaluates the color and brightness data of a given pixel, compares them with the data from neighboring pixels, and then uses a demosaicing algorithm to produce an appropriate color and brightness value for the pixel. The image processor also assesses the whole picture to guess at the correct distribution of contrast. By adjusting the gamma value (heightening or lowering the contrast range of an image's mid-tones), subtle tonal gradations, such as in human skin or the blue of the sky, become much more realistic.

Noise reduction

Noise is a phenomenon found in any electronic circuitry. In digital photography its effect is often visible as random spots of obviously wrong color in an otherwise smoothly-colored area. Noise increases with temperature and exposure times. When higher ISO settings are chosen the electronic signal in the image sensor is amplified, which at the same time increases the noise level, leading to a lower signal-to-noise ratio. The image processor attempts to separate the noise from the image information and to remove it. This can be quite a challenge, as the image may contain areas with fine textures which, if treated as noise, may lose some of their definition.

Image sharpening

As the color and brightness values for each pixel are interpolated some image sharpening is applied to even out any fuzziness that has occurred. To preserve the impression of depth, clarity and fine details, the image processor must sharpen edges and contours. It therefore must detect edges correctly and reproduce them smoothly and without over-sharpening.

Models

Image processor users are using industry standard products, application-specific standard products (ASSP) or even application-specific integrated circuits (ASIC) with trade names: Canon's is called DIGIC, Nikon's Expeed, Olympus' TruePic, Panasonic's Venus Engine and Sony's Bionz. Some are known to be based on the Fujitsu Milbeaut, the Texas Instruments OMAP, Panasonic MN103, Zoran Coach, Altek Sunny or Sanyo image/video processors.

ARM architecture processors with its NEON SIMD Media Processing Engines (MPE) are often used in mobile phones.

Processor brand names

Speed

With the ever-higher pixel count in image sensors, the image processor's speed becomes more critical: photographers don't want to wait for the camera's image processor to complete its job before they can carry on shooting - they don't even want to notice some processing is going on inside the camera. Therefore, image processors must be optimised to cope with more data in the same or even a shorter period of time.

Software

libcamera is a software library that supports using image signal processors for the capture of pictures.

See also

Related Research Articles

<span class="mw-page-title-main">Digital camera</span> Camera that captures photographs or video in digital format

A digital camera, also called a digicam, is a camera that captures photographs in digital memory. Most cameras produced today are digital, largely replacing those that capture images on photographic film or film stock. Digital cameras are now widely incorporated into mobile devices like smartphones with the same or more capabilities and features of dedicated cameras. High-end, high-definition dedicated cameras are still commonly used by professionals and those who desire to take higher-quality photographs.

The Foveon X3 sensor is a digital camera image sensor designed by Foveon, Inc., and manufactured by Dongbu Electronics. It uses an array of photosites that consist of three vertically stacked photodiodes. Each of the three stacked photodiodes has a different spectral sensitivity, allowing it to respond differently to different wavelengths. The signals from the three photodiodes are then processed as additive color data that are transformed to a standard RGB color space. In the late 1970s, a similar color sensor having three stacked photo detectors at each pixel location, with different spectral responses due to the differential absorption of light by the semiconductor, had been developed and patented by Kodak.

<span class="mw-page-title-main">Bayer filter</span> Color filter array

A Bayer filter mosaic is a color filter array (CFA) for arranging RGB color filters on a square grid of photosensors. Its particular arrangement of color filters is used in most single-chip digital image sensors used in digital cameras, and camcorders to create a color image. The filter pattern is half green, one quarter red and one quarter blue, hence is also called BGGR, RGBG, GRBG, or RGGB.

<span class="mw-page-title-main">Digital single-lens reflex camera</span> Digital cameras combining the parts of a single-lens reflex camera and a digital camera back

A digital single-lens reflex camera is a digital camera that combines the optics and mechanisms of a single-lens reflex camera with a solid-state image sensor and digitally records the images from the sensor.

<span class="mw-page-title-main">FR-V (microprocessor)</span>

The Fujitsu FR-V is one of the very few processors ever able to process both a very long instruction word (VLIW) and vector processor instructions at the same time, increasing throughput with high parallel computing while increasing performance per watt and hardware efficiency. The family was presented in 1999. Its design was influenced by the VPP500/5000 models of the Fujitsu VP/2000 vector processor supercomputer line.

<span class="mw-page-title-main">Full-frame DSLR</span> Image sensor format

A full-frame DSLR is a digital single-lens reflex camera (DSLR) with a 35 mm image sensor format. Historically, 35 mm was one of the standard film formats, alongside larger ones, such as medium format and large format. The full-frame DSLR is in contrast to full-frame mirrorless interchangeable-lens cameras, and DSLR and mirrorless cameras with smaller sensors, much smaller than a full 35 mm frame. Many digital cameras, both compact and SLR models, use a smaller-than-35 mm frame as it is easier and cheaper to manufacture imaging sensors at a smaller size. Historically, the earliest digital SLR models, such as the Nikon NASA F4 or Kodak DCS 100, also used a smaller sensor.

A camera raw image file contains unprocessed or minimally processed data from the image sensor of either a digital camera, a motion picture film scanner, or other image scanner. Raw files are so named because they are not yet processed, and contain large amounts of potentially redundant data. Normally, the image is processed by a raw converter, in a wide-gamut internal color space where precise adjustments can be made before conversion to a viewable file format such as JPEG or PNG for storage, printing, or further manipulation. There are dozens of raw formats in use by different manufacturers of digital image capture equipment.

<span class="mw-page-title-main">Lumix</span> Brand of digital cameras made by Panasonic

Lumix is Panasonic's brand of digital cameras, ranging from pocket point-and-shoot models to digital SLRs.

<span class="mw-page-title-main">Digital photography</span> Photography with a digital camera

Digital photography uses cameras containing arrays of electronic photodetectors interfaced to an analog-to-digital converter (ADC) to produce images focused by a lens, as opposed to an exposure on photographic film. The digitized image is stored as a computer file ready for further digital processing, viewing, electronic publishing, or digital printing. It is a form of digital imaging based on gathering visible light.

<span class="mw-page-title-main">Image sensor</span> Device that converts images into electronic signals

An image sensor or imager is a sensor that detects and conveys information used to form an image. It does so by converting the variable attenuation of light waves into signals, small bursts of current that convey the information. The waves can be light or other electromagnetic radiation. Image sensors are used in electronic imaging devices of both analog and digital types, which include digital cameras, camera modules, camera phones, optical mouse devices, medical imaging equipment, night vision equipment such as thermal imaging devices, radar, sonar, and others. As technology changes, electronic and digital imaging tends to replace chemical and analog imaging.

<span class="mw-page-title-main">Image stabilization</span> Techniques used to reduce blurring of images

Image stabilization (IS) is a family of techniques that reduce blurring associated with the motion of a camera or other imaging device during exposure.

<span class="mw-page-title-main">Image sensor format</span> Shape and size of a digital cameras image sensor

In digital photography, the image sensor format is the shape and size of the image sensor.

BIONZ is a line of image processors used in Sony digital cameras.

<span class="mw-page-title-main">Micro Four Thirds system</span> Digital camera design standard

The Micro Four Thirds system is a standard released by Olympus Imaging Corporation and Panasonic in 2008, for the design and development of mirrorless interchangeable lens digital cameras, camcorders and lenses. Camera bodies are available from Blackmagic, DJI, JVC, Kodak, Olympus, OM System, Panasonic, Sharp, Logitech Mevo and Xiaomi. MFT lenses are produced by Cosina Voigtländer, Kowa, Kodak, Mitakon, Olympus, Panasonic, Samyang, Sharp, Sigma, SLR Magic, Tamron, Tokina, TTArtisan, Veydra, Xiaomi, Laowa, Yongnuo, Zonlai, Lensbaby, Venus Optics and 7artisans amongst others.

<span class="mw-page-title-main">Leica S2</span> Camera model

The Leica S2 is a medium format digital SLR camera announced by Leica Camera on September 23, 2008.

<span class="mw-page-title-main">Mirrorless camera</span> Compact camera with a user-removable and replaceable lens

A mirrorless camera is a digital camera which, in contrast to DSLRs, does not use a mirror in order to ensure that the image presented to the photographer through the viewfinder is identical to that taken by the camera. They have come to replace DSLRs, which have historically dominated interchangeable lens cameras. Other terms include electronic viewfinder interchangeable lens (EVIL) cameras and compact system cameras (CSCs).

<span class="mw-page-title-main">Expeed</span> Nikon media processors

The Nikon Expeed image/video processors are media processors for Nikon's digital cameras. They perform a large number of tasks: Bayer filtering, demosaicing, image sensor corrections/dark-frame subtraction, image noise reduction, image sharpening, image scaling, gamma correction, image enhancement/Active D-Lighting, colorspace conversion, chroma subsampling, framerate conversion, lens distortion/chromatic aberration correction, image compression/JPEG encoding, video compression, display/video interface driving, digital image editing, face detection, audio processing/compression/encoding and computer data storage/data transmission.

The Socionext Milbeaut image/video processors are media processors in multi-processor system on a chip architecture. Started by Fujitsu with the M-1 Series in 2000 each generation has several variants regarding included modules and processor-cores, built for mobile phones, digital compact cameras, MILCs and DSLRs like Leica M and Leica S2, Nikon DSLRs, some Pentax K mount cameras and for the Sigma True-II processor.

CxProcess is the trademark of an image processing technology used in Minolta and Konica Minolta digital cameras.

References

  1. DIGITAL SIGNAL & IMAGE PROCESSING
  2. Fundamentals of digital image processing
  3. Merigot, Alain; Petrosino, Alfredo (2008-12-01). "Parallel processing for image and video processing: Issues and challenges". Parallel Computing. Parallel Processing for Image and Video Processing. 34 (12): 694–699. doi:10.1016/j.parco.2008.09.009. ISSN   0167-8191.
  4. "Handheld Products". 11 March 2006. Archived from the original on 11 March 2006. Retrieved 14 September 2019.
  5. Inside the Canon Rebel T4i DSLR Archived 2012-09-21 at the Wayback Machine Chipworks
  6. Amadeo, Ron (17 October 2017). "Surprise! The Pixel 2 is hiding a custom Google SoC for image processing". Ars Technica. Retrieved 19 October 2017.
  7. "7.8. Intel Image Processing Unit 6 (IPU6) Input System driver — The Linux Kernel documentation". docs.kernel.org. Retrieved 2024-08-30.
  8. Fujitsu Microelectronics-Leica's Image Processing System Solution For High-End DSLR Archived 2008-10-07 at the Wayback Machine
  9. Milbeaut and EXPEED Archived 2016-05-21 at the Wayback Machine byThom