Image fusion

Last updated

The image fusion process is defined as gathering all the important information from multiple images, and their inclusion into fewer images, usually a single one. This single image is more informative and accurate than any single source image, and it consists of all the necessary information. The purpose of image fusion is not only to reduce the amount of data but also to construct images that are more appropriate and understandable for the human and machine perception. [1] [2] In computer vision, multisensor image fusion is the process of combining relevant information from two or more images into a single image. [3] The resulting image will be more informative than any of the input images. [4]

Contents

In remote sensing applications, the increasing availability of space borne sensors gives a motivation for different image fusion algorithms. Several situations in image processing require high spatial and high spectral resolution in a single image. Most of the available equipment is not capable of providing such data convincingly. Image fusion techniques allow the integration of different information sources. The fused image can have complementary spatial and spectral resolution characteristics. However, the standard image fusion techniques can distort the spectral information of the multispectral data while merging.

In satellite imaging, two types of images are available. The panchromatic image acquired by satellites is transmitted with the maximum resolution available and the multispectral data are transmitted with coarser resolution. This will usually be two or four times lower. At the receiver station, the panchromatic image is merged with the multispectral data to convey more information.

Many methods exist to perform image fusion. The very basic one is the high-pass filtering technique. Later techniques are based on Discrete Wavelet Transform, uniform rational filter bank, and Laplacian pyramid.

Motivation

Multi sensor data fusion has become a discipline which demands more general formal solutions to a number of application cases. Several situations in image processing require both high spatial and high spectral information in a single image. [5] This is important in remote sensing. However, the instruments are not capable of providing such information either by design or because of observational constraints. One possible solution for this is data fusion.

Methods

Image fusion methods can be broadly classified into two groups – spatial domain fusion and transform domain fusion.

The fusion methods such as averaging, Brovey method, principal component analysis (PCA) and IHS based methods fall under spatial domain approaches. Another important spatial domain fusion method is the high-pass filtering based technique. Here the high frequency details are injected into upsampled version of MS images. The disadvantage of spatial domain approaches is that they produce spatial distortion in the fused image. Spectral distortion becomes a negative factor while we go for further processing, such as classification problem. Spatial distortion can be very well handled by frequency-domain approaches on image fusion. The multiresolution analysis has become a very useful tool for analysing remote sensing images. The discrete wavelet transform has become a very useful tool for fusion. Some other fusion methods are also there, such as Laplacian pyramid based, curvelet transform based etc. These methods show a better performance in spatial and spectral quality of the fused image compared to other spatial methods of fusion.

The images used in image fusion should already be registered. Misregistration is a major source of error in image fusion. Some well-known image fusion methods are:

Comparative analysis of image fusion methods demonstrates that different metrics support different user needs, sensitive to different image fusion methods, and need to be tailored to the application. Categories of image fusion metrics are based on information theory [4] features, structural similarity, or human perception. [6]

Multi-focus image fusion

Multi-focus image fusion is used to collect useful and necessary information from input images with different focus depths in order to create an output image that ideally has all information from input images. [2] [7] In visual sensor network (VSN), sensors are cameras which record images and video sequences. In many applications of VSN, a camera can’t give a perfect illustration including all details of the scene. This is because of the limited depth of focus exists in the optical lens of cameras. [8] Therefore, just the object located in the focal length of camera is focused and cleared and the other parts of image are blurred. VSN has an ability to capture images with different depth of focuses in the scene using several cameras. Due to the large amount of data generated by camera compared to other sensors such as pressure and temperature sensors and some limitation such as limited band width, energy consumption and processing time, it is essential to process the local input images to decrease the amount of transmission data. The aforementioned reasons emphasize the necessary of multi-focus images fusion. Multi-focus image fusion is a process which combines the input multi-focus images into a single image including all important information of the input images and it’s more accurate explanation of the scene than every single input image. [2]

Applications

In remote sensing

Image fusion in remote sensing has several application domains. An important domain is the multi-resolution image fusion (commonly referred to pan-sharpening). In satellite imagery we can have two types of images:

The SPOT PAN satellite provides high resolution (10m pixel) panchromatic data. While the LANDSAT TM satellite provides low resolution (30m pixel) multispectral images. Image fusion attempts to merge these images and produce a single high resolution multispectral image.

The standard merging methods of image fusion are based on Red–Green–Blue (RGB) to Intensity–Hue–Saturation (IHS) transformation. The usual steps involved in satellite image fusion are as follows:

  1. Resize the low resolution multispectral images to the same size as the panchromatic image.
  2. Transform the R, G and B bands of the multispectral image into IHS components.
  3. Modify the panchromatic image with respect to the multispectral image. This is usually performed by histogram matching of the panchromatic image with Intensity component of the multispectral images as reference.
  4. Replace the intensity component by the panchromatic image and perform inverse transformation to obtain a high resolution multispectral image.

Pan-sharpening can be done with Photoshop. [9] Other applications of image fusion in remote sensing are available. [10]

In medical imaging

Image fusion has become a common term used within medical diagnostics and treatment. [11] The term is used when multiple images of a patient are registered and overlaid or merged to provide additional information. Fused images may be created from multiple images from the same imaging modality, [12] or by combining information from multiple modalities, [13] such as magnetic resonance image (MRI), computed tomography (CT), positron emission tomography (PET), and single-photon emission computed tomography (SPECT). In radiology and radiation oncology, these images serve different purposes. For example, CT images are used more often to ascertain differences in tissue density while MRI images are typically used to diagnose brain tumors.

For accurate diagnosis, radiologists must integrate information from multiple image formats. Fused, anatomically consistent images are especially beneficial in diagnosing and treating cancer. With the advent of these new technologies, radiation oncologists can take full advantage of intensity modulated radiation therapy (IMRT). Being able to overlay diagnostic images into radiation planning images results in more accurate IMRT target tumor volumes.

See also

Related Research Articles

<span class="mw-page-title-main">Remote sensing</span> Acquisition of information at a significant distance from the subject

Remote sensing is the acquisition of information about an object or phenomenon without making physical contact with the object, in contrast to in situ or on-site observation. The term is applied especially to acquiring information about Earth and other planets. Remote sensing is used in numerous fields, including geophysics, geography, land surveying and most Earth science disciplines ; it also has military, intelligence, commercial, economic, planning, and humanitarian applications, among others.

<span class="mw-page-title-main">Computational photography</span> Set of digital image capture and processing techniques

Computational photography refers to digital image capture and processing techniques that use digital computation instead of optical processes. Computational photography can improve the capabilities of a camera, or introduce features that were not possible at all with film-based photography, or reduce the cost or size of camera elements. Examples of computational photography include in-camera computation of digital panoramas, high-dynamic-range images, and light field cameras. Light field cameras use novel optical elements to capture three dimensional scene information which can then be used to produce 3D images, enhanced depth-of-field, and selective de-focusing. Enhanced depth-of-field reduces the need for mechanical focusing systems. All of these features use computational imaging techniques.

<span class="mw-page-title-main">Satellite imagery</span> Images taken from an artificial satellite

Satellite images are images of Earth collected by imaging satellites operated by governments and businesses around the world. Satellite imaging companies sell images by licensing them to governments and businesses such as Apple Maps and Google Maps.

<span class="mw-page-title-main">Sensor fusion</span> Combining of sensor data from disparate sources

Sensor fusion is the process of combining sensor data or data derived from disparate sources such that the resulting information has less uncertainty than would be possible when these sources were used individually. For instance, one could potentially obtain a more accurate location estimate of an indoor object by combining multiple data sources such as video cameras and WiFi localization signals. The term uncertainty reduction in this case can mean more accurate, more complete, or more dependable, or refer to the result of an emerging view, such as stereoscopic vision.

<span class="mw-page-title-main">Multispectral imaging</span> Capturing image data across multiple electromagnetic spectrum ranges

Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum. The wavelengths may be separated by filters or detected with the use of instruments that are sensitive to particular wavelengths, including light from frequencies beyond the visible light range, i.e. infrared and ultra-violet. It can allow extraction of additional information the human eye fails to capture with its visible receptors for red, green and blue. It was originally developed for military target identification and reconnaissance. Early space-based imaging platforms incorporated multispectral imaging technology to map details of the Earth related to coastal boundaries, vegetation, and landforms. Multispectral imaging has also found use in document and painting analysis.

Spectral imaging is imaging that uses multiple bands across the electromagnetic spectrum. While an ordinary camera captures light across three wavelength bands in the visible spectrum, red, green, and blue (RGB), spectral imaging encompasses a wide variety of techniques that go beyond RGB. Spectral imaging may use the infrared, the visible spectrum, the ultraviolet, x-rays, or some combination of the above. It may include the acquisition of image data in visible and non-visible bands simultaneously, illumination from outside the visible range, or the use of optical filters to capture a specific spectral range. It is also possible to capture hundreds of wavelength bands for each pixel in an image.

Demosaicing, also known as color reconstruction, is a digital image processing algorithm used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA) such as a Bayes filter. It is also known as CFA interpolation or debayering.

<span class="mw-page-title-main">Hyperspectral imaging</span> Multi-wavelength imaging method

Hyperspectral imaging collects and processes information from across the electromagnetic spectrum. The goal of hyperspectral imaging is to obtain the spectrum for each pixel in the image of a scene, with the purpose of finding objects, identifying materials, or detecting processes. There are three general types of spectral imagers. There are push broom scanners and the related whisk broom scanners, which read images over time, band sequential scanners, which acquire images of an area at different wavelengths, and snapshot hyperspectral imagers, which uses a staring array to generate an image in an instant.

<span class="mw-page-title-main">Data fusion</span> Integration of multiple data sources to provide better information

Data fusion is the process of integrating multiple data sources to produce more consistent, accurate, and useful information than that provided by any individual data source.

Chemical imaging is the analytical capability to create a visual image of components distribution from simultaneous measurement of spectra and spatial, time information. Hyperspectral imaging measures contiguous spectral bands, as opposed to multispectral imaging which measures spaced spectral bands.

Pansharpening is a process of merging high-resolution panchromatic and lower resolution multispectral imagery to create a single high-resolution color image. Google Maps and nearly every map creating company use this technique to increase image quality. Pansharpening produces a high-resolution color image from three, four or more low-resolution multispectral satellite bands plus a corresponding high-resolution panchromatic band:

Low-res color bands + High-res grayscale band = Hi-res color image

<span class="mw-page-title-main">Airborne Real-time Cueing Hyperspectral Enhanced Reconnaissance</span> Aerial imaging system

Airborne Real-time Cueing Hyperspectral Enhanced Reconnaissance, also known by the acronym ARCHER, is an aerial imaging system that produces ground images far more detailed than plain sight or ordinary aerial photography can. It is the most sophisticated unclassified hyperspectral imaging system available, according to U.S. Government officials. ARCHER can automatically scan detailed imaging for a given signature of the object being sought, for abnormalities in the surrounding area, or for changes from previous recorded spectral signatures.

<span class="mw-page-title-main">China–Brazil Earth Resources Satellite program</span> Satellite cooperation program between China and Brazil

The China–Brazil Earth Resources Satellite program (CBERS) is a technological cooperation program between Brazil and China which develops and operates Earth observation satellites.

Multispectral remote sensing is the collection and analysis of reflected, emitted, or back-scattered energy from an object or an area of interest in multiple bands of regions of the electromagnetic spectrum. Subcategories of multispectral remote sensing include hyperspectral, in which hundreds of bands are collected and analyzed, and ultraspectral remote sensing where many hundreds of bands are used. The main purpose of multispectral imaging is the potential to classify the image using multispectral classification. This is a much faster method of image analysis than is possible by human interpretation.

<span class="mw-page-title-main">Snapshot hyperspectral imaging</span> Method for capturing hyperspectral images

Snapshot hyperspectral imaging is a method for capturing hyperspectral images during a single integration time of a detector array. No scanning is involved with this method, in contrast to push broom and whisk broom scanning techniques. The lack of moving parts means that motion artifacts should be avoided. This instrument typically features detector arrays with a high number of pixels.

The modular optoelectronic multispectral scanner (MOMS) is a scanning system for spaceborne, geoscientific remote sensing applications used in satellite navigation systems for sensing atmospheric and oceanic systems. The scanner is combination of separate spectrometer blocks.

PRISMA is an Italian Space Agency pre-operational and technology demonstrator mission focused on the development and delivery of hyperspectral products and the qualification of the hyperspectral payload in space.

Computational imaging is the process of indirectly forming images from measurements using algorithms that rely on a significant amount of computing. In contrast to traditional imaging, computational imaging systems involve a tight integration of the sensing system and the computation in order to form the images of interest. The ubiquitous availability of fast computing platforms, the advances in algorithms and modern sensing hardware is resulting in imaging systems with significantly enhanced capabilities. Computational Imaging systems cover a broad range of applications include computational microscopy, tomographic imaging, MRI, ultrasound imaging, computational photography, Synthetic Aperture Radar (SAR), seismic imaging etc. The integration of the sensing and the computation in computational imaging systems allows for accessing information which was otherwise not possible. For example:

<span class="mw-page-title-main">Remote sensing (geology)</span> Data acquisition method for earth sciences

Remote sensing in geology is remote sensing used in the geological sciences as a data acquisition method complementary to field observation, because it allows mapping of geological characteristics of regions without physical contact with the areas being explored. About one-fourth of the Earth's total surface area is exposed land where information is ready to be extracted from detailed earth observation via remote sensing. Remote sensing is conducted via detection of electromagnetic radiation by sensors. The radiation can be naturally sourced, or produced by machines and reflected off of the Earth surface. The electromagnetic radiation acts as an information carrier for two main variables. First, the intensities of reflectance at different wavelengths are detected, and plotted on a spectral reflectance curve. This spectral fingerprint is governed by the physio-chemical properties of the surface of the target object and therefore helps mineral identification and hence geological mapping, for example by hyperspectral imaging. Second, the two-way travel time of radiation from and back to the sensor can calculate the distance in active remote sensing systems, for example, Interferometric synthetic-aperture radar. This helps geomorphological studies of ground motion, and thus can illuminate deformations associated with landslides, earthquakes, etc.

Multi-focus image fusion is a multiple image compression technique using input images with different focus depths to make one output image that preserves all information.

References

  1. ↑ Zheng, Yufeng; Blasch, Erik; Liu, Zheng (2018). Multispectral Image Fusion and Colorization. SPIE Press. ISBN   9781510619067.
  2. 1 2 3 M., Amin-Naji; A., Aghagolzadeh (2018). "Multi-Focus Image Fusion in DCT Domain using Variance and Energy of Laplacian and Correlation Coefficient for Visual Sensor Networks". Journal of AI and Data Mining. 6 (2): 233–250. doi:10.22044/jadm.2017.5169.1624. ISSN   2322-5211.
  3. ↑ Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "Multi-focus image fusion for visual sensor networks in DCT domain". Computers & Electrical Engineering. 37 (5): 789–797. doi:10.1016/j.compeleceng.2011.04.016. S2CID   38131177.
  4. 1 2 Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "A non-reference image fusion metric based on mutual information of image features". Computers & Electrical Engineering. 37 (5): 744–756. doi:10.1016/j.compeleceng.2011.07.012. S2CID   7738541.
  5. ↑ AL Smadi, Ahmad (18 May 2021). "Smart pansharpening approach using kernel-based image filtering". IET Image Processing. 15 (11): 2629–2642. doi:10.1049/ipr2.12251. S2CID   235632628.
  6. ↑ Liu, Z.; Blasch, E.; Xue, Z.; Langaniere, R.; Wu, W. (2012). "Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision: A Comparative Survey". IEEE Transactions on Pattern Analysis and Machine Intelligence. 34 (1): 94–109. doi:10.1109/tpami.2011.109. PMID   21576753. S2CID   9248856.
  7. ↑ Naji, M. A.; Aghagolzadeh, A. (November 2015). "Multi-focus image fusion in DCT domain based on correlation coefficient". 2015 2nd International Conference on Knowledge-Based Engineering and Innovation (KBEI). pp. 632–639. doi:10.1109/KBEI.2015.7436118. ISBN   978-1-4673-6506-2. S2CID   44524869.
  8. ↑ Naji, M. A.; Aghagolzadeh, A. (November 2015). "A new multi-focus image fusion technique based on variance in DCT domain". 2015 2nd International Conference on Knowledge-Based Engineering and Innovation (KBEI). pp. 478–484. doi:10.1109/KBEI.2015.7436092. ISBN   978-1-4673-6506-2. S2CID   29215692.
  9. ↑ Pan-sharpening in Photoshop
  10. ↑ "Beyond Pan-sharpening: Pixel-level Fusion in Remote Sensing Applications" (PDF). Archived from the original (PDF) on 2015-09-01. Retrieved 2013-03-05.
  11. ↑ James, A.P.; Dasarathy, B V. (2014). "Medical Image Fusion: A survey of state of the art". Information Fusion. 19: 4–19. arXiv: 1401.0166 . doi:10.1016/j.inffus.2013.12.002. S2CID   15315731.
  12. ↑ Gooding, M.J.; et al. (2010). "Investigation into the fusion of multiple 4-D fetal echocardiography images to improve image quality". Ultrasound in Medicine and Biology. 36 (6): 957–66. doi:10.1016/j.ultrasmedbio.2010.03.017. PMID   20447758.
  13. ↑ Maintz, J.B.; Viergever, M.A. (1998). "A survey of medical image registration". Medical Image Analysis. 2 (1): 1–36. CiteSeerX   10.1.1.46.4959 . doi:10.1016/s1361-8415(01)80026-8. PMID   10638851.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.