Image registration

Last updated
Registering and summing multiple exposures of the same scene improve signal to noise ratio, allowing one to see things previously impossible to see. In this picture, the distant Alps are made visible, although they are tens of kilometers into the haze. Registrator Demo2.png
Registering and summing multiple exposures of the same scene improve signal to noise ratio, allowing one to see things previously impossible to see. In this picture, the distant Alps are made visible, although they are tens of kilometers into the haze.

Image registration is the process of transforming different sets of data into one coordinate system. Data may be multiple photographs, data from different sensors, times, depths, or viewpoints. [1] It is used in computer vision, medical imaging, [2] military automatic target recognition, and compiling and analyzing images and data from satellites. Registration is necessary in order to be able to compare or integrate the data obtained from these different measurements.

Contents

Algorithm classification

Intensity-based vs feature-based

Image registration or image alignment algorithms can be classified into intensity-based and feature-based. [3] One of the images is referred to as the moving or source and the others are referred to as the target, fixed or sensed images. Image registration involves spatially transforming the source/moving image(s) to align with the target image. The reference frame in the target image is stationary, while the other datasets are transformed to match to the target. [3] Intensity-based methods compare intensity patterns in images via correlation metrics, while feature-based methods find correspondence between image features such as points, lines, and contours. [3] Intensity-based methods register entire images or sub-images. If sub-images are registered, centers of corresponding sub images are treated as corresponding feature points. Feature-based methods establish a correspondence between a number of especially distinct points in images. Knowing the correspondence between a number of points in images, a geometrical transformation is then determined to map the target image to the reference images, thereby establishing point-by-point correspondence between the reference and target images. [3] Methods combining intensity-based and feature-based information have also been developed. [4]

Transformation models

Image registration algorithms can also be classified according to the transformation models they use to relate the target image space to the reference image space. The first broad category of transformation models includes linear transformations, which include rotation, scaling, translation, and other affine transforms. [5] Linear transformations are global in nature, thus, they cannot model local geometric differences between images. [3]

The second category of transformations allow 'elastic' or 'nonrigid' transformations. These transformations are capable of locally warping the target image to align with the reference image. Nonrigid transformations include radial basis functions (thin-plate or surface splines, multiquadrics, and compactly-supported transformations [3] ), physical continuum models (viscous fluids), and large deformation models (diffeomorphisms).

Transformations are commonly described by a parametrization, where the model dictates the number of parameters. For instance, the translation of a full image can be described by a single parameter, a translation vector. These models are called parametric models. Non-parametric models on the other hand, do not follow any parameterization, allowing each image element to be displaced arbitrarily. [6]

There are a number of programs that implement both estimation and application of a warp-field. It is a part of the SPM and AIR programs.

Transformations of coordinates via the law of function composition rather than addition

Alternatively, many advanced methods for spatial normalization are building on structure preserving transformations homeomorphisms and diffeomorphisms since they carry smooth submanifolds smoothly during transformation. Diffeomorphisms are generated in the modern field of Computational Anatomy based on flows since diffeomorphisms are not additive although they form a group, but a group under the law of function composition. For this reason, flows which generalize the ideas of additive groups allow for generating large deformations that preserve topology, providing 1-1 and onto transformations. Computational methods for generating such transformation are often called LDDMM [7] [8] [9] [10] which provide flows of diffeomorphisms as the main computational tool for connecting coordinate systems corresponding to the geodesic flows of Computational Anatomy.

There are a number of programs which generate diffeomorphic transformations of coordinates via diffeomorphic mapping including MRI Studio [11] and MRI Cloud.org [12]

Spatial vs frequency domain methods

Spatial methods operate in the image domain, matching intensity patterns or features in images. Some of the feature matching algorithms are outgrowths of traditional techniques for performing manual image registration, in which an operator chooses corresponding control points (CP) in images. When the number of control points exceeds the minimum required to define the appropriate transformation model, iterative algorithms like RANSAC can be used to robustly estimate the parameters of a particular transformation type (e.g. affine) for registration of the images.

Frequency-domain methods find the transformation parameters for registration of the images while working in the transform domain. Such methods work for simple transformations, such as translation, rotation, and scaling. Applying the phase correlation method to a pair of images produces a third image which contains a single peak. The location of this peak corresponds to the relative translation between the images. Unlike many spatial-domain algorithms, the phase correlation method is resilient to noise, occlusions, and other defects typical of medical or satellite images. Additionally, the phase correlation uses the fast Fourier transform to compute the cross-correlation between the two images, generally resulting in large performance gains. The method can be extended to determine rotation and scaling differences between two images by first converting the images to log-polar coordinates. [13] [14] Due to properties of the Fourier transform, the rotation and scaling parameters can be determined in a manner invariant to translation.

Single- vs multi-modality methods

Another classification can be made between single-modality and multi-modality methods. Single-modality methods tend to register images in the same modality acquired by the same scanner/sensor type, while multi-modality registration methods tended to register images acquired by different scanner/sensor types.

Multi-modality registration methods are often used in medical imaging as images of a subject are frequently obtained from different scanners. Examples include registration of brain CT/MRI images or whole body PET/CT images for tumor localization, registration of contrast-enhanced CT images against non-contrast-enhanced CT images [15] for segmentation of specific parts of the anatomy, and registration of ultrasound and CT images for prostate localization in radiotherapy.

Automatic vs interactive methods

Registration methods may be classified based on the level of automation they provide. Manual, interactive, semi-automatic, and automatic methods have been developed. Manual methods provide tools to align the images manually. Interactive methods reduce user bias by performing certain key operations automatically while still relying on the user to guide the registration. Semi-automatic methods perform more of the registration steps automatically but depend on the user to verify the correctness of a registration. Automatic methods do not allow any user interaction and perform all registration steps automatically.

Similarity measures for image registration

Image similarities are broadly used in medical imaging. An image similarity measure quantifies the degree of similarity between intensity patterns in two images. [3] The choice of an image similarity measure depends on the modality of the images to be registered. Common examples of image similarity measures include cross-correlation, mutual information, sum of squared intensity differences, and ratio image uniformity. Mutual information and normalized mutual information are the most popular image similarity measures for registration of multimodality images. Cross-correlation, sum of squared intensity differences and ratio image uniformity are commonly used for registration of images in the same modality.

Many new features have been derived for cost functions based on matching methods via large deformations have emerged in the field Computational Anatomy including Measure matching which are pointsets or landmarks without correspondence, Curve matching and Surface matching via mathematical currents and varifolds.

Uncertainty

There is a level of uncertainty associated with registering images that have any spatio-temporal differences. A confident registration with a measure of uncertainty is critical for many change detection applications such as medical diagnostics.

In remote sensing applications where a digital image pixel may represent several kilometers of spatial distance (such as NASA's LANDSAT imagery), an uncertain image registration can mean that a solution could be several kilometers from ground truth. Several notable papers have attempted to quantify uncertainty in image registration in order to compare results. [16] [17] However, many approaches to quantifying uncertainty or estimating deformations are computationally intensive or are only applicable to limited sets of spatial transformations.

Applications

Registration of two MRI images of the brain Mni-autoreg 03-registered.png
Registration of two MRI images of the brain

Image registration has applications in remote sensing (cartography updating), and computer vision. Due to the vast range of applications to which image registration can be applied, it is impossible to develop a general method that is optimized for all uses.

Medical image registration (for data of the same patient taken at different points in time such as change detection or tumor monitoring) often additionally involves elastic (also known as nonrigid) registration to cope with deformation of the subject (due to breathing, anatomical changes, and so forth). [18] [19] [20] Nonrigid registration of medical images can also be used to register a patient's data to an anatomical atlas, such as the Talairach atlas for neuroimaging.

In astrophotography image alignment and stacking are often used to increase the signal to noise ratio for faint objects. Without stacking it may be used to produce a timelapse of events such as a planet's rotation of a transit across the Sun. Using control points (automatically or manually entered), the computer performs transformations on one image to make major features align with a second or multiple images. This technique may also be used for images of different sizes, to allow images taken through different telescopes or lenses to be combined.

In cryo-TEM instability causes specimen drift and many fast acquisitions with accurate image registration is required to preserve high resolution and obtain high signal to noise images. For low SNR data, the best image registration is achieved by cross-correlating all permutations of images in an image stack. [21]

Image registration is an essential part of panoramic image creation. There are many different techniques that can be implemented in real time and run on embedded devices like cameras and camera-phones.

See also

Related Research Articles

<span class="mw-page-title-main">Image segmentation</span> Partitioning a digital image into segments

In digital image processing and computer vision, image segmentation is the process of partitioning a digital image into multiple image segments, also known as image regions or image objects. The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze. Image segmentation is typically used to locate objects and boundaries in images. More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.

Template matching is a technique in digital image processing for finding small parts of an image which match a template image. It can be used for quality control in manufacturing, navigation of mobile robots, or edge detection in images.

<span class="mw-page-title-main">Iterative reconstruction</span>

Iterative reconstruction refers to iterative algorithms used to reconstruct 2D and 3D images in certain imaging techniques. For example, in computed tomography an image must be reconstructed from projections of an object. Here, iterative reconstruction techniques are usually a better, but computationally more expensive alternative to the common filtered back projection (FBP) method, which directly calculates the image in a single reconstruction step. In recent research works, scientists have shown that extremely fast computations and massive parallelism is possible for iterative reconstruction, which makes iterative reconstruction practical for commercialization.

The scale-invariant feature transform (SIFT) is a computer vision algorithm to detect, describe, and match local features in images, invented by David Lowe in 1999. Applications include object recognition, robotic mapping and navigation, image stitching, 3D modeling, gesture recognition, video tracking, individual identification of wildlife and match moving.

<span class="mw-page-title-main">Iterative closest point</span> Algorithm

Iterative closest point (ICP) is an algorithm employed to minimize the difference between two clouds of points. ICP is often used to reconstruct 2D or 3D surfaces from different scans, to localize robots and achieve optimal path planning, to co-register bone models, etc.

<span class="mw-page-title-main">Image stitching</span> Combining multiple photographic images with overlapping fields of view

Image stitching or photo stitching is the process of combining multiple photographic images with overlapping fields of view to produce a segmented panorama or high-resolution image. Commonly performed through the use of computer software, most approaches to image stitching require nearly exact overlaps between images and identical exposures to produce seamless results, although some stitching algorithms actually benefit from differently exposed images by doing high-dynamic-range imaging in regions of overlap. Some digital cameras can stitch their photos internally.

In neuroimaging, spatial normalization is an image processing step, more specifically an image registration method. Human brains differ in size and shape, and one goal of spatial normalization is to deform human brain scans so one location in one subject's brain scan corresponds to the same location in another subject's brain scan.

Statistical shape analysis is an analysis of the geometrical properties of some given set of shapes by statistical methods. For instance, it could be used to quantify differences between male and female gorilla skull shapes, normal and pathological bone shapes, leaf outlines with and without herbivory by insects, etc. Important aspects of shape analysis are to obtain a measure of distance between shapes, to estimate mean shapes from samples, to estimate shape variability within samples, to perform clustering and to test for differences between shapes. One of the main methods used is principal component analysis (PCA). Statistical shape analysis has applications in various fields, including medical imaging, computer vision, computational anatomy, sensor measurement, and geographical profiling.

<span class="mw-page-title-main">Computer-aided diagnosis</span> Type of diagnosis assisted by computers

Computer-aided detection (CADe), also called computer-aided diagnosis (CADx), are systems that assist doctors in the interpretation of medical images. Imaging techniques in X-ray, MRI, Endoscopy, and ultrasound diagnostics yield a great deal of information that the radiologist or other medical professional has to analyze and evaluate comprehensively in a short time. CAD systems process digital images or videos for typical appearances and to highlight conspicuous sections, such as possible diseases, in order to offer input to support a decision taken by the professional.

The image fusion process is defined as gathering all the important information from multiple images, and their inclusion into fewer images, usually a single one. This single image is more informative and accurate than any single source image, and it consists of all the necessary information. The purpose of image fusion is not only to reduce the amount of data but also to construct images that are more appropriate and understandable for the human and machine perception. In computer vision, multisensor image fusion is the process of combining relevant information from two or more images into a single image. The resulting image will be more informative than any of the input images.

Digital image correlation and tracking is an optical method that employs tracking and image registration techniques for accurate 2D and 3D measurements of changes in images. This method is often used to measure full-field displacement and strains, and it is widely applied in many areas of science and engineering. Compared to strain gages and extensometers, the amount of information gathered about the fine details of deformation during mechanical tests is increased due to the ability to provide both local and average data using digital image correlation.

Brain morphometry is a subfield of both morphometry and the brain sciences, concerned with the measurement of brain structures and changes thereof during development, aging, learning, disease and evolution. Since autopsy-like dissection is generally impossible on living brains, brain morphometry starts with noninvasive neuroimaging data, typically obtained from magnetic resonance imaging (MRI). These data are born digital, which allows researchers to analyze the brain images further by using advanced mathematical and statistical methods such as shape quantification or multivariate analysis. This allows researchers to quantify anatomical features of the brain in terms of shape, mass, volume, and to derive more specific information, such as the encephalization quotient, grey matter density and white matter connectivity, gyrification, cortical thickness, or the amount of cerebrospinal fluid. These variables can then be mapped within the brain volume or on the brain surface, providing a convenient way to assess their pattern and extent over time, across individuals or even between different biological species. The field is rapidly evolving along with neuroimaging techniques—which deliver the underlying data—but also develops in part independently from them, as part of the emerging field of neuroinformatics, which is concerned with developing and adapting algorithms to analyze those data.

<span class="mw-page-title-main">3D reconstruction from multiple images</span> Creation of a 3D model from a set of images

3D reconstruction from multiple images is the creation of three-dimensional models from a set of images. It is the reverse process of obtaining 2D images from 3D scenes.

Medical image computing (MIC) is an interdisciplinary field at the intersection of computer science, information engineering, electrical engineering, physics, mathematics and medicine. This field develops computational and mathematical methods for solving problems pertaining to medical images and their use for biomedical research and clinical care.

<span class="mw-page-title-main">Point-set registration</span>

In computer vision, pattern recognition, and robotics, point-set registration, also known as point-cloud registration or scan matching, is the process of finding a spatial transformation that aligns two point clouds. The purpose of finding such a transformation includes merging multiple data sets into a globally consistent model, and mapping a new measurement to a known data set to identify features or to estimate its pose. Raw 3D point cloud data are typically obtained from Lidars and RGB-D cameras. 3D point clouds can also be generated from computer vision algorithms such as triangulation, bundle adjustment, and more recently, monocular image depth estimation using deep learning. For 2D point set registration used in image processing and feature-based image registration, a point set may be 2D pixel coordinates obtained by feature extraction from an image, for example corner detection. Point cloud registration has extensive applications in autonomous driving, motion estimation and 3D reconstruction, object detection and pose estimation, robotic manipulation, simultaneous localization and mapping (SLAM), panorama stitching, virtual and augmented reality, and medical imaging.

YaDICs is a program written to perform digital image correlation on 2D and 3D tomographic images. The program was designed to be both modular, by its plugin strategy and efficient, by it multithreading strategy. It incorporates different transformations, optimizing strategy, Global and/or local shape functions ...

Computational anatomy is an interdisciplinary field of biology focused on quantitative investigation and modelling of anatomical shapes variability. It involves the development and application of mathematical, statistical and data-analytical methods for modelling and simulation of biological structures.

Large deformation diffeomorphic metric mapping (LDDMM) is a specific suite of algorithms used for diffeomorphic mapping and manipulating dense imagery based on diffeomorphic metric mapping within the academic discipline of computational anatomy, to be distinguished from its precursor based on diffeomorphic mapping. The distinction between the two is that diffeomorphic metric maps satisfy the property that the length associated to their flow away from the identity induces a metric on the group of diffeomorphisms, which in turn induces a metric on the orbit of shapes and forms within the field of Computational Anatomy. The study of shapes and forms with the metric of diffeomorphic metric mapping is called diffeomorphometry.

Computational anatomy (CA) is a discipline within medical imaging focusing on the study of anatomical shape and form at the visible or gross anatomical scale of morphology. The field is broadly defined and includes foundations in anatomy, applied mathematics and pure mathematics, including medical imaging, neuroscience, physics, probability, and statistics. It focuses on the anatomical structures being imaged, rather than the medical imaging devices. The central focus of the sub-field of computational anatomy within medical imaging is mapping information across anatomical coordinate systems most often dense information measured within a magnetic resonance image (MRI). The introduction of flows into CA, which are akin to the equations of motion used in fluid dynamics, exploit the notion that dense coordinates in image analysis follow the Lagrangian and Eulerian equations of motion. In models based on Lagrangian and Eulerian flows of diffeomorphisms, the constraint is associated to topological properties, such as open sets being preserved, coordinates not crossing implying uniqueness and existence of the inverse mapping, and connected sets remaining connected. The use of diffeomorphic methods grew quickly to dominate the field of mapping methods post Christensen's original paper, with fast and symmetric methods becoming available.

Elastix is an image registration toolbox built upon the Insight Segmentation and Registration Toolkit (ITK). It is entirely open-source and provides a wide range of algorithms employed in image registration problems. Its components are designed to be modular to ease a fast and reliable creation of various registration pipelines tailored for case-specific applications. It was first developed by Stefan Klein and Marius Staring under the supervision of Josien P.W. Pluim at Image Sciences Institute (ISI). Its first version was command-line based, allowing the final user to employ scripts to automatically process big data-sets and deploy multiple registration pipelines with few lines of code. Nowadays, to further widen its audience, a version called SimpleElastix is also available, developed by Kasper Marstal, which allows the integration of elastix with high level languages, such as Python, Java, and R.

References

  1. ↑ Lisa Gottesfeld Brown, A survey of image registration techniques (abstract), ACM Computing Surveys archive, volume 24, issue 4, December 1992), pages 325 - 376
  2. ↑ biological imaging and brain mapping
  3. 1 2 3 4 5 6 7 A. Ardeshir Goshtasby: 2-D and 3-D Image Registration for Medical, Remote Sensing, and Industrial Applications, Wiley Press, 2005.
  4. ↑ Papademetris, Xenophon; Jackowski, Andrea P.; Schultz, Robert T.; Staib, Lawrence H.; Duncan, James S. (2004). "Integrated Intensity and Point-Feature Nonrigid Registration". Medical Image Computing and Computer-Assisted Intervention – MICCAI 2004. Lecture Notes in Computer Science. Vol. 3216. pp. 763–770. doi:10.1007/978-3-540-30135-6_93. ISBN   978-3-540-22976-6. ISSN   0302-9743.
  5. ↑ http://www.comp.nus.edu.sg/~cs4243/lecture/register.pdf [ bare URL PDF ]
  6. ↑ Sotiras, A.; Davatzikos, C.; Paragios, N. (July 2013). "Deformable Medical Image Registration: A Survey". IEEE Transactions on Medical Imaging. 32 (7): 1153–1190. doi:10.1109/TMI.2013.2265603. PMC   3745275 . PMID   23739795.
  7. ↑ Toga, Arthur W. (1998-11-17). Brain Warping. Academic Press. ISBN   9780080525549.
  8. ↑ "Landmark matching on brain surfaces via large deformation diffeomorphisms on the sphere — University of Utah". utah.pure.elsevier.com. Archived from the original on 2018-06-29. Retrieved 2016-03-21.
  9. ↑ Beg, M. Faisal; Miller, Michael I.; Trouvé, Alain; Younes, Laurent (2005). "Computing Large Deformation Metric Mappings via Geodesic Flows of Diffeomorphisms". International Journal of Computer Vision. 61 (2): 139–157. doi:10.1023/B:VISI.0000043755.93987.aa. S2CID   17772076 . Retrieved 2016-03-21.
  10. ↑ Joshi, S. C.; Miller, M. I. (2000-01-01). "Landmark matching via large deformation diffeomorphisms". IEEE Transactions on Image Processing. 9 (8): 1357–1370. Bibcode:2000ITIP....9.1357J. doi:10.1109/83.855431. ISSN   1057-7149. PMID   18262973.
  11. ↑ "MRI Studio".
  12. ↑ "MRICloud Brain Mapping".
  13. ↑ B. Srinivasa Reddy; B. N. Chatterji (Aug 1996). "An FFT-Based Technique for Translation, Rotation and Scale-Invariant Image Registration". IEEE Transactions on Image Processing. 5 (8): 1266–1271. doi:10.1109/83.506761. PMID   18285214. S2CID   6562358.
  14. ↑ Zokai, S., Wolberg, G., "Image Registration Using Log-Polar Mappings for Recovery of Large-Scale Similarity and Projective Transformations". IEEE Transactions on Image Processing, vol. 14, No. 10, October, 2005.
  15. ↑ Ristea, Nicolae-Catalin; Miron, Andreea-Iuliana; Savencu, Olivian; Georgescu, Mariana-Iuliana; Verga, Nicolae; Khan, Fahad Shahbaz; Ionescu, Radu Tudor (2021-10-21). "CyTran: Cycle-Consistent Transformers for Non-Contrast to Contrast CT Translation". arXiv:2110.06400 [cs, eess].
  16. ↑ Simonson, K., Drescher, S., Tanner, F., A Statistics Based Approach to Binary Image Registration with Uncertainty Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, No. 1, January 2007
  17. ↑ Domokos, C., Kato, Z., Francos, J., Parametric estimation of affine deformations of binary images. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, 2008
  18. ↑ Zhao, Shengyu; Lau, Tingfung; Luo, Ji; Chang, Eric I-Chao; Xu, Yan (2020). "Unsupervised 3D End-to-End Medical Image Registration With Volume Tweening Network". IEEE Journal of Biomedical and Health Informatics. 24 (5): 1394–1404. doi:10.1109/JBHI.2019.2951024. ISSN   2168-2208.
  19. ↑ Chen, Junyu; He, Yufan; Frey, Eric C.; Li, Ye; Du, Yong (2021-04-13). "ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration". arXiv:2104.06468 [cs, eess].
  20. ↑ Burduja, Mihail; Ionescu, Radu Tudor (2021). "Unsupervised Medical Image Alignment With Curriculum Learning". 2021 IEEE International Conference on Image Processing (ICIP): 3787–3791. doi:10.1109/ICIP42928.2021.9506067.
  21. ↑ Savitsky; El Baggari; Clement; Hovden; Kourkoutis (2018). "Image registration of low signal-to-noise cryo-STEM data". Ultramicroscopy. 191: 56–65. arXiv: 1710.09281 . doi:10.1016/j.ultramic.2018.04.008. PMID   29843097. S2CID   26983019.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.