Comparison gallery of image scaling algorithms

Last updated

This gallery shows the results of numerous image scaling algorithms.

Contents

Scaling methods

An image size can be changed in several ways. Consider resizing a 160x160 pixel photo to the following 40x40 pixel thumbnail and then scaling the thumbnail to a 160x160 pixel image. Also consider doubling the size of the following image containing text.

Low-resolution images
ThumbnailText
40 by 40 thumbnail of 'Green Sea Shell'.png Image-before-scaling.png
Comparison of scaling methods
Original photoUpscaled thumbnailUpscaled textAlgorithm and description
160 by 160 thumbnail of 'Green Sea Shell'.png 40 by 40 thumbnail of 'Green Sea Shell' (x4 Nearest Neighbour).png Image-after-trivial-scaling.png

Nearest-neighbor interpolation

One of the simpler ways of increasing the size, replacing every pixel with a number of pixels of the same color. The resulting image is larger than the original, and preserves all the original detail, but has (possibly undesirable) jaggedness. The diagonal lines of the "W", for example, now show the "stairway" shape characteristic of nearest-neighbor interpolation. Other scaling methods below are better at preserving smooth contours in the image.

160 by 160 thumbnail of 'Green Sea Shell'.png 40 by 40 thumbnail of 'Green Sea Shell' (x4 Bilinear).png Image-after-linear-interpolation.png

Bilinear interpolation

Linear (or bilinear, in two dimensions) interpolation is typically good for changing the size of an image, but causes some undesirable softening of details and can still be somewhat jagged.

160 by 160 thumbnail of 'Green Sea Shell'.png 40 by 40 thumbnail of 'Green Sea Shell' (x4 Bicubic).png Image-after-cubic-interpolation.png

Bicubic interpolation

Better scaling methods include Lanczos resampling and Mitchell-Netravali filters.

160 by 160 thumbnail of 'Green Sea Shell'.png 40 by 40 thumbnail of 'Green Sea Shell' fourier interpolated.png Image-after-fourier saturated.png

Fourier-based interpolation

Simple Fourier based interpolation based on padding of the frequency domain with zero components (a smooth-window-based approach would reduce the ringing). Beside the good conservation of details, notable is the ringing and the circular bleeding of content from the left border to right border (and way around).

160 by 160 thumbnail of 'Green Sea Shell'.png 40 by 40 thumbnail of 'Green Sea Shell' (x4 DCCI).png Wiki dcci 2x.png

Edge-directed interpolation

Edge-directed interpolation algorithms aim to preserve edges in the image after scaling, unlike other algorithms which can produce staircase artifacts around diagonal lines or curves. Examples of algorithms for this task include New Edge-Directed Interpolation (NEDI), [1] [2] Edge-Guided Image Interpolation (EGGI), [3] Iterative Curvature-Based Interpolation (ICBI),[ citation needed ] and Directional Cubic Convolution Interpolation (DCCI). [4] A study found that DCCI had the best scores in PSNR and SSIM on a series of test images. [5]

160 by 160 thumbnail of 'Green Sea Shell'.png 40 by 40 thumbnail of 'Green Sea Shell' (Hq4x).png Image-after-hq2x.png

Pixel art scaling algorithms (hqx)

For magnifying computer graphics with low resolution and few colors (usually from 2 to 256 colors), better results will be achieved by pixel art scaling algorithms such as hqx or xbr. These produce sharp edges and maintain high level of detail. Unfortunately due to the standardized size of 218x80 pixels, the "Wiki" image cannot use HQ4x or 4xBRZ to better demonstrate the artifacts they may produce such as row shifting.


The example images use HQ4x and HQ2x respectively.

160 by 160 thumbnail of 'Green Sea Shell'.png 160 by 160 upscaled thumbnail of 'Green Sea Shell' (4xBRZ).png Image after scaling (2xBRZ).png

Pixel art scaling algorithms (xbr)

The xbr family is very useful for creating smooth edges. It will however deform the shape significantly, which in many cases creates a very appealing result. However it will create an effect similar to posterization by grouping together local areas into a single colour. It will also remove small details if in-between larger ones which connect together.

The example images use 4xBRZ and 2xBRZ respectively.

160 by 160 thumbnail of 'Green Sea Shell'.png 160 by 160 thumbnail of 'Green Sea Shell'.png 160 by 160 upscaled thumbnail of 'Green Sea Shell' (GemCutter Preserve Details).png 160 by 160 upscaled thumbnail of 'Green Sea Shell' (Smooth Edges).png Image after scaling (GemCutter Preserve Details).png


Image-after-scaling smooth.png

Pixel art scaling algorithms (GemCutter)

An adaptable technique which can deliver variable amounts of detail or smoothness. It aims to preserve the shape and coordinates of original details, without blurring those details into neighboring ones. It will avoid blending pixels which directly touch each other, and instead only blend pixels with their diagonal neighbors.

The "Cutter" name comes from its tendency to cut corners of squares and turn them into diamonds, as well as create distinct faces along stair-stepped pixels, ie: those which exist on along the angles of edges found on a diamond. The "Gem" prefix both refers to the diamond cut, and also many traditional gem cuts which involve cutting corners at a 45 degree angle.


The example images use GemCutter Preserve Details (Top), and GemCutter Smooth Edges (Bottom).

160 by 160 thumbnail of 'Green Sea Shell'.png 40 by 40 thumbnail of 'Green Sea Shell' inscape-potrace 48colors.png Image scaling example image - intermediate vector representation after applying inkscape vectorization.svg

Image tracing

Vectorization first creates a resolution-independent vector representation of the graphic to be scaled. Then the resolution-independent version is rendered as a raster image at the desired resolution. This technique is used by Adobe Illustrator Live Trace, Inkscape, and several recent papers. [6] Scalable Vector Graphics are well suited to simple geometric images, while photographs do not fare well with vectorization due to their complexity.


Note that the special characteristics of vectors allow for greater resolution example images. The other algorithms are standardized to a resolution of 160x160 and 218x80 pixels respectively.

160 by 160 thumbnail of 'Green Sea Shell'.png Green Sea Shell (EDSR).png Wiki (waifu2x).png

Deep convolutional neural networks

Using machine learning, convincing details are generated as best guesses by learning common patterns from a training data set. The upscaled result is sometimes described as a hallucination because the information introduced may not correspond to the content of the source. Enhanced deep residual network (EDSR) methods have been developed by optimizing conventional residual neural network architecture. [7] Programs that use this method include waifu2x, Imglarger and Neural Enhance.

160 by 160 thumbnail of 'Green Sea Shell'.png Green Sea Shell (ESRGAN).png RealESRGAN-x4plus Time Test Augmentation.png

Deep convolutional neural networks using perceptual loss

Developed on the basis of the super-resolution generative adversarial network (SRGAN) method, [8] enhanced SRGAN (ESRGAN) [9] is an incremental tweaking of the same generative adversarial network basis. Both methods rely on a perceptual loss function [10] to evaluate training iterations.

Related Research Articles

Template matching is a technique in digital image processing for finding small parts of an image which match a template image. It can be used for quality control in manufacturing, navigation of mobile robots, or edge detection in images.

Super-resolution imaging (SR) is a class of techniques that enhance (increase) the resolution of an imaging system. In optical SR the diffraction limit of systems is transcended, while in geometrical SR the resolution of digital imaging sensors is enhanced.

Texture synthesis is the process of algorithmically constructing a large digital image from a small digital sample image by taking advantage of its structural content. It is an object of research in computer graphics and is used in many fields, amongst others digital image editing, 3D computer graphics and post-production of films.

<span class="mw-page-title-main">Image scaling</span> Changing the resolution of a digital image

In computer graphics and digital imaging, imagescaling refers to the resizing of a digital image. In video technology, the magnification of digital material is known as upscaling or resolution enhancement.

Demosaicing, also known as color reconstruction, is a digital image processing algorithm used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA) such as a Bayer filter. It is also known as CFA interpolation or debayering.

Compressed sensing is a signal processing technique for efficiently acquiring and reconstructing a signal, by finding solutions to underdetermined linear systems. This is based on the principle that, through optimization, the sparsity of a signal can be exploited to recover it from far fewer samples than required by the Nyquist–Shannon sampling theorem. There are two conditions under which recovery is possible. The first one is sparsity, which requires the signal to be sparse in some domain. The second one is incoherence, which is applied through the isometric property, which is sufficient for sparse signals. Compressed sensing has applications in, for example, MRI where the incoherence condition is typically satisfied.

<span class="mw-page-title-main">Object detection</span> Computer technology related to computer vision and image processing

Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class in digital images and videos. Well-researched domains of object detection include face detection and pedestrian detection. Object detection has applications in many areas of computer vision, including image retrieval and video surveillance.

<span class="mw-page-title-main">Deep learning</span> Branch of machine learning

Deep learning is the subset of machine learning methods based on artificial neural networks (ANNs) with representation learning. The adjective "deep" refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

<span class="mw-page-title-main">Feature learning</span> Set of learning techniques in machine learning

In machine learning, feature learning or representation learning is a set of techniques that allows a system to automatically discover the representations needed for feature detection or classification from raw data. This replaces manual feature engineering and allows a machine to both learn the features and use them to perform a specific task.

Adversarial machine learning is the study of the attacks on machine learning algorithms, and of the defenses against such attacks. A survey from May 2020 exposes the fact that practitioners report a dire need for better protecting machine learning systems in industrial applications.

<span class="mw-page-title-main">Generative adversarial network</span> Deep learning method

A generative adversarial network (GAN) is a class of machine learning frameworks and a prominent framework for approaching generative AI. The concept was initially developed by Ian Goodfellow and his colleagues in June 2014. In a GAN, two neural networks contest with each other in the form of a zero-sum game, where one agent's gain is another agent's loss.

Neural architecture search (NAS) is a technique for automating the design of artificial neural networks (ANN), a widely used model in the field of machine learning. NAS has been used to design networks that are on par or outperform hand-designed architectures. Methods for NAS can be categorized according to the search space, search strategy and performance estimation strategy used:

<span class="mw-page-title-main">Neural style transfer</span> Type of software algorithm for image manipulation

Neural style transfer (NST) refers to a class of software algorithms that manipulate digital images, or videos, in order to adopt the appearance or visual style of another image. NST algorithms are characterized by their use of deep neural networks for the sake of image transformation. Common uses for NST are the creation of artificial artwork from photographs, for example by transferring the appearance of famous paintings to user-supplied photographs. Several notable mobile apps use NST techniques for this purpose, including DeepArt and Prisma. This method has been used by artists and designers around the globe to develop new artwork based on existent style(s).

An energy-based model (EBM) (Canonical Ensemble Learning(CEL) or Learning via Canonical Ensemble (LCE)) is an application of canonical ensemble formulation of statistical physics for learning from data problems. Approach prominently appears in generative models.

Deep learning in photoacoustic imaging

Deep learning in photoacoustic imaging combines the hybrid imaging modality of photoacoustic imaging (PA) with the rapidly evolving field of deep learning. Photoacoustic imaging is based on the photoacoustic effect, in which optical absorption causes a rise in temperature, which causes a subsequent rise in pressure via thermo-elastic expansion. This pressure rise propagates through the tissue and is sensed via ultrasonic transducers. Due to the proportionality between the optical absorption, the rise in temperature, and the rise in pressure, the ultrasound pressure wave signal can be used to quantify the original optical energy deposition within the tissue.

The Fréchet inception distance (FID) is a metric used to assess the quality of images created by a generative model, like a generative adversarial network (GAN). Unlike the earlier inception score (IS), which evaluates only the distribution of generated images, the FID compares the distribution of generated images with the distribution of a set of real images.

<span class="mw-page-title-main">Video super-resolution</span> Generating high-resolution video frames from given low-resolution ones

Video super-resolution (VSR) is the process of generating high-resolution video frames from the given low-resolution video frames. Unlike single-image super-resolution (SISR), the main goal is not only to restore more fine details while saving coarse ones, but also to preserve motion consistency.

A vision transformer (ViT) is a transformer designed for computer vision. A ViT breaks down an input image into a series of patches, serialises each patch into a vector, and maps it to a smaller dimension with a single matrix multiplication. These vector embeddings are then processed by a transformer encoder as if they were token embeddings.

Small object detection is a particular case of object detection where various techniques are employed to detect small objects in digital images and videos. "Small objects" are objects having a small pixel footprint in the input image. In areas such as aerial imagery, state-of-the-art object detection techniques under performed because of small objects.

Xiaoming Liu is a Chinese-American computer scientist and an academic. He is a Professor in the Department of Computer Science and Engineering, MSU Foundation Professor as well as Anil K. and Nandita Jain Endowed Professor of Engineering at Michigan State University.

References

  1. ↑ "Edge-Directed Interpolation" . Retrieved 19 February 2016.
  2. ↑ Xin Li; Michael T. Orchard. "NEW EDGE DIRECTED INTERPOLATION" (PDF). 2000 IEEE International Conference on Image Processing: 311. Archived from the original (PDF) on 2016-02-14. Retrieved 2016-07-03.
  3. ↑ Zhang, D.; Xiaolin Wu (2006). "An Edge-Guided Image Interpolation Algorithm via Directional Filtering and Data Fusion". IEEE Transactions on Image Processing. 15 (8): 2226–38. Bibcode:2006ITIP...15.2226Z. doi:10.1109/TIP.2006.877407. PMID   16900678. S2CID   9760560.
  4. ↑ Dengwen Zhou; Xiaoliu Shen. "Image Zooming Using Directional Cubic Convolution Interpolation" . Retrieved 13 September 2015.
  5. ↑ Shaode Yu; Rongmao Li; Rui Zhang; Mou An; Shibin Wu; Yaoqin Xie (2013). "Performance evaluation of edge-directed interpolation methods for noise-free images". arXiv: 1303.6455 [cs.CV].
  6. ↑ Johannes Kopf and Dani Lischinski (2011). "Depixelizing Pixel Art". ACM Transactions on Graphics. 30 (4): 99:1–99:8. doi:10.1145/2010324.1964994. Archived from the original on 2015-09-01. Retrieved 24 October 2012.
  7. ↑ Lim, Bee; Son, Sanghyun; Kim, Heewon; Nah, Seungjun; Kyoung Mu Lee (2017). "Enhanced Deep Residual Networks for Single Image Super-Resolution". arXiv: 1707.02921 [cs.CV].
  8. ↑ "Generative Adversarial Network and Super Resolution GAN(SRGAN)". 26 April 2020. Retrieved 26 August 2020.
  9. ↑ Wang, Xintao; Yu, Ke; Wu, Shixiang; Gu, Jinjin; Liu, Yihao; Dong, Chao; Chen Change Loy; Qiao, Yu; Tang, Xiaoou (2018). "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks". arXiv: 1809.00219 [cs.CV].
  10. ↑ "Perceptual Loss Functions". 17 May 2019. Retrieved 26 August 2020.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.