Seam carving

Last updated
Original image to be made narrower Broadway tower edit.jpg
Original image to be made narrower
Scaling is undesirable because the castle is distorted. Broadway tower edit scale.png
Scaling is undesirable because the castle is distorted.
Cropping is undesirable because part of the castle is removed. Broadway tower edit cropped.png
Cropping is undesirable because part of the castle is removed.
Seam carving Broadway tower edit Seam Carving.png
Seam carving

Seam carving (or liquid rescaling) is an algorithm for content-aware image resizing, developed by Shai Avidan, of Mitsubishi Electric Research Laboratories (MERL), and Ariel Shamir, of the Interdisciplinary Center and MERL. It functions by establishing a number of seams (paths of least importance) in an image and automatically removes seams to reduce image size or inserts seams to extend it. Seam carving also allows manually defining areas in which pixels may not be modified, and features the ability to remove whole objects from photographs.

Contents

The purpose of the algorithm is image retargeting, which is the problem of displaying images without distortion on media of various sizes (cell phones, projection screens) using document standards, like HTML, that already support dynamic changes in page layout and text but not images. [1]

Image Retargeting was invented by Vidya Setlur, Saeko Takage, Ramesh Raskar, Michael Gleicher and Bruce Gooch in 2005. [2] The work by Setlur et al. won the 10-year impact award in 2015[ where? ].

Seams

Seams can be either vertical or horizontal. A vertical seam is a path of pixels connected from top to bottom in an image with one pixel in each row. [1] A horizontal seam is similar with the exception of the connection being from left to right. The importance/energy function values a pixel by measuring its contrast with its neighbor pixels.

Process

The below example describes the process of seam carving:

StepImage
1) Start with an image.
BroadwayTowerSeamCarvingA.png
2) Calculate the weight/density/energy of each pixel. This can be done by various algorithms: gradient magnitude, entropy, visual saliency, eye-gaze movement. [1] Here we use gradient magnitude.
BroadwayTowerSeamCarvingB.png
3) From the energy, make a list of seams. Seams are ranked by energy, with low energy seams being of least importance to the content of the image. Seams can be calculated via the dynamic programming approach below.
BroadwayTowerSeamCarvingC.png
4) Remove low-energy seams as needed.
BroadwayTowerSeamCarvingF.png
5) Final image.
BroadwayTowerSeamCarvingE.png

The seams to remove depends only on the dimension (height or width) one wants to shrink. It is also possible to invert step 4 so the algorithm enlarges in one dimension by copying a low energy seam and averaging its pixels with its neighbors. [1]

Computing seams

Computing a seam consists of finding a path of minimum energy cost from one end of the image to another. This can be done via Dijkstra's algorithm, dynamic programming, greedy algorithm or graph cuts among others. [1]

Dynamic programming

Dynamic programming is a programming method that stores the results of sub-calculations in order to simplify calculating a more complex result. Dynamic programming can be used to compute seams. If attempting to compute a vertical seam (path) of lowest energy, for each pixel in a row we compute the energy of the current pixel plus the energy of one of the three possible pixels above it.

The images below depict a DP process to compute one optimal seam. [1] Each square represents a pixel, with the top-left value in red representing the energy value of that pixel. The value in black represents the cumulative sum of energies leading up to and including that pixel.

The energy calculation is trivially parallelized for simple functions. The calculation of the DP array can also be parallelized with some interprocess communication. However, the problem of making multiple seams at the same time is harder for two reasons: the energy needs to be regenerated for each removal for correctness and simply tracing back multiple seams can form overlaps. Avidan 2007 computes all seams by removing each seam iteratively and storing an "index map" to record all the seams generated. The map holds a "nth seam" number for each pixel on the image, and can be used later for size adjustment. [1]

If one ignores both issues however, a greedy approximation for parallel seam carving is possible. To do so, one starts with the minimum-energy pixel at one end, and keep choosing the minimum energy path to the other end. The used pixels are marked so that they are not picked again. [3] Local seams can also be computed for smaller parts of the image in parallel for a good approximation. [4]

Issues

  1. The algorithm may need user-provided information to reduce errors. This can consist of painting the regions which are to be preserved. With human faces it is possible to use face detection.
  2. Sometimes the algorithm, by removing a low energy seam, may end up inadvertently creating a seam of higher energy. The solution to this is to simulate a removal of a seam, and then check the energy delta to see if the energy increases (forward energy). If it does, prefer other seams instead. [5]

Implementations

Interactive SVG demonstrating seam-carving using ImageMagick's liquid-rescale function. In the SVG file, hover over the percentages to compare the original image (top), its width rescaled to the percentage using seam-carving (middle), and rescaled to the same size using interpolation (bottom). Broadway tower seam carving interactive.svg
Interactive SVG demonstrating seam-carving using ImageMagick's liquid-rescale function. In the SVG file, hover over the percentages to compare the original image (top), its width rescaled to the percentage using seam-carving (middle), and rescaled to the same size using interpolation (bottom).
Interactive SVG demonstrating seam-carving using ImageMagick's liquid-rescale function. In the SVG file, hover over the percentages as above. Note that the faces are affected less than their surroundings. Creation of Adam seam carving interactive.svg
Interactive SVG demonstrating seam-carving using ImageMagick's liquid-rescale function. In the SVG file, hover over the percentages as above. Note that the faces are affected less than their surroundings.

Adobe Systems acquired a non-exclusive license to seam carving technology from MERL, [6] and implemented it as a feature in Photoshop CS4, where it is called Content Aware Scaling. [7] As the license is non-exclusive, other popular computer graphics applications (e. g. GIMP, digiKam, and ImageMagick) as well as some stand-alone programs (e. g. iResizer) [8] also have implementations of this technique, some of which are released as free and open source software. [9] [10] [11]

Improvements and extensions

A 2010 review of eight image retargeting methods found that seam carving produced output that was ranked among the worst of the tested algorithms. It was, however, a part of one of the highest-ranking algorithms: the multi-operator extension mentioned above (combined with cropping and scaling). [15]

See also

Related Research Articles

<span class="mw-page-title-main">Rendering (computer graphics)</span> Process of generating an image from a model

Rendering or image synthesis is the process of generating a photorealistic or non-photorealistic image from a 2D or 3D model by means of a computer program. The resulting image is referred to as the render. Multiple models can be defined in a scene file containing objects in a strictly defined language or data structure. The scene file contains geometry, viewpoint, texture, lighting, and shading information describing the virtual scene. The data contained in the scene file is then passed to a rendering program to be processed and output to a digital image or raster graphics image file. The term "rendering" is analogous to the concept of an artist's impression of a scene. The term "rendering" is also used to describe the process of calculating effects in a video editing program to produce the final video output.

<span class="mw-page-title-main">Flood fill</span> Algorithm in computer graphics to add color or texture

Flood fill, also called seed fill, is a flooding algorithm that determines and alters the area connected to a given node in a multi-dimensional array with some matching attribute. It is used in the "bucket" fill tool of paint programs to fill connected, similarly-colored areas with a different color, and in games such as Go and Minesweeper for determining which pieces are cleared. A variant called boundary fill uses the same algorithms but is defined as the area connected to a given node that does not have a particular attribute.

<span class="mw-page-title-main">Radiosity (computer graphics)</span> Computer graphics rendering method using diffuse reflection

In 3D computer graphics, radiosity is an application of the finite element method to solving the rendering equation for scenes with surfaces that reflect light diffusely. Unlike rendering methods that use Monte Carlo algorithms, which handle all types of light paths, typical radiosity only account for paths which leave a light source and are reflected diffusely some number of times before hitting the eye. Radiosity is a global illumination algorithm in the sense that the illumination arriving on a surface comes not just directly from the light sources, but also from other surfaces reflecting light. Radiosity is viewpoint independent, which increases the calculations involved, but makes them useful for all viewpoints.

<span class="mw-page-title-main">Ray tracing (graphics)</span> Rendering method

In 3-D computer graphics, ray tracing is a technique for modeling light transport for use in a wide variety of rendering algorithms for generating digital images.

<span class="mw-page-title-main">Image segmentation</span> Partitioning a digital image into segments

In digital image processing and computer vision, image segmentation is the process of partitioning a digital image into multiple image segments, also known as image regions or image objects. The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze. Image segmentation is typically used to locate objects and boundaries in images. More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.

<span class="mw-page-title-main">Volume rendering</span> Representing a 3D-modeled object or dataset as a 2D projection

In scientific visualization and computer graphics, volume rendering is a set of techniques used to display a 2D projection of a 3D discretely sampled data set, typically a 3D scalar field.

The light field is a vector function that describes the amount of light flowing in every direction through every point in space. The space of all possible light rays is given by the five-dimensional plenoptic function, and the magnitude of each ray is given by its radiance. Michael Faraday was the first to propose that light should be interpreted as a field, much like the magnetic fields on which he had been working. The phrase light field was coined by Andrey Gershun in a classic 1936 paper on the radiometric properties of light in three-dimensional space.

<span class="mw-page-title-main">3D display</span> Display device

A 3D display is a display device capable of conveying depth to the viewer. Many 3D displays are stereoscopic displays, which produce a basic 3D effect by means of stereopsis, but can cause eye strain and visual fatigue. Newer 3D displays such as holographic and light field displays produce a more realistic 3D effect by combining stereopsis and accurate focal length for the displayed content. Newer 3D displays in this manner cause less visual fatigue than classical stereoscopic displays.

The Saffron Type System is a system for rendering high-quality scalable type on digital displays. It was developed by Mitsubishi Electric Research Laboratories, and is built on a core of adaptively-sampled distance field (ADF) technology. Saffron has been licensed to Adobe and Monotype and is shipping in numerous products such as the Adobe Flash Player and Amazon Kindle. Saffron has been implemented in both software and hardware.

In parallel computing, an embarrassingly parallel workload or problem is one where little or no effort is needed to separate the problem into a number of parallel tasks. This is often the case where there is little or no dependency or need for communication between those parallel tasks, or for results between them.

<span class="mw-page-title-main">Pixel-art scaling algorithms</span> Upscaling filters for pixel art graphics

Pixel art scaling algorithms are graphical filters that enhance hand-drawn 2D pixel art graphics. The re-scaling of pixel art is a specialist sub-field of image rescaling.

Texture synthesis is the process of algorithmically constructing a large digital image from a small digital sample image by taking advantage of its structural content. It is an object of research in computer graphics and is used in many fields, amongst others digital image editing, 3D computer graphics and post-production of films.

In computer graphics, per-pixel lighting refers to any technique for lighting an image or scene that calculates illumination for each pixel on a rendered image. This is in contrast to other popular methods of lighting such as vertex lighting, which calculates illumination at each vertex of a 3D model and then interpolates the resulting values over the model's faces to calculate the final per-pixel color values.

<span class="mw-page-title-main">Image scaling</span> Changing the resolution of a digital image

In computer graphics and digital imaging, imagescaling refers to the resizing of a digital image. In video technology, the magnification of digital material is known as upscaling or resolution enhancement.

<span class="mw-page-title-main">Livewire Segmentation Technique</span>

Livewire, is a segmentation technique which allows a user to select regions of interest to be extracted quickly and accurately, using simple mouse clicks. It is based on the lowest cost path algorithm, by Edsger W. Dijkstra. Firstly convolve the image with a Sobel filter to extract edges. Each pixel of the resulting image is a vertex of the graph and has edges going to the 4 pixels around it, as up, down, left, right. The edge costs are defined based on a cost function. In 1995, Eric N. Mortensen and William A. Barrett made some extension work on livewire segmentation tool, which is known as Intelligent Scissors.

<span class="mw-page-title-main">ImageMagick</span> Free and open-source image manipulation software

ImageMagick, invoked from the command line as magick, is a free and open-source cross-platform software suite for displaying, creating, converting, modifying, and editing raster images. ImageMagick was created by John Cristy in 1987, it can read and write over 200 image file formats. It is widely used in open-source applications.

As applied in the field of computer vision, graph cut optimization can be employed to efficiently solve a wide variety of low-level computer vision problems, such as image smoothing, the stereo correspondence problem, image segmentation, object co-segmentation, and many other computer vision problems that can be formulated in terms of energy minimization. Many of these energy minimization problems can be approximated by solving a maximum flow problem in a graph. Under most formulations of such problems in computer vision, the minimum energy solution corresponds to the maximum a posteriori estimate of a solution. Although many computer vision algorithms involve cutting a graph, the term "graph cuts" is applied specifically to those models which employ a max-flow/min-cut optimization.

Matthew Brand is a scientist and artist based in Berlin, Germany. Brand's research focuses on mathematical and computational models of perception, learning, and control, in which each is treated as an optimization problem. He is best known for working on the following topics:

<span class="mw-page-title-main">Ramesh Raskar</span>

Ramesh Raskar is a Massachusetts Institute of Technology associate professor and head of the MIT Media Lab's Camera Culture research group. Previously he worked as a senior research scientist at Mitsubishi Electric Research Laboratories (MERL) during 2002 to 2008. He holds 132 patents in computer vision, computational health, sensors and imaging. He received the $500K Lemelson–MIT Prize in 2016. The prize money will be used for launching REDX.io, a group platform for co-innovation in Artificial Intelligence. He is well known for inventing EyeNetra, EyeCatra and EyeSelfie, Femto-photography and his TED talk for cameras to see around corners.

<span class="mw-page-title-main">Saliency map</span>

In computer vision, a saliency map is an image that highlights the region on which people's eyes focus first. The goal of a saliency map is to reflect the degree of importance of a pixel to the human visual system. For example, in this image, a person first looks at the fort and light clouds, so they should be highlighted on the saliency map. Saliency maps engineered in artificial or computer vision are typically not the same as the actual saliency map constructed by biological or natural vision.

References

  1. 1 2 3 4 5 6 7 Avidan, Shai; Shamir, Ariel (July 2007). "Seam carving for content-aware image resizing". ACM SIGGRAPH 2007 papers. p. 10. doi: 10.1145/1275808.1276390 . ISBN   978-1-4503-7836-9.
  2. Vidya Setlur; Saeko Takage; Ramesh Raskar; Michael Gleicher; Bruce Gooch (December 2005). "Automatic image retargeting". Proceedings of the 4th international conference on Mobile and ubiquitous multimedia - MUM '05. pp. 59–68. doi: 10.1145/1149488.1149499 . ISBN   0-473-10658-2.
  3. Bist; Palakkode (2016). "Parallel Seam Carving". www.andrew.cmu.edu.
  4. 1 2 Chen-Kuo Chiang; Shu-Fan Wang; Yi-Ling Chen; Shang-Hong Lai (November 2009). "Fast JND-Based Video Carving With GPU Acceleration for Real-Time Video Retargeting". IEEE Transactions on Circuits and Systems for Video Technology. 19 (11): 1588–1597. doi:10.1109/TCSVT.2009.2031462. S2CID   15124131.
  5. 1 2 Improved Seam Carving for Video Retargeting. Michael Rubinstein, Ariel Shamir, Shai Avidan. SIGGRAPH 2008.
  6. Mitsubishi Electric press release, Business Wire, December 16, 2008.
  7. Adobe Photoshop CS4 new feature list.
  8. iResizer Content aware image resizing software by Teorex
  9. Liquid Rescale, seam carving plug-in for GIMP
  10. Announcement of inclusion in digiKam
  11. Seam carving capability included in ImageMagick
  12. "Improved seam carving with forward energy".
  13. Multi-operator Media Retargeting. Michael Rubinstein, Ariel Shamir, Shai Avidan. SIGGRAPH 2009.
  14. Real-time content-aware image resizing Science in China Series F: Information Sciences, 2009 SCIENCE IN CHINA PRESS. Archived July 7, 2011, at the Wayback Machine
  15. Rubinstein, Michael; Gutierrez, Diego; Sorkine, Olga; Shamir, Ariel (2010). "A Comparative Study of Image Retargeting" (PDF). ACM Transactions on Graphics. 29 (5): 1–10. doi:10.1145/1882261.1866186. See also the RetargetMe benchmark.