Tone mapping

Last updated
Tone mapped high-dynamic-range (HDR) image of St. Kentigerns Roman Catholic Church in Blackpool, Lancashire, England, UK St Kentigerns Church HDR (8226826999).jpg
Tone mapped high-dynamic-range (HDR) image of St. Kentigerns Roman Catholic Church in Blackpool, Lancashire, England, UK

Tone mapping is a technique used in image processing and computer graphics to map one set of colors to another to approximate the appearance of high-dynamic-range (HDR) images in a medium that has a more limited dynamic range. Print-outs, CRT or LCD monitors, and projectors all have a limited dynamic range that is inadequate to reproduce the full range of light intensities present in natural scenes. Tone mapping addresses the problem of strong contrast reduction from the scene radiance to the displayable range while preserving the image details and color appearance important to appreciate the original scene content.

Contents

Inverse tone mapping is the inverse technique that allows to expand the luminance range, mapping a low dynamic range image into a higher dynamic range image. [1] It is notably used to upscale SDR videos to HDR videos. [2]

Background

The introduction of film-based photography created issues since capturing the enormous dynamic range of lighting from the real world on a chemically limited negative was very difficult. Early film developers attempted to remedy this issue by designing the film stocks and the print development systems that gave a desired S-shaped tone curve with slightly enhanced contrast (about 15%) in the middle range and gradually compressed highlights and shadows . The advent of the Zone System, which bases exposure on the desired shadow tones along with varying the length of time spent in the chemical developer (thus controlling highlight tones) extended the tonal range of black and white (and later, color) negative film from its native range of about seven stops to about ten. Photographers have also used dodging and burning to overcome the limitations of the print process .

The advent of digital photography gave hope for better solutions to this problem. One of the earliest algorithms employed by Land and McCann in 1971 was Retinex, inspired by theories of lightness perception .This method is inspired by the eye’s biological mechanisms of adaptation when lighting conditions are an issue. Gamut mapping algorithms were also extensively studied in the context of color printing. Computational models such as CIECAM02 or iCAM were used to predict color appearance. Despite this, if algorithms could not sufficiently map tones and colors, a skilled artist was still needed, as is the case with cinematographic movie post-processing.

Computer graphic techniques capable of rendering high-contrast scenes shifted the focus from color to luminance as the main limiting factor of display devices. Several tone mapping operators were developed to map high dynamic range (HDR) images to standard displays. More recently, this work has branched away from utilizing luminance to extend image contrast and towards other methods such as user-assisted image reproduction. Currently, image reproduction has shifted towards display-driven solutions since displays now possess advanced image processing algorithms that help adapt rendering of the image to viewing conditions, save power, up-scale color gamut and dynamic range.

Purpose and methods

The goals of tone mapping can be differently stated depending on the particular application. In some cases producing just aesthetically pleasing images is the main goal, while other applications might emphasize reproducing as many image details as possible, or maximizing the image contrast. The goal in realistic rendering applications might be to obtain a perceptual match between a real scene and a displayed image even though the display device is not able to reproduce the full range of luminance values.

Various tone mapping operators have been developed in the recent years. They all can be divided in two main types:

A simple example of global tone mapping filter is (Reinhard), where Vin is the luminance of the original pixel and Vout is the luminance of the filtered pixel. [4] This function will map the luminance Vin in the domain to a displayable output range of While this filter provides a decent contrast for parts of the image with low luminance (particularly when Vin < 1), parts of the image with higher luminance will get increasingly lower contrast as the luminance of the filtered image goes to 1. Variations on this filter are commonly used in rendering. [5]

A perhaps more useful global tone mapping method is gamma compression, which has the filter where A > 0 and 0 < γ < 1. This function will map the luminance Vin in the domain to the output range γ regulates the contrast of the image; a lower value for lower contrast. While a lower constant γ gives a lower contrast and perhaps also a duller image, it increases the exposure of underexposed parts of the image while at the same time, if A < 1, it can decrease the exposure of overexposed parts of the image enough to prevent them from being overexposed.

An even more sophisticated group of tone mapping algorithms is based on contrast or gradient domain methods, which are 'local'. Such operators concentrate on preserving contrast between neighboring regions rather than absolute value, an approach motivated by the fact that the human perception is most sensitive to contrast in images rather than absolute intensities. Those tone mapping methods usually produce very sharp images, which preserve very well small contrast details; however, this is often done at the cost of flattening an overall image contrast, and may as a side effect produce halo-like glows around dark objects. Examples of such tone mapping methods include: gradient domain high dynamic range compression and A Perceptual Framework for Contrast Processing of High Dynamic Range Images (a tone mapping is one of the applications of this framework).

Another approach to tone mapping of HDR images is inspired by the anchoring theory of lightness perception. This theory explains many characteristics of the human visual system such as lightness constancy and its failures (as in the checker shadow illusion), which are important in the perception of images. The key concept of this tone mapping method (Lightness Perception in Tone Reproduction ) is a decomposition of an HDR image into areas (frameworks) of consistent illumination and the local calculation of the lightness values. The net lightness of an image is calculated by merging of the frameworks proportionally to their strength. Particularly important is the anchoring—relating of the luminance to a known luminance, namely estimating which luminance value is perceived as white in the scene. This approach to tone mapping does not affect the local contrast and preserves the natural colors of an HDR image due to the linear handling of luminance.

One simple form of tone mapping takes a standard image (not HDR – the dynamic range already compressed) and applies unsharp masking with a large radius, which increases local contrast rather than sharpening. See unsharp masking: local contrast enhancement for details.

One of the commonly used tone mapping algorithms is the iCAM06 which is based on both the color appearance model and hierarchical mapping. After bilateral filtering, the image is broken into a base layer and a detail layer. White point adaptation and chrominance adaptation are applied to the base layer, while detail enhancement is applied to the detail layer. Eventually the two layers are merged and converted to the IPT color space. In general, this method is good but has some shortcomings, specifically in how computationally heavy the filtering method is. A proposed solution to this involves performance optimization of the filter. The base layer of the image is also converted to the RGB space for tone compression. This method also allows for more output adjustment and saturation enhancement, making it be less computationally intensive and better at reducing the overall halo effect.

Digital photography

Tone mapped HDR image of Dundas Square; Tone mapping was done as post-processing technique, using Photomatix photographic software. Yonge-Dundas Square - 2010 (HDR).jpg
Tone mapped HDR image of Dundas Square; Tone mapping was done as post-processing technique, using Photomatix photographic software.

Forms of tone mapping long precede digital photography. The manipulation of film and development process to render high contrast scenes, especially those shot in bright sunlight, on printing paper with a relatively low dynamic range, is effectively a form of tone mapping, although it is not usually called that. Local adjustment of tonality in film processing is primarily done via dodging and burning, and is particularly advocated by and associated with Ansel Adams, as described in his book The Print; see also his Zone System.

The normal process of exposure compensation, brightening shadows and altering contrast applied globally to digital images as part of a professional or serious amateur workflow is also a form of tone mapping.

However, HDR tone mapping, usually using local operators, has become increasingly popular amongst digital photographers as a post-processing technique, where several exposures at different shutter speeds are combined to produce an HDR image and a tone mapping operator is then applied to the result. There are now many examples of locally tone mapped digital images, inaccurately known as "HDR photographs", on the internet, and these are of varying quality. This popularity is partly driven by the distinctive appearance of locally tone mapped images, which many people find attractive, and partly by a desire to capture high-contrast scenes that are hard or impossible to photograph in a single exposure, and may not render attractively even when they can be captured. Although digital sensors actually capture a higher dynamic range than film, they completely lose detail in extreme highlights, clipping them to pure white, producing an unattractive result when compared with negative film, which tends to retain color and some detail in highlights.

In some cases local tone mapping is used even though the dynamic range of the source image could be captured on the target media, either to produce the distinctive appearance of a locally tone mapped image, or to produce an image closer to the photographer's artistic vision of the scene by removing sharp contrasts, which often look unattractive. In some cases, tone mapped images are produced from a single exposure which is then manipulated with conventional processing tools to produce the inputs to the HDR image generation process. This avoids the artifacts that can appear when different exposures are combined, due to moving objects in the scene or camera shake. However, when tone mapping is applied to a single exposure in this way, the intermediate image has only normal dynamic range, and the amount of shadow or highlight detail that can be rendered is only that which was captured in the original exposure.

Display devices

One of the original goals of tone mapping was to be able to reproduce a given scene or image onto a display device such that the brightness sensation of the image to a human viewer closely matches the real-world brightness sensation. However, a perfect match for this problem is never possible and thus the output image on a display is often built from a tradeoff between different image features. Choosing between features is often based on the necessary application, and given appropriate metrics for the application, one possible solution is to treat the issue as an optimization problem .

For this method, models for the Human Visual System (HVS) and the display are first generated, along with a simple tone mapping operator. The contrast distortions are weighted according to their individual visibilities approximated by the HVS. With these models, an objective function that defines the tone curve can be created and solved using a fast quadratic solver.

With the addition of filters, this method can also be extended to videos. The filters ensure that the rapid changing of the tone-curve between frames are not salient in the final output image.

Example of the imaging process

Tone mapped High dynamic range image example showing stained glass windows in south alcove of Old St Paul's, Wellington, New Zealand. Old saint pauls 1.jpg
Tone mapped High dynamic range image example showing stained glass windows in south alcove of Old St Paul's, Wellington, New Zealand.
The six individual exposures used to create the previous image. In the low exposure images, the room is dark and unclear, but the details of the windows are visible. In the high exposure images, the windows are bright and unclear, but the details of the room are revealed. HDRI-Example.jpg
The six individual exposures used to create the previous image. In the low exposure images, the room is dark and unclear, but the details of the windows are visible. In the high exposure images, the windows are bright and unclear, but the details of the room are revealed.

The images on the right show the interior of a church, a scene which has a variation in radiance much larger than that which can be displayed on a monitor or recorded by a conventional camera. The six individual exposures from the camera show the radiance of the scene in some range transformed to the range of brightnesses that can be displayed on a monitor. The range of radiances recorded in each photo is limited, so not all details can be displayed at once: for example, details of the dark church interior cannot be displayed at the same time as those of the bright stained-glass window. An algorithm is applied to the six images to recreate the high dynamic range radiance map of the original scene (a high dynamic range image). Alternatively, some higher-end consumer and specialist scientific digital cameras are able to record a high dynamic range image directly, for example with RAW images.

In the ideal case, a camera might measure luminance directly and store this in the HDR image; however, most high dynamic range images produced by cameras today are not calibrated or even proportional to luminance, due to practical reasons such as cost and time required to measure accurate luminance values it is often sufficient for artists to use multiple exposures to gain an "HDR image" which grossly approximates the true luminance signal.

The high dynamic range image is passed to a tone mapping operator, in this case a local operator, which transforms the image into a low dynamic range image suitable for viewing on a monitor. Relative to the church interior, the stained-glass window is displayed at a much lower brightness than a linear mapping between scene radiance and pixel intensity would produce. However, this inaccuracy is perceptually less important than the image detail, which can now be shown in both the window and the church interior simultaneously.

Local tone mapping technique of HDR image processing often produces a number of characteristic effects in images such as bright halos around dark objects, dark halos around bright objects, and sometimes a "cartoon-like" appearance due to extremely vivid colors and lack of large-scale color variations. These results are caused by application of geometric space distortion of captured image along with color space distortion, while only color space distortions are a tone mapping effect, and all other distortions are rather a custom filtering technique than any tone or color space mapping. So, the results of local tone mapping are often judged as perverting the nature of a documentalist photographic image and far from photographic realism.

Not all tone mapped images are visually distinctive. Reducing dynamic range with tone mapping is often useful in bright sunlit scenes, where the difference in intensity between direct illumination and shadow is great. In these cases the global contrast of the scene is reduced, but the local contrast maintained, while the image as a whole continues to look natural. Use of tone mapping in this context may not be apparent from the final image:

Tone mapping can also produce distinctive visual effects in the final image, such as the visible halo around the tower in the Cornell Law School image below. It can be used to produce these effects even when the dynamic range of the original image is not particularly high. Halos in images come about because the local tone mapping operator will brighten areas around dark objects, to maintain the local contrast in the original image, which fools the human visual system into perceiving the dark objects as being dark, even if their actual luminance is the same as that of areas of the image perceived as being bright. Usually this effect is subtle, but if the contrasts in the original image are extreme, or the photographer deliberately sets the luminance gradient to be very steep, the halos become visible.

See also

Related Research Articles

<span class="mw-page-title-main">Halftone</span> Printing process

Halftone is the reprographic technique that simulates continuous-tone imagery through the use of dots, varying either in size or in spacing, thus generating a gradient-like effect. "Halftone" can also be used to refer specifically to the image that is produced by this process.

Gamma correction or gamma is a nonlinear operation used to encode and decode luminance or tristimulus values in video or still image systems. Gamma correction is, in the simplest cases, defined by the following power-law expression:

<span class="mw-page-title-main">Exposure (photography)</span> Amount of light captured by a camera

In photography, exposure is the amount of light per unit area reaching a frame of photographic film or the surface of an electronic image sensor. It is determined by shutter speed, lens F-number, and scene luminance. Exposure is measured in units of lux-seconds, and can be computed from exposure value (EV) and scene luminance in a specified region.

<span class="mw-page-title-main">Multi-exposure HDR capture</span> Technique to capture HDR images and videos

In photography and videography, multi-exposure HDR capture is a technique that creates high dynamic range (HDR) images by taking and combining multiple exposures of the same subject matter at different exposure levels. Combining multiple images in this way results in an image with a greater dynamic range than what would be possible by taking one single image. The technique can also be used to capture video by taking and combining multiple exposures for each frame of the video. The term "HDR" is used frequently to refer to the process of creating HDR images from multiple exposures. Many smartphones have an automated HDR feature that relies on computational imaging techniques to capture and combine multiple exposures.

<span class="mw-page-title-main">High-dynamic-range rendering</span> Rendering of computer graphics scenes by using lighting calculations done in high-dynamic-range

High-dynamic-range rendering, also known as high-dynamic-range lighting, is the rendering of computer graphics scenes by using lighting calculations done in high dynamic range (HDR). This allows preservation of details that may be lost due to limiting contrast ratios. Video games and computer-generated movies and special effects benefit from this as it creates more realistic scenes than with more simplistic lighting models.

Shadow and highlight enhancement refers to an image processing technique used to correct exposure.

Logluv TIFF is an encoding used for storing high-dynamic-range imaging data inside a TIFF image. It was originally developed by Greg Ward for storing HDR-output of his Radiance-photonmapper at a time where storage space was a crucial factor. Its implementation in TIFF also allowed the combination with image-compression algorithms without great programming effort. As such it has to be considered a smart compromise between the imposed limitations. It is slightly related to RGBE, the most successful HDRI storage format, an earlier invention of Greg Ward.

High dynamic range (HDR), also known as wide dynamic range, extended dynamic range, or expanded dynamic range, is a dynamic range higher than usual.

<span class="mw-page-title-main">Exposing to the right</span> Photographic technique

In digital photography, exposing to the right (ETTR) is the technique of adjusting the exposure of an image as high as possible at base ISO to collect the maximum amount of light and thus get the optimum performance out of the digital image sensor.

In the theory of photography, tone reproduction is the mapping of scene luminance and color to print reflectance or display luminance, with the aim of subjectively "properly" reproducing brightness and "brightness differences".

The term post-processing is used in the video/film business for quality-improvement image processing methods used in video playback devices, such as stand-alone DVD-Video players; video playing software; and transcoding software. It is also commonly used in real-time 3D rendering to add additional effects.

Image quality can refer to the level of accuracy with which different imaging systems capture, process, store, compress, transmit and display the signals that form an image. Another definition refers to image quality as "the weighted combination of all of the visually significant attributes of an image". The difference between the two definitions is that one focuses on the characteristics of signal processing in different imaging systems and the latter on the perceptual assessments that make an image pleasant for human viewers.

<span class="mw-page-title-main">HDR PhotoStudio</span>

HDR PhotoStudio is a discontinued high dynamic range (HDR) graphics application developed by Unified Color for the Windows and macOS operating systems. In addition to being a HDR-merge application, HDR PhotoStudio offered a set of image editing operations that worked in its dynamic range, human color range (gamut), and in high precision. It also had a Color Integrity feature that enabled preserving an image's color tone during image editing operations — for example changing an image's contrast would not change its chromatic data. This problem is usually referred to as "color shift".

<span class="mw-page-title-main">Exposure fusion</span>

In image processing, computer graphics, and photography, exposure fusion is a technique for blending multiple exposures of the same scene (bracketing) into a single image. As in high dynamic range imaging, the goal is to capture a scene with a higher dynamic range than the camera is capable of capturing with a single exposure.

<span class="mw-page-title-main">Luminance HDR</span>

Luminance HDR, formerly Qtpfsgui, is graphics software used for the creation and manipulation of high-dynamic-range images. Released under the terms of the GPL, it is available for Linux, Windows and Mac OS X. Luminance HDR supports several High Dynamic Range (HDR) as well as Low Dynamic Range (LDR) file formats.

A color appearance model (CAM) is a mathematical model that seeks to describe the perceptual aspects of human color vision, i.e. viewing conditions under which the appearance of a color does not tally with the corresponding physical measurement of the stimulus source.

Aurora HDR is photographic software developed by Skylum Software for Mac OS X and Windows. It attempted to create a powerful and simple yet fast HDR photo editing software.

High-dynamic-range television (HDR-TV) is a technology that uses high dynamic range (HDR) to improves the quality of display signals. It is contrasted with the retroactively-named standard dynamic range (SDR). HDR changes the way the luminance and colors of videos and images are represented in the signal, and allows brighter and more detailed highlight representation, darker and more detailed shadows, and more intense colors.

<span class="mw-page-title-main">EasyHDR</span>

EasyHDR is a High Dynamic Range (HDR) image processing software that merges differently exposed photographs into HDR radiance map and tone maps them. With a single photograph, easyHDR can be used to increase contrast and refresh colors.

JPEG XT is an image compression standard which specifies backward-compatible extensions of the base JPEG standard.

References

  1. ^ Livingstone, M. 2002. "Vision and Art: The Biology of Seeing." Harry N Abrams
  2. ^ Hunt, R. 2004. "The Reproduction of Colour in Photography, Printing and Television: 6th Edition." John Wiley & Sons.
  3. ^ Adams, A. 1981. "The Print, The Ansel Adams Photography Series 3." New York Graphic Society
  4. ^ Land, E. H., and McCann, J. J. 1971. "Lightness and the retinex theory." Journal of the Optical Society of America 61, 1, 1–11.
  5. ^ Kate Devlin, Alan Chalmers, Alexander Wilkie, Werner Purgathofer. "STAR Report on Tone Reproduction and Physically Based Spectral Rendering" in Eurographics 2002. DOI: 10.1145/1073204.1073242
  6. ^ Raanan Fattal, Dani Lischinski, Michael Werman. "Gradient Domain High Dynamic Range Compression"
  7. ^ Rafal Mantiuk, Karol Myszkowski, Hans-Peter Seidel. "A Perceptual Framework for Contrast Processing of High Dynamic Range Images"
  8. ^ Alan Gilchrist. "An Anchoring Theory of Lightness Perception".
  9. ^ Grzegorz Krawczyk, Karol Myszkowski, Hans-Peter Seidel. "Lightness Perception in Tone Reproduction for High Dynamic Range Images"
  10. ^ Fairchild, M. D., Johnson, G.M.: ‘The iCAM framework for image appearance, differences and quality’. J Electron. Imaging, 2004
  11. ^ Xiao, J., Li, W., Liu, G., Shaw, S., & Zhang, Y. (n.d.). Hierarchical tone mapping based on image color appearance model.
  12. ^ Mantiuk, R., Daly, S., & Kerofsky, L. (n.d.). Display Adaptive Tone Mapping. http://resources.mpi-inf.mpg.de/hdr/datmo/mantiuk08datm.pdf
  13. ^ https://web.archive.org/web/20150206044300/http://docs.opencv.org/trunk/doc/tutorials/photo/hdr_imaging/hdr_imaging.html
  14. ^ Durand and Julie Dorsey, “Fast Bilateral Filtering for the Display of High-Dynamic-Range Images”. ACM Transactions on Graphics, 2002, 21, 3, 257 - 266. https://people.csail.mit.edu/fredo/PUBLI/Siggraph2002/DurandBilateral.pdf
  1. Banterle, Francesco; Ledda, Patrick; Debattista, Kurt; Chalmers, Alan (2006-11-29). "Inverse tone mapping". Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia. GRAPHITE '06. New York, NY, USA: Association for Computing Machinery. pp. 349–356. doi:10.1145/1174429.1174489. ISBN   978-1-59593-564-9. S2CID   5417678.
  2. "Inverse tone mapping - upscaling SDR content to HDR". Dolby. 2021-06-18. Retrieved 2022-04-06.
  3. G. Qiu et al, "Tone Mapping for HDR Image using Optimization-A New Closed Form Solution", Proc. ICPR 2006, 18th International Conference on Pattern Recognition, vol.1, pp.996-999
  4. Reinhard, Erik (2002). "Photographic tone reproduction for digital images" (PDF). ACM Transactions on Graphics. 21 (3): 267–276. doi:10.1145/566654.566575.
  5. Taylor, Matt. "Tone Mapping" . Retrieved 8 August 2021.

Tone mapping algorithms