Inpainting

Last updated
Image restoration and recolorization using artificial intelligence Restoration using Artificial intelligence.jpg
Image restoration and recolorization using artificial intelligence

Inpainting is a conservation process where damaged, deteriorated, or missing parts of an artwork are filled in to present a complete image. [1] This process is commonly used in image restoration. It can be applied to both physical and digital art mediums such as oil or acrylic paintings, chemical photographic prints, sculptures, or digital images and video.

Contents

With its roots in physical artwork, such as painting and sculpture, traditional inpainting is performed by a trained art conservator who has carefully studied the artwork to determine the mediums and techniques used in the piece, potential risks of treatments, and ethical appropriateness of treatment.

History

The modern use of inpainting can be traced back to Pietro Edwards (1744–1821), Director of the Restoration of the Public Pictures in Venice, Italy. Using a scientific approach, Edwards focused his restoration efforts on the intentions of the artist. [2]

It was during the 1930 International Conference for the Study of Scientific Methods for the Examination and Preservation of Works of Art, that the modern approach to inpainting was established. Helmut Ruhemann (1891–1973), a German restorer and conservator, led the discussions on the use of inpainting in conservation. Helmut Ruhemann was a leading figure in modernizing restoration and conservation. [3] His greatest contribution to the field of conservation "was his insistence on following the methods of the original painter exactly, and on understanding the painter's artistic intention". [4] After his career of over 40 years as a conservator, Ruhemann published his treatise The Cleaning of Paintings: Problems & Potentialities in 1968. In describing his method, Ruhemann states that "The surface [of the fill] should be slightly lower than that of the surrounding paint to allow for the thickness of the inpainting...Inpainting medium should look and behave like the original medium, but must not darken with age." [5] Cesare Brandi (1906–1988) developed the teoria del restauro, the inpainting approach combining aesthetics and psychology. However, this approach was used primarily by Italian restorers and conservators, with the terminology becoming widespread in the 1990s. [6]

Technological advancements led to new applications of inpainting. Widespread use of digital techniques range from entirely automatic computerized inpainting to tools used to simulate the process manually. [7] Since the mid-1990s, the process of inpainting has evolved to include digital media. More commonly known as image or video interpolation, a form of estimation, digital inpainting includes the use of computer software that relies on sophisticated algorithms to replace lost or corrupted parts of the image data.

Ethics

In order to preserve the integrity of an original artwork, any inpainting technique or treatment applied to physical or digital work should be reversible or distinguishable from the original content of the artwork. [8] Prior to any treatments, conservators proceed according to the American Institute of Conservation of Historical and Artistic Works. [9]

There are several ethic considerations before Inpainting can be justified. Various deliberation decisions over the ethical appropriateness of the amount and type of inpainting done, resides on many factors. As most conservation treatments, inpainting's ethical questions rest mainly with authenticity, reversibility and documentation.

Any intervention to compensate for loss should be documented in treatment records and reports and should be detectable by common examination methods. Such compensation should be reversible and should not falsely modify the known aesthetic, conceptual, and physical characteristics of the cultural property, especially by removing or obscuring original material. [10]

New technologies and the aesthetic demand for perfect images without imperfections challenge conservators' ethical practices to protect the integrity of originals. [11]

Methods

Inpainting methods and techniques depend on the desired goal and type of image being treated. Treatments to fill in the gaps are different between physical and digital art. In inpainting, detailed records of the initial state of the images can help with the treatment and replicate the original closer.

Physical inpainting

Piero della Francesca - Scene after and before restoration. (The Queen of Sheba in) Adoration of the (Holy) Wood. Piero della Francesca - Scene after and before restoration - WGA17592.jpg
Piero della Francesca – Scene after and before restoration. (The Queen of Sheba in) Adoration of the (Holy) Wood.

Inpainting is rooted in the conservation and restoration of paintings. Inpainting can aim to make a visual improvement to the artwork as a whole by repairing missing or damaged parts using methods and materials equivalent to the original artist's work. [12]

Application techniques

By studying the painting methods of various artists and the composition of paints used historically, conservators are able to restore works very closely to their original visual appearance. The picture as a whole determines how to fill in the gap.

Helmut Ruhemann's inpainting techniques by Jessell have procedures to "preserve" the quality of oil and tempera paintings. [13]

Digital inpainting

Many programs are able to reconstruct missing or damaged areas of digital photographs and videos. Most widely known for use with digital images is Adobe Photoshop. Given the various abilities of the digital camera and the digitization of old photos, inpainting has become an automatic process that can be performed on digital images. The inpainting techniques can be applied to object removal, text removal, and other automatic modifications of images and videos.

In video special effects, inpainting is usually performed after video matting. They can also be observed in applications like image compression and super-resolution. In photography and cinema, it is used for film restoration to reverse, repair, or mitigate deterioration (e.g., physical damage such as cracks in photographs, scratches and dust spots in film, or chemical damage resulting in image loss; performed infrared cleaning). It can also be used for removing red-eye, the stamped date from photographs, and objects for creative effect. This technique can be used to replace any lost blocks in the coding and transmission of images, for example, in a streaming video. It can also be used to remove logos or watermarks in videos.

Digital image restoration and reconstruction Digital Image Restoration and Reconstraction.jpg
Digital image restoration and reconstruction

Deep learning neural network-based inpainting can be used for decensoring images. [14] Deep image prior-based techniques can be used for digital image inpainting, where a trained deep learning model is either unavailable or infeasible.

Three main groups of 2D image-inpainting algorithms can be found in the literature. The first one to be noted is structural (or geometric) inpainting, the second one is texture inpainting, the last one is a combination of these two techniques. They use the information of the known or non-destroyed image areas in order to fill the gap, similar to how physical images are restored.

Structural

Structural or geometric inpainting is used for smooth images that have strong, defined borders. [15] There are many different approaches to geometric inpainting, but they all come from the idea that geometry can be recovered from similar areas or domains. Bertalmio [15] proposed a method of structural inpainting that mimics how conservators address painting restoration. Bertalmio proposed that by progressively transferring similar information from the borders of an inpainting domain inwards, the gap can be filled. [16]

Textural

While structural/geometric inpainting works to repair smooth images, textural inpainting works best with images that are heavily textured. [17] Texture has a repetitive pattern which means that a missing portion cannot be restored by continuing the level lines into the gap; level lines provide a complete, stable representation of an image. [16] To repair texture in an image, one can combine frequency and spatial domain information to fill in a selected area with a desired texture. This method, while the most simple and very effective, works well when selecting a texture to be in-painted. For a texture that covers a wider area or a larger frame one would have to go through the image segmenting the areas to be in-painted and selecting the corresponding textures from throughout the image; there are programs that can help find the corresponding areas that work in a similar way as 'find and replace' works in a word processor. [18]

Combined structural and textural

Combined structural and textural inpainting approaches simultaneously try to perform texture- and structure-filling in regions of missing image information. Most parts of an image consist of texture and structure and the boundaries between image regions contain a large amount of structural information. This is the result when blending different textures together. That is why some state-of-the-art methods attempt to combine structural and textural inpainting.

A more traditional method is to use differential equations (such as Laplace's equation) with Dirichlet boundary conditions for continuity so as to create a seemingly seamless fit. This works well if missing information lies within the homogeneous portion of an object area. [19]

Other methods follow isophote directions (in an image, a contour of equal luminance), to do the inpainting. [20]

Model based inpainting follows the Bayesian approach for which missing information is best fitted or estimated from the combination of the models of the underlying images, as well as the image data actually being observed. In deterministic language, this has led to various variational inpainting models. [21]

Manual computer methods include using a clone tool to copy existing parts of the image to restore a damaged texture. Texture synthesis may also be used. [22]

Exemplar-based image inpainting attempts to automate the clone tool process. It fills "holes" in the image by searching for similar patches in a nearby source region of the image, and copying the pixels from the most similar patch into the hole. By performing the fill at the patch level as opposed to the pixel level, the algorithm reduces blurring artifacts caused by prior techniques. [23] [24]

See also

Related Research Articles

<span class="mw-page-title-main">Conservation and restoration of cultural property</span> Process of protecting cultural property

The conservation and restoration of cultural property focuses on protection and care of cultural property, including artworks, architecture, archaeology, and museum collections. Conservation activities include preventive conservation, examination, documentation, research, treatment, and education. This field is closely allied with conservation science, curators and registrars.

Texture synthesis is the process of algorithmically constructing a large digital image from a small digital sample image by taking advantage of its structural content. It is an object of research in computer graphics and is used in many fields, amongst others digital image editing, 3D computer graphics and post-production of films.

<span class="mw-page-title-main">Clone tool</span>

The clone tool, as it is known in Adobe Photoshop, Inkscape, GIMP, and Corel PhotoPaint, is used in digital image editing to replace information for one part of a picture with information from another part. In other image editing software, its equivalent is sometimes called a rubber stamp tool or a clone brush.

<span class="mw-page-title-main">Conservation science (cultural property)</span> Interdisciplinary study

With respect to cultural property, conservation science is the interdisciplinary study of the conservation of art, architecture, technical art history and other cultural works through the use of scientific inquiry. General areas of research include the technology and structure of artistic and historic works. In other words, the materials and techniques from which cultural, artistic and historic objects are made. There are three broad categories of conservation science with respect to cultural heritage: understanding the materials and techniques used by artists, study of the causes of deterioration, and improving techniques and materials for examination and treatment. Conservation science includes aspects of materials science, chemistry, physics, biology, and engineering, as well as art history and anthropology. Institutions such as the Getty Conservation Institute specialize in publishing and disseminating information relating to both tools used for and outcomes of conservation science research, as well as recent discoveries in the field.

<span class="mw-page-title-main">Conservator-restorer</span> Professional responsible for the preservation of artistic and cultural artifacts

A conservator-restorer is a professional responsible for the preservation of artistic and cultural artifacts, also known as cultural heritage. Conservators possess the expertise to preserve cultural heritage in a way that retains the integrity of the object, building or site, including its historical significance, context and aesthetic or visual aspects. This kind of preservation is done by analyzing and assessing the condition of cultural property, understanding processes and evidence of deterioration, planning collections care or site management strategies that prevent damage, carrying out conservation treatments, and conducting research. A conservator's job is to ensure that the objects in a museum's collection are kept in the best possible condition, as well as to serve the museum's mission to bring art before the public.

<span class="mw-page-title-main">Conservation and restoration of iron and steel objects</span>

Iron, steel, and ferrous metals constitute a large portion of collections in museums. The conservation and restoration of iron and steel objects is an activity dedicated to the preservation and protection of objects of historical and personal value made from iron or steel. When applied to cultural heritage this activity is generally undertaken by a conservator-restorer. Historically, objects made from iron or steel were created for religious, artistic, technical, military and domestic uses. Though it is generally not possible to completely halt deterioration of any object, the act of conservation and restoration strives to prevent and slow the deterioration of the object as well as protecting the object for future use. One of the first steps in caring for iron is to examine them and determine their state, determine if they are corroding, and consider options for treatment.

<span class="mw-page-title-main">Paintings conservator</span>

A paintings conservator is an individual responsible for protecting cultural heritage in the form of painted works of art. These individuals are most often under the employ of museums, conservation centers, or other cultural institutions. They oversee the physical care of collections, and are trained in chemistry and practical application of techniques for repairing and restoring paintings.

<span class="mw-page-title-main">Conservation and restoration of outdoor artworks</span>

The conservation and restoration of outdoor artworks is the activity dedicated to the preservation and protection of artworks that are exhibited or permanently installed outside. These works may be made of wood, stone, ceramic material, plastic, bronze, copper, or any other number of materials and may or may not be painted. When applied to cultural heritage this activity is generally undertaken by a conservator-restorer.

<span class="mw-page-title-main">Conservation and restoration of books, manuscripts, documents, and ephemera</span> Activity to extend the life of historical materials

The conservation and restoration of books, manuscripts, documents, and ephemera is an activity dedicated to extending the life of items of historical and personal value made primarily from paper, parchment, and leather. When applied to cultural heritage, conservation activities are generally undertaken by a conservator. The primary goal of conservation is to extend the lifespan of the object as well as maintaining its integrity by keeping all additions reversible. Conservation of books and paper involves techniques of bookbinding, restoration, paper chemistry, and other material technologies including preservation and archival techniques.

<span class="mw-page-title-main">Photograph conservator</span> Professional who examines photographs

A photograph conservator is a professional who examines, documents, researches, and treats photographs, including documenting the structure and condition of art works through written and photographic records, monitoring conditions of works in storage and exhibition and transit environments. This person also performs all aspects of the treatment of photographs and related artworks with adherence to the professional Code of Ethics.

<span class="mw-page-title-main">Conservation and restoration of wooden furniture</span> Preservation of heritage collections

The conservation and restoration of wooden furniture is an activity dedicated to the preservation and protection of wooden furniture objects of historical and personal value. When applied to cultural heritage this activity is generally undertaken by a conservator-restorer. Furniture conservation and restoration can be divided into two general areas: structure and finish. Structure generally relates to wood and can be divided into solid, joined, and veneered wood. The finish of furniture can be painted or transparent.

The lining of paintings is a process of conservation science and art restoration used to strengthen, flatten or consolidate oil or tempera paintings on canvas by attaching a new support to the back of the existing one. The process is sometimes referred to as relining. Most often a new support will be added directly to the back of an existing canvas. In cases of extreme decay, however, the original canvas may be completely removed and replaced. The height of the practice's use peaked in the late 19th century and in the following years its usefulness has been debated. There are many different factors that influence whether lining a painting will be successful. By paying close attention to an artwork's condition and response to treatment, conservation professionals better understand the lining process and when to apply it.

<span class="mw-page-title-main">Cultural property radiography</span>

The radiography of cultural property is the use of radiography to understand intrinsic details about objects. Most commonly this involves X-rays of paintings to reveal underdrawing, pentimenti alterations in the course of painting or by later restorers, and sometimes previous paintings on the support. Many pigments such as lead white show well in radiographs.

<span class="mw-page-title-main">Conservation and restoration of paintings</span> Preservation of heritage collections

The conservation and restoration of paintings is carried out by professional painting conservators. Paintings cover a wide range of various mediums, materials, and their supports. Painting types include fine art to decorative and functional objects spanning from acrylics, frescoes, and oil paint on various surfaces, egg tempera on panels and canvas, lacquer painting, water color and more. Knowing the materials of any given painting and its support allows for the proper restoration and conservation practices. All components of a painting will react to its environment differently, and impact the artwork as a whole. These material components along with collections care will determine the longevity of a painting. The first steps to conservation and restoration is preventive conservation followed by active restoration with the artist's intent in mind.

The conservation and restoration of ancient Greek pottery is a sub-section of the broader topic of conservation and restoration of ceramic objects. Ancient Greek pottery is one of the most commonly found types of artifacts from the ancient Greek world. The information learned from vase paintings forms the foundation of modern knowledge of ancient Greek art and culture. Most ancient Greek pottery is terracotta, a type of earthenware ceramic, dating from the 11th century BCE through the 1st century CE. The objects are usually excavated from archaeological sites in broken pieces, or shards, and then reassembled. Some have been discovered intact in tombs. Professional conservator-restorers, often in collaboration with curators and conservation scientists, undertake the conservation-restoration of ancient Greek pottery.

<span class="mw-page-title-main">Conservation and restoration of taxidermy</span>

The conservation of taxidermy is the ongoing maintenance and preservation of zoological specimens that have been mounted or stuffed for display and study. Taxidermy specimens contain a variety of organic materials, such as fur, bone, feathers, skin, and wood, as well as inorganic materials, such as burlap, glass, and foam. Due to their composite nature, taxidermy specimens require special care and conservation treatments for the different materials.

The conservation-restoration of panel paintings involves preventive and treatment measures taken by paintings conservators to slow deterioration, preserve, and repair damage. Panel paintings consist of a wood support, a ground, and an image layer. They are typically constructed of two or more panels joined together by crossbeam braces which can separate due to age and material instability caused by fluctuations in relative humidity and temperature. These factors compromise structural integrity and can lead to warping and paint flaking. Because wood is particularly susceptible to pest damage, an IPM plan and regulation of the conditions in storage and display are essential. Past treatments that have fallen out of favor because they can cause permanent damage include transfer of the painting onto a new support, planing, and heavy cradling. Today's conservators often have to remediate damage from previous restoration efforts. Modern conservation-restoration techniques favor minimal intervention that accommodates wood's natural tendency to react to environmental changes. Treatments may include applying flexible battens to minimize deformation or simply leaving distortions alone, instead focusing on preventive care to preserve the artwork in its original state.

Cultural property imaging is a necessary part of long term preservation of cultural heritage. While the physical conditions of objects will change over time, imaging serves as a way to document and represent heritage in a moment in time of the life of the item. Different methods of imaging produce results that are applicable in various circumstances. Not every method is appropriate for every object, and not every object needs to be imaged by multiple methods. In addition to preservation and conservation-related concerns, imaging can also serve to enhance research and study of cultural heritage.

The conservation and restoration of Pompeian frescoes describes the activities, methods, and techniques that have historically been and are currently being used to care for the preserved remains of the frescoes from the archeological site of Pompeii, Italy. The ancient city of Pompeii is famously known for its demise in A.D. 79 after the fatal eruption of Mount Vesuvius wiped out the population and buried the city beneath layers of compact lava material. In 1738, King Charles III or Charles of Bourbon, began explorations in Portici, Resina, Castellammare di Stabia, a Civita, where it was believed that the ancient cities of Pompeii, Stabiae, and Herculaneum were buried beneath. The first phase of the excavations at Pompeii started in 1748, which led to the first conservation and restoration efforts of the frescoes since their burial, and in 1764, open-air excavations began at Pompeii. Pompeii has a long history of excavation and restoration that began without a strong foundation or strategy. After centuries of cronyism, recurring financial shortages, and on-again-off-again restoration, the city's frescoes and structures were left in poor condition. In 1997, Pompeii was added to the UNESCO List of World Heritage Sites.

The conservation-restoration of dye diffusion transfer prints is the process undertaken by conservator-restorers of caring for and maintaining dye diffusion transfer prints to preserve their form, and the information they contain. It covers the processes that can be taken by conservators, archivists, and other museum professionals. This practice includes understanding the composition and agents of deterioration of dye diffusion transfer prints, as well as the preventive conservation and interventive conservation measures that can be taken.

References

  1. ↑ Newman, Richard (2011). Conservation and care of museum collections (1st ed.). MFA publications. p. 29. ISBN   978-0-87846-729-7.
  2. ↑ Darrow, E.J. "Pietro Edwards and the restoration of the public pictures of Venice, 1778–1819: necessity introduced these arts". Research Works Archive. University of Washington. Retrieved 29 March 2020.
  3. ↑ "Helmut Ruhemann Papers (1930-1979) | Archive | National Gallery, London". www.nationalgallery.org.uk. Retrieved 2024-03-20.
  4. ↑ Jessell, Bettina (1977). "HELMUT RUHEMANN'S INPAINTING TECHNIQUES". cool.culturalheritage.org. Retrieved 2024-03-20.
  5. ↑ Garland, Patricia, 2011, Chapter 3, Tradition of retouching Practices in America, pp 34–52, in Painting Conservation Catalog, Volume III Inpainting, The Paintings Specialty Group of the American Institute for Conservation, https://www.culturalheritage.org/docs/default-source/resource-guides/painting-conservation-catalog-volume-3-(inpainting).pdf, Retrieved November 2, 2019.
  6. ↑ Idelson, Antonia Iaccarino; Severini, Leonardo (18 June 2018). "Inpainting". The Encyclopedia of Archaeological Sciences: 1–4. doi: 10.1002/9781119188230.saseas0330 . ISBN   9780470674611.
  7. ↑ Bertalmio, Marcelo; Sapiro, Guillermo. "Image Inpainting" (PDF). University of Minnesota. Archived from the original (PDF) on 29 March 2020. Retrieved 29 March 2020.
  8. ↑ "Inpainting | Museu Nacional d'Art de Catalunya". www.museunacional.cat. 25 March 2014. Retrieved 29 March 2020.
  9. ↑ (1994) “AIC Code of Ethics and Guidelines for Practice”[1] https://www.culturalheritage.org/docs/default-source/administration/governance/code-of-ethics-and-guidelines-for-practice.pdf. Retrieved November 3, 2019.
  10. ↑ American Institute of Conservation of Historical and Artistic Works. (1994). "AIC Code of Ethics and Guidelines for Practice". https://www.nps.gov/training/tel/Guides/HPS1022_AIC_Code_of_Ethics.pdf. Retrieved March 27, 2020.
  11. ↑ Antonio laccarino Idelson. (28 June 2018). "Inpainting". The Encyclopedia of Archaeological Sciences. https://onlinelibrary.wiley.com/doi/full/10.1002/9781119188230.saseas0330 Retrieved March 26, 2020.
  12. ↑ Idelson, Antonio; Severini, Leonardo (28 June 2018). "Inpainting". The Encyclopedia of Archeological Sciences: 1–4. https://onlinelibrary.wiley.com/doi/full/10.1002/9781119188230.saseas0330. Retrieved October 30, 2019.
  13. ↑ Bettina Jessell. (1977). "Helmut Ruhemann's Inpainting Techniques". Journal of the American Institute for Conservation, JAIC 1977, Vol. 17, Number 1, Article 1 (pp. 08 - 08). Retrieved March 26, 2020.
  14. ↑ "This Researcher Created 'DeepCreamPy,' a Machine Learning Algorithm That Uncensors Hentai – Motherboard". 2018-10-31.
  15. 1 2 Bugeau, Aurelie & Marcelo Bertalmio(2011).Combining Texture Synthesis and Diffusion for Image Inpainting.Hal Archives https://hal.archives-ouvertes.fr/hal-00551587/ Retrieved November 1 2019.
  16. 1 2 Cao, Frederic et al. (2011) "Geometrically Guided Exemplar-Based Inpainting." SIAM Journal of Imaging Sciences Vol 4. no.4 pp. 1143–1179. https://perso.telecom-paristech.fr/gousseau/inpaintingSIAM.pdf reterieved November 2 2019.
  17. ↑ Bugeau, Aurelie & Marcelo Bertalmio(2011).Combining Texture Synthesis and Diffusion for Image Inpainting.Hal Archives https://hal.archives-ouvertes.fr/hal-00551587/
  18. ↑ Bertalmio, Marcelo & Guillermo Saprio."Image Inpainting" https://conservancy.umn.edu/bitstream/handle/11299/3365/1/1655.pdf
  19. ↑ Peterson, Ivars (11 May 2002). "Filling in Blanks". Science News. 161 (19): 299–300. doi:10.2307/4013521. JSTOR   4013521.
  20. ↑ Bertalmío, M.; Sapiro, G.; Caselles, V.; Ballester, C. (2000). "Image Inpainting". Proceedings of the 27th annual conference on Computer graphics and interactive techniques – SIGGRAPH '00. pp. 417–424. doi:10.1145/344779.344972. ISBN   1581132085. S2CID   308278.
  21. ↑ Chan, T. F.; Shen, J. (2001). "Mathematical Models for Local Nontexture Inpainting". SIAM J. Appl. Math. 62 (3): 1019–1043. doi:10.1137/S0036139900368844.
  22. ↑ Igehy, Homan; Pereira, Lucas (1997). "Image Replacement through Texture Synthesis". Proceedings of International Conference on Image Processing. Vol. 3. pp. 186–189. doi:10.1109/ICIP.1997.632049. ISBN   0-8186-8183-7. S2CID   5895502.
  23. ↑ Criminisi, A.; Perez, P.; Toyama, K. (2003). "Object Removal by Exemplar-Based Inpainting". 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. Vol. 2. pp. II-721–II-728. doi:10.1109/CVPR.2003.1211538. ISBN   0-7695-1900-8. S2CID   315138.
  24. ↑ Lorenzi, L.; Melgani, F.; Mercier, G. (2011). "Inpainting Strategies for Reconstruction of Missing Data in VHR Images". IEEE Geoscience and Remote Sensing Letters. 8 (5): 914–918. Bibcode:2011IGRSL...8..914L. doi:10.1109/LGRS.2011.2141112. S2CID   2119517.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.