Chroma key compositing, or chroma keying, is a visual-effects and post-production technique for compositing (layering) two or more images or video streams together based on color hues (chroma range). The technique has been used in many fields to remove a background from the subject of a photo or video – particularly the newscasting, motion picture, and video game industries. A color range in the foreground footage is made transparent, allowing separately filmed background footage or a static image to be inserted into the scene. The chroma keying technique is commonly used in video production and post-production. This technique is also referred to as color keying, color separation overlay (CSO; primarily by the BBC [3] ), or by various terms for specific colour-related variants such as green screen or blue screen; chroma keying can be done with backgrounds of any color that are uniform and distinct, but green and blue backgrounds are more commonly used because they differ most distinctly in hue from any human skin colour. No part of the subject being filmed or photographed may duplicate the color used as the backing, or the part may be erroneously identified as part of the backing. [4]
It is commonly used for live weather forecast broadcasts in which a news presenter is seen standing in front of a CGI map instead of a large blue or green background. Chroma keying is also common in the entertainment industry for visual effects in movies and video games. Rotoscopy may instead be carried out on subjects that are not in front of a green (or blue) screen. Motion tracking can also be used in conjunction with chroma keying, such as to move the background as the subject moves.
Prior to the introduction of travelling mattes and optical printing, double exposure was used to introduce elements into a scene which were not present in the initial exposure. This was done using black draping where a green screen would be used today. George Albert Smith first used this approach in 1898. In 1903, The Great Train Robbery by Edwin S. Porter used double exposure to add background scenes to windows which were black when filmed on set, using a garbage matte to expose only the window areas. [5]
In order to have figures in one exposure actually move in front of a substituted background in the other, a travelling matte was needed, to occlude the correct portion of the background in each frame. In 1918 Frank Williams patented a travelling matte technique, again based on using a black background. This was used in many films, such as The Invisible Man . [6] : 4
In the 1920s, Walt Disney used a white backdrop to include human actors with cartoon characters and backgrounds in his Alice Comedies . [6] : 5
The blue screen method was developed in the 1930s at RKO Radio Pictures. At RKO, Linwood Dunn used an early version of the travelling matte to create "wipes" – where there were transitions like a windshield wiper in films such as Flying Down to Rio (1933). Credited to Larry Butler, a scene featuring a genie escaping from a bottle was the first use of a proper bluescreen process to create a travelling matte for The Thief of Bagdad (1940), which won the Academy Award for Best Special Effects that year. In 1950, Warner Brothers employee and ex-Kodak researcher Arthur Widmer began working on an ultraviolet travelling matte process. He also began developing bluescreen techniques: one of the first films to use them was the 1958 adaptation of the Ernest Hemingway novella, The Old Man and the Sea , starring Spencer Tracy. [7]
The name "Chroma-Key" was RCA's trade name for the process, as used on its NBC television broadcasts, incorporating patents granted to RCA's Albert N. Goldsmith. [8] A very early broadcast use was NBC's George Gobel Show in fall 1957. [9]
Petro Vlahos was awarded an Academy Award for his refinement of these techniques in 1964. His technique exploits the fact that most objects in real-world scenes have a color whose blue-color component is similar in intensity to their green-color component. Zbigniew Rybczyński also contributed to bluescreen technology. An optical printer with two projectors, a film camera and a "beam splitter", was used to combine the actor in front of a blue screen together with the background footage, one frame at a time. In the early 1970s, American and British television networks began using green backdrops instead of blue for their newscasts. During the 1980s, minicomputers were used to control the optical printer. For the film The Empire Strikes Back , Richard Edlund created a "quad optical printer" that accelerated the process considerably and saved money. He received a special Academy Award for his innovation.
For decades, travelling matte shots had to be done "locked-down", so that neither the matted subject nor the background could shift their camera perspective at all. Later, computer-timed, motion-control cameras alleviated this problem, as both the foreground and background could be filmed with the same camera moves.
Meteorologists on television often use a field monitor, to the side of the screen, to see where they are putting their hands against the background images. A newer technique is to project a faint image onto the screen.
Some films make heavy use of chroma key to add backgrounds that are constructed entirely using computer-generated imagery (CGI). Performances from different takes can be composited together, which allows actors to be filmed separately and then placed together in the same scene. Chroma key allows performers to appear to be in any location without leaving the studio.
Advances in computer technology have simplified the incorporation of motion into composited shots, even when using handheld cameras. Reference points such as a painted grid, X's marked with tape, or equally spaced tennis balls attached to the wall, can be placed onto the colored background to serve as markers. In post-production, a computer can use these markers to compute the camera's position and thus render an image that matches the perspective and movement of the foreground perfectly. Modern advances in software and computational power have eliminated the need to accurately place the markers — the software figures out their position in space; a potential disadvantage of this is that it requires camera movement, possibly contributing to modern cinematographic techniques whereby the camera is always in motion.
The principal subject is filmed or photographed against a background consisting of a single color or a relatively narrow range of colours, usually blue or green because these colours are considered to be the furthest away from skin tone. [4] The portions of the video which match the pre-selected color are replaced by the alternate background video. This process is commonly known as "keying", "keying out" or simply a "key".
Green is used as a backdrop for TV and electronic cinematography more than any other color because television weather presenters tended to wear blue suits. When chroma keying first came into use in television production, the blue screen that was then the norm in the movie industry was used out of habit, until other practical considerations caused the television industry to move from blue to green screens. Broadcast-quality color television cameras use separate red, green, and blue image sensors, and early analog TV chroma keyers required RGB component video to work reliably. From a technological perspective it was equally possible to use the blue or green channel, but because blue clothing was an ongoing challenge, the green screen came into common use. Newscasters sometimes forget the chroma key dress code, and when the key is applied to clothing of the same color as the background, the person would seem to disappear into the key. Because green clothing is less common than blue, it soon became apparent that it was easier to use a green matte screen than it was to constantly police the clothing choices of on-air talent. Also, because the human eye is more sensitive to green wavelengths, which lie in the middle of the visible light spectrum, the green analog video channel typically carried more signal strength, giving a better signal to noise ratio compared to the other component video channels, so green screen keys could produce the cleanest key. In the digital television and cinema age, much of the tweaking that was required to make a good quality key has been automated. However, the one constant that remains is some level of color coordination to keep foreground subjects from being keyed out. [10]
Before electronic chroma keying, compositing was done on (chemical) film. The camera colour negative was printed onto high-contrast black and white negative, using either a filter or the high contrast film's colour sensitivity to expose only blue (and higher) frequencies. Blue light only shines through the colour negative where there is not blue in the scene, so this left the film clear where the blue screen was, and opaque elsewhere, except it also produced clear for any white objects (since they also contained blue). Removing these spots could be done by a suitable double-exposure with the colour positive (thus turning any area containing red or green opaque), and many other techniques. The result was film that was clear where the blue screen was, and opaque everywhere else. This is called a female matte, similar to an alpha matte in digital keying. Copying this film onto another high-contrast negative produced the opposite male matte. The background negative was then packed with the female matte and exposed onto a final strip of film, then the camera negative was packed with the male matte and was double-printed onto this same film. These two images combined creates the final effect.
The most important factor for a key is the colour separation of the foreground (the subject) and background (the screen) – a blue screen will be used if the subject is predominantly green (for example plants), despite the camera being more sensitive to green light.
In analog television, colour is represented by the phase of the chroma subcarrier relative to a reference oscillator. Chroma key is achieved by comparing the phase of the video to the phase corresponding to the pre-selected colour. In-phase portions of the video are replaced by the alternate background video.[ citation needed ]
In digital colour TV, colour is represented by three numbers (red, green, blue intensity levels). Chroma key is achieved by a simple numerical comparison between the video and the pre-selected colour. If the colour at a particular point on the screen matches (either exactly, or in a range), then the video at that point is replaced by the alternate background.
In order to create an illusion that characters and objects filmed are present in the intended background scene, the lighting in the two scenes must be a reasonable match. For outdoor scenes, overcast days create a diffuse, evenly coloured light which can be easier to match in the studio, whereas direct sunlight needs to be matched in both direction and overall colour based on time of day.
A studio shot taken in front of a green screen will naturally have ambient light the same colour as the screen, due to its light scattering. This effect is known as spill. [6] : p20 This can look unnatural or cause portions of the characters to disappear, so must be compensated for, or avoided by using a larger screen placed far from the actors. [11]
The depth of field used to record the scene in front of the coloured screen should match that of the background. This can mean recording the actors with a larger depth of field than normal. [12]
A chroma key subject must avoid wearing clothes which are similar in colour to the chroma key colour(s) (unless intentional e.g., wearing a green top to make it appear that the subject has no body), because the clothing may be replaced with the background image/video. An example of intentional use of this is when an actor wears a blue covering over a part of his body to make it invisible in the final shot. This technique can be used to achieve an effect similar to that used in the Harry Potter films to create the effect of an invisibility cloak. The actor can also be filmed against a chroma-key background and inserted into the background shot with a distortion effect, in order to create a cloak that is marginally detectable. [13]
Difficulties emerge with blue screen when a costume in an effects shot must be blue, such as Superman's traditional blue outfit. In the 2002 film Spider-Man , in scenes where both Spider-Man and the Green Goblin are in the air, Spider-Man had to be shot in front of a green screen and the Green Goblin had to be shot in front of a blue screen. The colour difference is because Spider-Man wears a costume which is red and blue in colour and the Green Goblin wears a costume which is entirely green in colour. If both were shot in front of the same screen, parts of one character would be erased from the shot.
For a clean division of foreground from background, it is also important that clothing and hair in the foreground shot have a fairly simple silhouette, as fine details such as frizzy hair may not resolve properly. Similarly, partially transparent elements of the costume cause problems. [12]
Blue was originally used for the film industry as making the separations required a film that would only respond to the screen colour, and film that responded only to blue and higher frequencies (ultraviolet, etc.) was far easier to manufacture and make reliable than film that somehow excluded both frequencies higher and lower than the screen colour.
In television and digital film making, however, it is equally easy to extract any colour, and green quickly became the favoured colour. Bright green is less likely to be in the foreground objects, colour film emulsions usually had much finer grain in the green, and lossy compression used for analog video signals and digital images and movies retain more detail in the green channel. Green can also be used outdoors where the light colour temperature is significantly blue. Red is avoided as it is in human skin, and any other colour is a mix of primaries and thus produces a less clean extraction.
A so-called "yellow screen" is accomplished with a white backdrop. Ordinary stage lighting is used in combination with a bright yellow sodium lamp. The sodium light falls almost entirely in a narrow frequency band, which can then be separated from the other light using a prism, and projected onto a separate but synchronized film carrier within the camera. This second film is high-contrast black and white, and is processed to produce the matte. [6] : 16
A newer technique is to use a retroreflective curtain in the background, along with a ring of bright LEDs around the camera lens. This requires no light to shine on the background other than the LEDs, which use an extremely small amount of power and space unlike big stage lights, and require no rigging. This advance was made possible by the invention in the 1990s of practical blue LEDs, which also allow for emerald green LEDs.
There is also a form of colour keying that uses light spectrum invisible to human eye. Called Thermo-Key, it uses infrared as the key colour, which would not be replaced by background image during postprocessing. [14] [15]
For Star Trek: The Next Generation , an ultraviolet light matting process was proposed by Don Lee of CIS Hollywood and developed by Gary Hutzel and the staff of Image G. This involved a fluorescent orange backdrop which made it easier to generate a holdout matte, thus allowing the effects team to produce effects in a quarter of the time needed for other methods. [16]
In principle, any type of still background can be used as a chroma key instead of a solid colour. First the background is captured without actors or other foreground elements; then the scene is recorded. The image of the background is used to cancel the background in the actual footage; for example in a digital image, each pixel will have a different chroma key. This is sometimes referred to as a difference matte. [17] However, this makes it easy for objects to be accidentally removed if they happen to be similar to the background, or for the background to remain due to camera noise or if it happens to change slightly from the reference footage. [18] A background with a repeating pattern alleviates many of these issues, and can be less sensitive to wardrobe colour than solid-colour backdrops. [19]
There is some use of the specific full-intensity magenta colour #FF00FF in digital colour images to encode (1-bit) transparency; this is sometimes referred to as "magic pink". [20] This is not a photographic technique and the extraction of the foreground from the background is trivial.
The biggest challenge when setting up a blue screen or green screen is even lighting and the avoidance of shadow because it is best to have as narrow a colour range as possible being replaced. A shadow would present itself as a darker colour to the camera and might not register for replacement. This can sometimes be seen in low-budget or live broadcasts where the errors cannot be manually repaired or scenes reshot. The material being used affects the quality and ease of having it evenly lit. Materials which are shiny will be far less successful than those that are not. A shiny surface will have areas that reflect the lights making them appear pale, while other areas may be darkened. A matte surface will diffuse the reflected light and have a more even colour range. In order to get the cleanest key from shooting green screen, it is necessary to create a value difference between the subject and the green screen. In order to differentiate the subject from the screen, a two-stop difference can be used, either by making the green screen two stops higher than the subject, or vice versa.
Sometimes a shadow can be used to create a visual effect. Areas of the blue screen or green screen with a shadow on them can be replaced with a darker version of the desired background video image, making it look like the person is casting a shadow on them. Any spill of the chroma key colour will make the result look unnatural. A difference in the focal length of the lenses used can affect the success of chroma key.
Another challenge for blue screen or green screen is proper camera exposure. Underexposing or overexposing a coloured backdrop can lead to poor saturation levels. In the case of video cameras, underexposed images can contain high amounts of noise, as well. The background must be bright enough to allow the camera to create a bright and saturated image.
There are several different quality- and speed-optimised techniques for implementing colour keying in software. [21] [22]
In most versions, a function f(r, g, b) → α is applied to every pixel in the image. α (alpha) has a meaning similar to that in alpha compositing techniques. α ≤ 0 means the pixel is fully in the green screen, α ≥ 1 means the pixel is fully in the foreground object, and intermediate values indicate the pixel is partially covered by the foreground object (or it is transparent). A further function g(r, g, b) → (r, g, b) is needed to remove green spill on the foreground objects.
A very simple f() function for green screen is A(r+b) − Bg where A and B are user adjustable constants with a default value of 1.0. A very simple g() is (r, min(g,b), b). This is fairly close to the capabilities of analog and film-based screen pulling.
Modern examples [22] of these functions are best described by two closed nested surfaces in 3D RGB space, often quite complex. Colours inside the inner surface are considered green screen. Colours outside the outer surface are opaque foreground. Colours between the surfaces are partially covered, they are more opaque the closer they are to the outer surface. Sometimes more closed surfaces are used to determine how to remove green spill. It is also very common for f() to depend on more than just the current pixel's colour, it may also use the (x, y) position, the values of nearby pixels, the value from reference images or a statistical colour model of the scene, [23] and values from user-drawn masks. These produce closed surfaces in space with more than three dimensions.
A different class of algorithm tries to figure out a 2D path that separates the foreground from the background. This path can be the output, or the image can be drawn by filling the path with α = 1 as a final step. An example of such an algorithm is the use of active contour. Most research in recent years[ when? ] has been into these algorithms.
Digital compositing is the process of digitally assembling multiple images to make a final image, typically for print, motion pictures or screen display. It is the digital analogue of optical film compositing. It's part of VFX processing.
Special effects are illusions or visual tricks used in the theatre, film, television, video game, amusement park and simulator industries to simulate the imagined events in a story or virtual world. It is sometimes abbreviated as SFX, but this may also refer to sound effects.
Cinematography is the art of motion picture photography.
Visual effects is the process by which imagery is created or manipulated outside the context of a live-action shot in filmmaking and video production. The integration of live-action footage and other live-action footage or CGI elements to create realistic imagery is called VFX.
The Color Graphics Adapter (CGA), originally also called the Color/Graphics Adapter or IBM Color/Graphics Monitor Adapter, introduced in 1981, was IBM's first color graphics card for the IBM PC and established a de facto computer display standard.
Compositing is the process or technique of combining visual elements from separate sources into single images, often to create the illusion that all those elements are parts of the same scene. Live-action shooting for compositing is variously called "chroma key", "blue screen", "green screen" and other names. Today, most compositing is achieved through digital image manipulation. Pre-digital compositing techniques, however, go back as far as the trick films of Georges Méliès in the late 19th century, and some are still in use.
Color digital images are made of pixels, and pixels are made of combinations of primary colors represented by a series of code. A channel in this context is the grayscale image of the same size as a color image, made of just one of these primary colors. For instance, an image from a standard digital camera will have a red, green and blue channel. A grayscale image has just one channel.
Petro Vlahos was an American engineer and inventor, considered to be one of the pioneering scientific and technical innovators of the motion picture and television industries. He is remembered in particular for creating the Ultimatte process, which refined the colour process known as chroma keying. In recognition of his contributions, he was awarded multiple Oscars, as well as an Emmy Award.
Mattes are used in photography and special effects filmmaking to combine two or more image elements into a single, final image. Usually, mattes are used to combine a foreground image with a background image. In this case, the matte is the background painting. In film and stage, mattes can be physically huge sections of painted canvas, portraying large scenic expanses of landscapes.
This article contains a list of cinematic techniques that are divided into categories and briefly described.
A front projection effect is an in-camera visual effects process in film production for combining foreground performance with pre-filmed background footage. In contrast to rear projection, which projects footage onto a screen from behind the performers, front projection projects the pre-filmed material over the performers and onto a highly reflective background surface.
Rear projection is one of many in-camera effects cinematic techniques in film production for combining foreground performances with pre-filmed backgrounds. It was widely used for many years in driving scenes, or to show other forms of "distant" background motion.
The sodium vapor process is a photochemical film technique for combining actors and background footage. It originated in the British film industry in the late 1950s and was used extensively by Walt Disney Productions in the 1960s and 1970s as an alternative to the more common bluescreen process. Wadsworth E. Pohl is credited with the invention or development of both of these processes, and received an Academy Award in 1965 for the sodium vapor process as used in the film Mary Poppins.
Reverse bluescreen is a visual effects technique pioneered by Jonathan Erland of Apogee Inc.(John Dykstra's company) for shooting the flying sequences in the film Firefox. Its objective is to enable the matting of subjects that confound the conventional process, such as those exhibiting reflective surfaces. It derives its name from the fact that it reverses, or inverts, the basic bluescreen process. Erland received a Scientific and Engineering Award from the Academy of Motion Picture Arts and Sciences for this technique.
In cinematography, bipacking, or a bipack, is the process of loading two reels of film into a camera, so that they both pass through the camera gate together. It was used both for in-camera effects and as an early subtractive colour process.
Primatte is a brand of chroma key software used in motion picture, television and photographic host applications to remove solid colored backgrounds and replace them with transparency to facilitate ‘background replacement’. It uses a unique algorithm based on three multi-faceted polyhedrons floating in RGB colorspace that are used to isolate color regions in the foreground image. Primatte is often referred to as a compositing technology and is usually used as a plug-in for host products such as Adobe After Effects, Adobe Photoshop, Autodesk Media and Entertainment Inferno or Flame, Eyeon Fusion and several other compositing and editing software packages.
Articles related to the field of motion pictures include:
This glossary of motion picture terms is a list of definitions of terms and concepts related to motion pictures, filmmaking, cinematography, and the film industry in general.
The Williams process or Williams doublematting process is a matte creation technique patented by the American cinematographer Frank D. Williams in 1918. Unlike prior matte techniques, it allowed for the integration of the actors' movements with previously shot backgrounds.
If the foreground is a person then blue or green backing colour is recommended as these colours are not present in human flesh pigments. In fact, human skin colour is 70% red for all people regardless of race.