The first release, present in Windows Vista, focuses on audio and video playback quality, high-definition content (i.e. HDTV), content protection and a more unified approach for digital data access control for digital rights management (DRM) and its interoperability. It integrates DXVA 2.0 for offloading more of the video processing pipeline to hardware, for better performance. Videos are processed in the colorspace they were encoded in, and are handed off to the hardware, which composes the image in its native colorspace. This prevents intermediate colorspace conversions to improve performance. MF includes a new video renderer, called Enhanced Video Renderer (EVR), which is the next iteration of VMR 7 and 9. EVR has better support for playback timing and synchronization. It uses the Multimedia Class Scheduler Service (MMCSS), a new service that prioritizes real time multimedia processing, to reserve the resources required for the playback, without any tearing or glitches.
The second release included in Windows 7 introduces expanded media format support and DXVA HD for acceleration of HD content if WDDM 1.1 drivers are used.[1]
Architecture
Media Foundation Architecture
The MF architecture is divided into the Control layer, Core Layer and the Platform layer. The core layer encapsulates most of the functionality of Media Foundation. It consists of the media foundation pipeline, which has three components: Media Source, Media Sink and Media Foundation Transforms (MFT). A media source is an object that acts as the source of multimedia data, either compressed or uncompressed. It can encapsulate various data sources, like a file, or a network server or even a camcorder, with source specific functionality abstracted by a common interface. A source object can use a source resolver object which creates a media source from an URI, file or bytestream. Support for non-standard protocols can be added by creating a source resolver for them. A source object can also use a sequencer object to use a sequence of sources (a playlist) or to coalesce multiple sources into single logical source. A media sink is the recipient of processed multimedia data. A media sink can either be a renderer sink, which renders the content on an output device, or an archive sink, which saves the content onto a persistent storage system such as a file. A renderer sink takes uncompressed data as input whereas an archive sink can take either compressed or uncompressed data, depending on the output type. The data from media sources to sinks are acted upon by MFTs; MFTs are certain functions which transform the data into another form. MFTs can include multiplexers and demultiplexers, codecs or DSP effects like reverb. The core layer uses services like file access and networking and clock synchronization to time the multimedia rendering. These are part of the Platform layer, which provides services necessary for accessing the source and sink byte streams, presentation clocks and an object model that lets the core layer components function asynchronously, and is generally implemented as OS services. Pausing, stopping, fast forward, reverse or time-compression can be achieved by controlling the presentation clock.
However, the media pipeline components are not connected; rather they are just presented as discrete components. An application running in the Control layer has to choose which source types, transforms and sinks are needed for the particular video processing task at hand, and set up the "connections" between the components (a topology) to complete the data flow pipeline. For example, to play back a compressed audio/video file, the pipeline will consist of a file source object, a demultiplexer for the specific file container format to split the audio and video streams, codecs to decompress the audio and video streams, DSP processors for audio and video effects and finally the EVR renderer, in sequence. Or for a video capture application, the camcorder will act as video and audio sources, on which codec MFTs will work to compress the data and feed to a multiplexer that coalesces the streams into a container; and finally a file sink or a network sink will write it to a file or stream over a network. The application also has to co-ordinate the flow of data between the pipeline components. The control layer has to "pull" (request) samples from one pipeline component and pass it onto the next component in order to achieve data flow within the pipeline. This is in contrast to DirectShow's "push" model where a pipeline component pushes data to the next component. Media Foundation allows content protection by hosting the pipeline within a protected execution environment, called the Protected Media Path. The control layer components are required to propagate the data through the pipeline at a rate that the rendering synchronizes with the presentation clock. The rate (or time) of rendering is embedded as a part of the multimedia stream as metadata. The source objects extract the metadata and pass it over. Metadata is of two types: coded metadata, which is information about bit rate and presentation timings, and descriptive metadata, like title and author names. Coded metadata is handed over to the object that controls the pipeline session, and descriptive metadata is exposed for the application to use if it chooses to.
Media Foundation provides a Media Session object that can be used to set up the topologies, and facilitate a data flow, without the application doing it explicitly. It exists in the control layer, and exposes a Topology loader object. The application specifies the required pipeline topology to the loader, which then creates the necessary connections between the components. The media session object manages the job of synchronizing with the presentation clock. It creates the presentation clock object, and passes a reference to it to the sink. It then uses the timer events from the clock to propagate data along the pipeline. It also changes the state of the clock to handle pause, stop or resume requests from the application.
Practical MF Architectures
Theoretically there is only one Media Foundation architecture and this is the Media Session, Pipeline, Media Source, Transform and Media Sink model. However this architecture can be complex to set up and there is considerable scope for lightweight, relatively easy to configure MF components designed to handle the processing of media data for simple point solutions. Thus practical considerations necessitated the implementation of variations on the fundamental Pipeline design and components such as the Source Reader and Sink Writer which operate outside the Pipeline model were developed. Some sources [2] split the Media Foundation architecture into three general classes.
The Pipeline Architecture
The Reader-Writer Architecture
Hybrids between the Pipeline and Reader-Writer Architectures
The Pipeline Architecture is distinguished by the use of a distinct Media Session object and Pipeline. The media data flows from one or more Media Sources to one or more Media Sinks and, optionally, through zero or more Media Transforms. It is the Media Session that manages the flow of the media data through the Pipeline and that Pipeline can have multiple forks and branches. An MF application can get access to the media data as it traverses from a Media Source to a Media Sink by implementing a custom Media Transform component and inserting it in an appropriate location in the Pipeline.
The Reader-Writer Architecture uses a component called a Source Reader to provide the media data and a Sink Writer component to consume it. The Source Reader does contain a type of internal pipeline but this is not accessible to the application. A Source Reader is not a Media Source and a Sink Writer is not a Media Sink and neither can be directly included in a Pipeline or managed by a Media Session. In general, the media data flows from the Source Reader to the Sink Writer by the actions of the application. The application will either take the packets of media data (called Media Samples) from the Source Reader and give them directly them to the Sink Writer or it will set up a callback function on the Source Reader which performs the same operation. In effect, as it manages the data transport, the application itself performs a similar role to that of the Media Session in a Pipeline Architecture application. Since the MF application manages the transmission of the Media Samples between the Source Reader and Sink Writer it will always have access to the raw media data. The Source Reader and Sink Writer components do have a limited ability to automatically load Media Transforms to assist with the conversion of the format of the media data, however, this is done internally and the application has little control over it.
The Source Reader and Sink Writer provide ease of use and the Pipeline Architecture offers extremely sophisticated control over the flow of the media data. However, many of the components available to a Pipeline (such as the Enhanced Video Renderer) are simply not readily usable in a Reader-Writer architecture application. Since the structure of a Media Sample produced by a Source Reader is identical to that output by a Media Source it is possible to set up a Pipeline Architecture in which the Media Samples are intercepted as they pass through the Pipeline and a copy is given to a Media Sink. This is known as a Hybrid Architecture and it makes it possible to have an application which takes advantage of the sophisticated processing abilities of the Media Session and Pipeline while utilizing the ease of use of a Sink Writer. The Sink Writer is not part of the Pipeline and it does not interact with the Media Session. In effect, the media data is processed by a special Media Sink called a Sample Grabber Sink which consumes the media data and hands a copy off to the Sink Writer as it does so. It is also possible to implement a Hybrid Architecture with a custom Media Transform which copies the Media Samples and passes them to a Sink Writer as they pass through the Pipeline. In both cases a special component in the Pipeline effectively acts like a simple Reader-Writer application and feeds a Sink Writer. In general, Hybrid Architectures use a Pipeline and a Sink Writer. Theoretically, it is possible to implement a mechanism in which a Source Reader could somehow inject Media Samples into a Pipeline but, unlike the Sample Grabber Sink, no such standard component exists.
Media Foundation Transform
Media Foundation Transforms (MFTs) represent a generic model for processing media data. They are used in Media Foundation primarily to implement decoders, encoders, mixers and digital signal processors (DSPs) – between media sources and media sinks. Media Foundation Transforms are an evolution of the transform model first introduced with DirectX Media Objects (DMOs). Their behaviors are more clearly specified. Hybrid DMO/MFT Objects can also be created. Applications can use MFTs inside the Media Foundation pipeline, or use them directly as stand-alone objects. MFTs can be any of the following type:
Audio and video codecs
Audio and video effects
Multiplexers and demultiplexers
Tees
Color-space converters
Sample-rate converters
Video scalers
Microsoft recommends developers to write a Media Foundation Transform instead of a DirectShow filter, for Windows Vista, Windows 7 & Windows 8.[3] For video editing and video capture, Microsoft recommends using DirectShow as they are not the primary focus of Media Foundation in Windows Vista. Starting with Windows 7, MFTs also support hardware-accelerated video processing, encoding and decoding for AVStream-based media devices.[4]
Enhanced Video Renderer
Media Foundation uses the Enhanced Video Renderer (EVR) for rendering video content, which acts as a mixer as well. It can mix up to 16 simultaneous streams, with the first stream being a reference stream. All but the reference stream can have per-pixel transparency information, as well as any specified z-order. The reference stream cannot have transparent pixels, and has a fixed z-order position, at the back of all streams. The final image is composited onto a single surface by coloring each pixel according to the color and transparency of the corresponding pixel in all streams.
Internally, the EVR uses a mixer object for mixing the streams. It can also deinterlace the output and apply color correction, if required. The composited frame is handed off to a presenter object, which schedules them for rendering onto a Direct3D device, which it shares with the DWM and other applications using the device. The frame rate of the output video is synchronized with the frame rate of the reference stream. If any of the other streams (called substreams) have a different frame rate, EVR discards the extra frames (if the substream has a higher frame rate), or uses the same frame more than once (if it has a lower frame rate).
Supported media formats
Windows Media Audio and Windows Media Video are the only default supported formats for encoding through Media Foundation in Windows Vista. For decoding, an MP3 file source is available in Windows Vista to read MP3 streams but an MP3 file sink to output MP3 is only available in Windows 7.[5] Format support is extensible however; developers can add support for other formats by writing encoder/decoder MFTs and/or custom media sources/media sinks.
Windows 7 expands upon the codec support available in Windows Vista. It includes AVI, WAV, AAC/ADTS file sources to read the respective formats,[5] an MPEG-4 file source to read MP4, M4A, M4V, MP4V, MOV and 3GPcontainer formats[6] and an MPEG-4 file sink to output to MP4 format.[7]
Similar to Windows Vista, transcoding (encoding) support is not exposed through any built-in Windows application but several codecs are included as Media Foundation Transforms (MFTs).[5] In addition to Windows Media Audio and Windows Media Video encoders and decoders, and ASF file sink and file source introduced in Windows Vista,[5] Windows 7 includes an H.264 encoder with Baseline profile level 3 and Main profile support [8] and an AAC Low Complexity (AAC-LC) profile encoder [9]
Firefox v24 and later on Windows 7 and later (only for H.264 playback)
GoldWave 5.60 and later relies on Media Foundation for importing and exporting audio. For export, AAC and Apple Lossless formats can be saved via Media Foundation
Any application that uses Protected Media Path in Windows also uses Media Foundation.
Related Research Articles
A codec is a computer hardware or software component that encodes or decodes a data stream or signal. Codec is a portmanteau of coder/decoder.
In information technology, lossy compression or irreversible compression is the class of data compression methods that uses inexact approximations and partial data discarding to represent the content. These techniques are used to reduce data size for storing, handling, and transmitting content. Higher degrees of approximation create coarser images as more details are removed. This is opposed to lossless data compression which does not degrade the data. The amount of data reduction possible using lossy compression is much higher than using lossless techniques.
Ogg is a free, open container format maintained by the Xiph.Org Foundation. The authors of the Ogg format state that it is unrestricted by software patents and is designed to provide for efficient streaming and manipulation of high-quality digital multimedia. Its name is derived from "ogging", jargon from the computer game Netrek.
Windows Media Audio (WMA) is a series of audio codecs and their corresponding audio coding formats developed by Microsoft. It is a proprietary technology that forms part of the Windows Media framework. WMA consists of four distinct codecs. The original WMA codec, known simply as WMA, was conceived as a competitor to the popular MP3 and RealAudio codecs. WMA Pro, a newer and more advanced codec, supports multichannel and high-resolution audio. A lossless codec, WMA Lossless, compresses audio data without loss of audio fidelity. WMA Voice, targeted at voice content, applies compression using a range of low bit rates. Microsoft has also developed a digital container format called Advanced Systems Format to store audio encoded by WMA.
Advanced Audio Coding (AAC) is an audio coding standard for lossy digital audio compression. It was designed to be the successor of the MP3 format and generally achieves higher sound quality than MP3 at the same bit rate.
Musepack or MPC is an open source lossy audio codec, specifically optimized for transparent compression of stereo audio at bitrates of 160–180 kbit/s. It was formerly known as MPEGplus, MPEG+ or MP+.
DirectShow, codename Quartz, is a multimedia framework and API produced by Microsoft for software developers to perform various operations with media files or streams. It is the replacement for Microsoft's earlier Video for Windows technology. Based on the Microsoft Windows Component Object Model (COM) framework, DirectShow provides a common interface for media across various programming languages, and is an extensible, filter-based framework that can render or record media files on demand at the request of the user or developer. The DirectShow development tools and documentation were originally distributed as part of the DirectX SDK. Currently, they are distributed as part of the Windows SDK.
High-Efficiency Advanced Audio Coding (HE-AAC) is an audio coding format for lossy data compression of digital audio defined as an MPEG-4 Audio profile in ISO/IEC 14496–3. It is an extension of Low Complexity AAC (AAC-LC) optimized for low-bitrate applications such as streaming audio. The usage profile HE-AAC v1 uses spectral band replication (SBR) to enhance the modified discrete cosine transform (MDCT) compression efficiency in the frequency domain. The usage profile HE-AAC v2 couples SBR with Parametric Stereo (PS) to further enhance the compression efficiency of stereo signals.
A multimedia framework is a software framework that handles media on a computer and through a network. A good multimedia framework offers an intuitive API and a modular architecture to easily add support for new audio, video and container formats and transmission protocols. It is meant to be used by applications such as media players and audio or video editors, but can also be used to build videoconferencing applications, media converters and other multimedia tools. Data is processed among modules automatically, it is unnecessary for app to pass buffers between connected modules one by one.
A container format or metafile is a file format that allows multiple data streams to be embedded into a single file, usually along with metadata for identifying and further detailing those streams. Notable examples of container formats include archive files and formats used for multimedia playback. Among the earliest cross-platform container formats were Distinguished Encoding Rules and the 1985 Interchange File Format.
FAAC is a software project which includes the AAC encoder FAAC and decoder FAAD2. It supports MPEG-2 AAC as well as MPEG-4 AAC. It supports several MPEG-4 Audio object types, file formats, multichannel and gapless encoding/decoding and MP4 metadata tags. The encoder and decoder is compatible with standard-compliant audio applications using one or more of these object types and facilities. It also supports Digital Radio Mondiale.
Indeo Video is a family of audio and video formats and codecs first released in 1992, and designed for real-time video playback on desktop CPUs. While its original version was related to Intel's DVI video stream format, a hardware-only codec for the compression of television-quality video onto compact discs, Indeo was distinguished by being one of the first codecs allowing full-speed video playback without using hardware acceleration. Also unlike Cinepak and TrueMotion S, the compression used the same Y'CbCr 4:2:0 colorspace as the ITU's H.261 and ISO's MPEG-1. Indeo use was free of charge to allow for broadest usage.
DirectX Video Acceleration (DXVA) is a Microsoft API specification for the Microsoft Windows and Xbox 360 platforms that allows video decoding to be hardware-accelerated. The pipeline allows certain CPU-intensive operations such as iDCT, motion compensation and deinterlacing to be offloaded to the GPU. DXVA 2.0 allows more operations, including video capturing and processing operations, to be hardware-accelerated as well.
Flash Video is a container file format used to deliver digital video content over the Internet using Adobe Flash Player version 6 and newer. Flash Video content may also be embedded within SWF files. There are two different Flash Video file formats: FLV and F4V. The audio and video data within FLV files are encoded in the same way as SWF files. The F4V file format is based on the ISO base media file format, starting with Flash Player 9 update 3. Both formats are supported in Adobe Flash Player and developed by Adobe Systems. FLV was originally developed by Macromedia. In the early 2000s, Flash Video was the de facto standard for web-based streaming video. Users include Hulu, VEVO, Yahoo! Video, metacafe, Reuters.com, and many other news providers.
Windows Imaging Component (WIC) is a COM-based imaging codec framework introduced in Windows Vista (and later available in Windows XP Service Pack 3) for working with and processing digital images and image metadata.
The following is a list of H.264/MPEG-4 AVC products and implementations.
Microsoft Expression Encoder is a discontinued transcoding and non-linear video editing software application for Microsoft Windows. It can create video streams for distribution via Microsoft Silverlight. This utility is created to record the screen for various purposes such asYouTube, Twitch etc.
Opus is a lossy audio coding format developed by the Xiph.Org Foundation and standardized by the Internet Engineering Task Force, designed to efficiently code speech and general audio in a single format, while remaining low-latency enough for real-time interactive communication and low-complexity enough for low-end embedded processors. Opus replaces both Vorbis and Speex for new applications, and several blind listening tests have ranked it higher-quality than any other standard audio format at any given bitrate until transparency is reached, including MP3, AAC, and HE-AAC.
HTML audio is a subject of the HTML specification, incorporating audio input, playback, and synthesis, as well as speech to text, all in the browser.
References
↑ "DXVA-HD". Archived from the original on 2012-04-20. Retrieved 2010-04-18.
This page is based on this Wikipedia article Text is available under the CC BY-SA 4.0 license; additional terms may apply. Images, videos and audio are available under their respective licenses.
