Multimedia framework

Last updated

A multimedia framework is a software framework that handles media on a computer and through a network. A good multimedia framework offers an intuitive API and a modular architecture to easily add support for new audio, video and container formats and transmission protocols. It is meant to be used by applications such as media players and audio or video editors, but can also be used to build videoconferencing applications, media converters and other multimedia tools. Data is processed among modules automatically, it is unnecessary for app to pass buffers between connected modules one by one.

Contents

In contrast to function libraries, a multimedia framework provides a run time environment for the media processing. Ideally such an environment provides execution contexts for the media processing blocks separated from the application using the framework. The separation supports the independent processing of multimedia data in a timely manner. These separate contexts can be implemented as threads.

Overview

Multimedia frameworks act as an abstraction layer, simplifying the complexity of handling multimedia data formats, decoding and encoding streams, synchronizing audio and video playback, and managing multimedia resources. They provide a consistent interface for developers to access various multimedia components and services offered by the underlying operating system or hardware.

Functionality

Modern multimedia frameworks typically offer a wide range of functionality, including but not limited to:

  1. Codec Support: Multimedia frameworks often incorporate codecs for various audio and video formats, allowing developers to easily encode, decode, and manipulate multimedia data.
  2. Synchronization: These frameworks provide mechanisms to synchronize audio and video streams, maintaining proper lip-sync and smooth playback.
  3. Playback Control: Multimedia frameworks offer playback control features, such as play, pause, stop, seek, and volume adjustment, allowing developers to create interactive multimedia applications.
  4. Streaming: Many frameworks support the streaming of multimedia content over networks, enabling real-time delivery and playback.
  5. Capturing and Recording: Multimedia frameworks often include APIs for capturing audio and video from devices like cameras and microphones, as well as recording multimedia streams. [1]

Several prominent multimedia frameworks have gained widespread adoption, and they vary depending on the programming language and platform:

See also

Related Research Articles

Theora is a free lossy video compression format. It was developed by the Xiph.Org Foundation and distributed without licensing fees alongside their other free and open media projects, including the Vorbis audio format and the Ogg container.

Monkey's Audio is an algorithm and file format for lossless audio data compression. Lossless data compression does not discard data during the process of encoding, unlike lossy compression methods such as Advanced Audio Coding, MP3, Vorbis, and Opus. Therefore, it may be decompressed to a file that is identical to the source material.

<span class="mw-page-title-main">FFmpeg</span> Multimedia framework

FFmpeg is a free and open-source software project consisting of a suite of libraries and programs for handling video, audio, and other multimedia files and streams. At its core is the command-line ffmpeg tool itself, designed for processing video and audio files. It is widely used for format transcoding, basic editing, video scaling, video post-production effects, and standards compliance.

<span class="mw-page-title-main">VLC media player</span> Free and open-source media-player and streaming-media-server

VLC media player is a free and open-source, portable, cross-platform media player software and streaming media server developed by the VideoLAN project. VLC is available for desktop operating systems and mobile platforms, such as Android, iOS and iPadOS. VLC is also available on digital distribution platforms such as Apple's App Store, Google Play, and Microsoft Store.

<span class="mw-page-title-main">GStreamer</span> Multimedia framework

GStreamer is a pipeline-based multimedia framework that links together a wide variety of media processing systems to complete complex workflows. For instance, GStreamer can be used to build a system that reads files in one format, processes them, and exports them in another. The formats and processes can be changed in a plug and play fashion.

<span class="mw-page-title-main">DirectShow</span> Application programming interface for multimedia processing

DirectShow, codename Quartz, is a multimedia framework and API produced by Microsoft for software developers to perform various operations with media files or streams. It is the replacement for Microsoft's earlier Video for Windows technology. Based on the Microsoft Windows Component Object Model (COM) framework, DirectShow provides a common interface for media across various programming languages, and is an extensible, filter-based framework that can render or record media files on demand at the request of the user or developer. The DirectShow development tools and documentation were originally distributed as part of the DirectX SDK. Currently, they are distributed as part of the Windows SDK.

OpenMAX, often shortened as "OMX", is a non-proprietary and royalty-free cross-platform set of C-language programming interfaces. It provides abstractions for routines that are especially useful for processing of audio, video, and still images. It is intended for low power and embedded system devices that need to efficiently process large amounts of multimedia data in predictable ways, such as video codecs, graphics libraries, and other functions for video, image, audio, voice and speech.

The following comparison of video players compares general and technical information for notable software media player programs.

SMPTE 421, informally known as VC-1, is a video coding format. Most of it was initially developed as Microsoft's proprietary video format Windows Media Video 9 in 2003. With some enhancements including the development of a new Advanced Profile, it was officially approved as a SMPTE standard on April 3, 2006. It was primarily marketed as a lower-complexity competitor to the H.264/MPEG-4 AVC standard. After its development, several companies other than Microsoft asserted that they held patents that applied to the technology, including Panasonic, LG Electronics and Samsung Electronics.

FAAC is a software project which includes the AAC encoder FAAC and decoder FAAD2. It supports MPEG-2 AAC as well as MPEG-4 AAC. It supports several MPEG-4 Audio object types, file formats, multichannel and gapless encoding/decoding and MP4 metadata tags. The encoder and decoder is compatible with standard-compliant audio applications using one or more of these object types and facilities. It also supports Digital Radio Mondiale.

<span class="mw-page-title-main">Indeo</span> Audio and video formats by Intel

Indeo Video is a family of audio and video formats and codecs first released in 1992, and designed for real-time video playback on desktop CPUs. While its original version was related to Intel's DVI video stream format, a hardware-only codec for the compression of television-quality video onto compact discs, Indeo was distinguished by being one of the first codecs allowing full-speed video playback without using hardware acceleration. Also unlike Cinepak and TrueMotion S, the compression used the same Y'CbCr 4:2:0 colorspace as the ITU's H.261 and ISO's MPEG-1. Indeo use was free of charge to allow for broadest usage.

libavcodec is a free and open-source library of codecs for encoding and decoding video and audio data.

The Java Media Framework (JMF) is a Java library that enables audio, video and other time-based media to be added to Java applications and applets. This optional package, which can capture, play, stream, and transcode multiple media formats, extends the Java Platform, Standard Edition and allows development of cross-platform multimedia applications.

Media Foundation (MF) is a COM-based multimedia framework pipeline and infrastructure platform for digital media in Windows Vista, Windows 7, Windows 8, Windows 8.1, Windows 10, and Windows 11. It is the intended replacement for Microsoft DirectShow, Windows Media SDK, DirectX Media Objects (DMOs) and all other so-called "legacy" multimedia APIs such as Audio Compression Manager (ACM) and Video for Windows (VfW). The existing DirectShow technology is intended to be replaced by Media Foundation step-by-step, starting with a few features. For some time there will be a co-existence of Media Foundation and DirectShow. Media Foundation will not be available for previous Windows versions, including Windows XP.

<span class="mw-page-title-main">Moonlight (runtime)</span> Implementation of Microsoft Silverlight for some Unix-based operating systems

Moonlight is a discontinued free and open source implementation for Linux and other Unix-based operating systems of the Microsoft Silverlight application framework, developed and then abandoned by the Mono Project. Like Silverlight, Moonlight was a web application framework which provided capabilities similar to those of Adobe Flash, integrating multimedia, graphics, animations and interactivity into a single runtime environment.

Video Acceleration API (VA-API) is an open source application programming interface that allows applications such as VLC media player or GStreamer to use hardware video acceleration capabilities, usually provided by the graphics processing unit (GPU). It is implemented by the free and open-source library libva, combined with a hardware-specific driver, usually provided together with the GPU driver.

<span class="mw-page-title-main">Libav</span> Multimedia framework

Libav is an abandoned free software project, forked from FFmpeg in 2011, that contains libraries and programs for handling multimedia data.

<span class="mw-page-title-main">Opus (audio format)</span> Lossy audio coding format

Opus is a lossy audio coding format developed by the Xiph.Org Foundation and standardized by the Internet Engineering Task Force, designed to efficiently code speech and general audio in a single format, while remaining low-latency enough for real-time interactive communication and low-complexity enough for low-end embedded processors. Opus replaces both Vorbis and Speex for new applications, and several blind listening tests have ranked it higher-quality than any other standard audio format at any given bitrate until transparency is reached, including MP3, AAC, and HE-AAC.

<span class="mw-page-title-main">Distributed Codec Engine</span>

Distributed Codec Engine (DCE) is an API and its implementation as software library ("libdce") by Texas Instruments. The library was released under the Revised BSD License and some additional terms.

HTML audio is a subject of the HTML specification, incorporating audio input, playback, and synthesis, as well as speech to text, all in the browser.

References

  1. Le Feuvre, Jean; Concolato, Cyril; Moissinac, Jean-Claude (2007-09-29). "GPAC: Open source multimedia framework". Proceedings of the 15th ACM international conference on Multimedia. MM '07. New York, NY, USA: Association for Computing Machinery. pp. 1009–1012. doi:10.1145/1291233.1291452. ISBN   978-1-59593-702-5.
  2. [GStreamer: open source multimedia framework "GStreamer: open source multimedia framework"].{{cite web}}: Check |url= value (help)