Multimedia framework

Last updated

A multimedia framework is a software framework that handles media on a computer and through a network. A good multimedia framework offers an intuitive API and a modular architecture to easily add support for new audio, video and container formats and transmission protocols. It is meant to be used by applications such as media players and audio or video editors, but can also be used to build videoconferencing applications, media converters and other multimedia tools. Data is processed among modules automatically, it is unnecessary for app to pass buffers between connected modules one by one.

Contents

In contrast to function libraries, a multimedia framework provides a run time environment for the media processing. Ideally such an environment provides execution contexts for the media processing blocks separated from the application using the framework. The separation supports the independent processing of multimedia data in a timely manner. These separate contexts can be implemented as threads.

Overview

Multimedia frameworks act as an abstraction layer, simplifying the complexity of handling multimedia data formats, decoding and encoding streams, synchronizing audio and video playback, and managing multimedia resources. They provide a consistent interface for developers to access various multimedia components and services offered by the underlying operating system or hardware.

Functionality

Modern multimedia frameworks typically offer a wide range of functionality, including but not limited to:

  1. Codec Support: Multimedia frameworks often incorporate codecs for various audio and video formats, allowing developers to easily encode, decode, and manipulate multimedia data.
  2. Synchronization: These frameworks provide mechanisms to synchronize audio and video streams, maintaining proper lip-sync and smooth playback.
  3. Playback Control: Multimedia frameworks offer playback control features, such as play, pause, stop, seek, and volume adjustment, allowing developers to create interactive multimedia applications.
  4. Streaming: Many frameworks support the streaming of multimedia content over networks, enabling real-time delivery and playback.
  5. Capturing and Recording: Multimedia frameworks often include APIs for capturing audio and video from devices like cameras and microphones, as well as recording multimedia streams. [1]

Several prominent multimedia frameworks have gained widespread adoption, and they vary depending on the programming language and platform:

See also

Related Research Articles

Theora is a free lossy video compression format. It was developed by the Xiph.Org Foundation and distributed without licensing fees alongside their other free and open media projects, including the Vorbis audio format and the Ogg container.

Monkey's Audio is an algorithm and file format for lossless audio data compression. Lossless data compression does not discard data during the process of encoding, unlike lossy compression methods such as Advanced Audio Coding, MP3, Vorbis, and Opus. Therefore, it may be decompressed to a file that is identical to the source material.

<span class="mw-page-title-main">FFmpeg</span> Multimedia framework

FFmpeg is a free and open-source software project consisting of a suite of libraries and programs for handling video, audio, and other multimedia files and streams. At its core is the command-line ffmpeg tool itself, designed for processing of video and audio files. It is widely used for format transcoding, basic editing, video scaling, video post-production effects and standards compliance.

<span class="mw-page-title-main">GStreamer</span> Multimedia framework

GStreamer is a pipeline-based multimedia framework that links together a wide variety of media processing systems to complete complex workflows. For instance, GStreamer can be used to build a system that reads files in one format, processes them, and exports them in another. The formats and processes can be changed in a plug and play fashion.

<span class="mw-page-title-main">DirectShow</span> Microsoft API

DirectShow, codename Quartz, is a multimedia framework and API produced by Microsoft for software developers to perform various operations with media files or streams. It is the replacement for Microsoft's earlier Video for Windows technology. Based on the Microsoft Windows Component Object Model (COM) framework, DirectShow provides a common interface for media across various programming languages, and is an extensible, filter-based framework that can render or record media files on demand at the request of the user or developer. The DirectShow development tools and documentation were originally distributed as part of the DirectX SDK. Currently, they are distributed as part of the Windows SDK.

OpenMAX, often shortened as "OMX", is a non-proprietary and royalty-free cross-platform set of C-language programming interfaces. It provides abstractions for routines that are especially useful for processing of audio, video, and still images. It is intended for low power and embedded system devices that need to efficiently process large amounts of multimedia data in predictable ways, such as video codecs, graphics libraries, and other functions for video, image, audio, voice and speech.

The following comparison of video players compares general and technical information for notable software media player programs.

SMPTE 421, informally known as VC-1, is a video coding format. Most of it was initially developed as Microsoft's proprietary video format Windows Media Video 9 in 2003. With some enhancements including the development of a new Advanced Profile, it was officially approved as a SMPTE standard on April 3, 2006. It was primarily marketed as a lower-complexity competitor to the H.264/MPEG-4 AVC standard. After its development, several companies other than Microsoft asserted that they held patents that applied to the technology, including Panasonic, LG Electronics and Samsung Electronics.

FAAC or Freeware Advanced Audio Coder is a software project which includes the AAC encoder FAAC and decoder FAAD2. It supports MPEG-2 AAC as well as MPEG-4 AAC. It supports several MPEG-4 Audio object types, file formats, multichannel and gapless encoding/decoding and MP4 metadata tags. The encoder and decoder is compatible with standard-compliant audio applications using one or more of these object types and facilities. It also supports Digital Radio Mondiale.

<span class="mw-page-title-main">Indeo</span> Audio and video formats by Intel

Indeo Video is a family of audio and video formats and codecs first released in 1992, and designed for real-time video playback on desktop CPUs. While its original version was related to Intel's DVI video stream format, a hardware-only codec for the compression of television-quality video onto compact discs, Indeo was distinguished by being one of the first codecs allowing full-speed video playback without using hardware acceleration. Also unlike Cinepak and TrueMotion S, the compression used the same Y'CbCr 4:2:0 colorspace as the ITU's H.261 and ISO's MPEG-1. Indeo use was free of charge to allow for broadest usage.

libavcodec is a free and open-source library of codecs for encoding and decoding video and audio data.

The Java Media Framework (JMF) is a Java library that enables audio, video and other time-based media to be added to Java applications and applets. This optional package, which can capture, play, stream, and transcode multiple media formats, extends the Java Platform, Standard Edition and allows development of cross-platform multimedia applications.

Media Foundation (MF) is a COM-based multimedia framework pipeline and infrastructure platform for digital media in Windows Vista, Windows 7, Windows 8, Windows 8.1, Windows 10, and Windows 11. It is the intended replacement for Microsoft DirectShow, Windows Media SDK, DirectX Media Objects (DMOs) and all other so-called "legacy" multimedia APIs such as Audio Compression Manager (ACM) and Video for Windows (VfW). The existing DirectShow technology is intended to be replaced by Media Foundation step-by-step, starting with a few features. For some time there will be a co-existence of Media Foundation and DirectShow. Media Foundation will not be available for previous Windows versions, including Windows XP.

Video Acceleration API (VA-API) is an open source application programming interface that allows applications such as VLC media player or GStreamer to use hardware video acceleration capabilities, usually provided by the graphics processing unit (GPU). It is implemented by the free and open-source library libva, combined with a hardware-specific driver, usually provided together with the GPU driver.

The HTML5 specification introduced the video element for the purpose of playing videos, partially replacing the object element. HTML5 video is intended by its creators to become the new standard way to show video on the web, instead of the previous de facto standard of using the proprietary Adobe Flash plugin, though early adoption was hampered by lack of agreement as to which video coding formats and audio coding formats should be supported in web browsers. As of 2020, HTML5 video is the only widely supported video playback technology in modern browsers, with the Flash plugin being phased out.

<span class="mw-page-title-main">Opus (audio format)</span> Lossy audio coding format

Opus is a lossy audio coding format developed by the Xiph.Org Foundation and standardized by the Internet Engineering Task Force, designed to efficiently code speech and general audio in a single format, while remaining low-latency enough for real-time interactive communication and low-complexity enough for low-end embedded processors. Opus replaces both Vorbis and Speex for new applications, and several blind listening tests have ranked it higher-quality than any other standard audio format at any given bitrate until transparency is reached, including MP3, AAC, and HE-AAC.

<span class="mw-page-title-main">Distributed Codec Engine</span>

Distributed Codec Engine (DCE) is an API and its implementation as software library ("libdce") by Texas Instruments. The library was released under the Revised BSD License and some additional terms.

HTML5 Audio is a subject of the HTML5 specification, incorporating audio input, playback, and synthesis, as well as in the browser. iOS

mpv (media player) Free and open-source media player software

mpv is free and open-source media player software based on MPlayer, mplayer2 and FFmpeg. It runs on several operating systems, including Unix-like operating systems and Microsoft Windows, along with having an Android port called mpv-android. It is cross-platform, running on ARM, PowerPC, x86/IA-32, x86-64, and MIPS architecture.

VisualOn is a Silicon Valley–based multimedia software company that provides high-definition audio and video entertainment to smartphones, tablets, laptops, connected TVs and other mobile and convergent devices. VisualOn's patented technology is modular and platform-agnostic. VisualOn supports streaming, VOD, mobile TV and other multimedia applications.

References

  1. Le Feuvre, Jean; Concolato, Cyril; Moissinac, Jean-Claude (2007-09-29). "GPAC: open source multimedia framework". Proceedings of the 15th ACM international conference on Multimedia. MM '07. New York, NY, USA: Association for Computing Machinery: 1009–1012. doi:10.1145/1291233.1291452. ISBN   978-1-59593-702-5.
  2. [GStreamer: open source multimedia framework "GStreamer: open source multimedia framework"].{{cite web}}: Check |url= value (help)