Multiview Video Coding

Last updated

Multi view Video Coding (MVC, also known as MVC 3D) is a stereoscopic video coding standard for video compression that allows for encoding of video sequences captured simultaneously from multiple camera angles in a single video stream. [1] It uses the 2D plus Delta method and is an amendment to the H.264 (MPEG-4 AVC) video compression standard, developed jointly by MPEG and VCEG, [2] with contributions from a number of companies, primarily Panasonic and LG Electronics. [3]

Contents

MVC formatting is intended for encoding stereoscopic (two-view) 3D video, as well as free viewpoint television and multi-view 3D television. The Stereo High profile has been standardized in June 2009; the profile is based on the MVC tool set and is used in stereoscopic Blu-ray 3D releases. [4]

Technical overview

MVC is based on the idea that video recordings of the same scene from multiple angles share many common elements. It is possible to encode all simultaneous frames captured in the same elementary stream and to share as much information as possible across the different layers. This can reduce the size of the encoded video. [5] [6]

Multiview video contains a large amount of inter-view statistical dependencies, since all cameras capture the same scene from different viewpoints. Therefore, combined temporal and inter-view prediction is important for efficient MVC encoding. A frame from a certain camera can be predicted not only from temporally related frames from the same camera, but also from the frames of neighboring cameras. These interdependencies can be used for efficient prediction. [7] [8] [9]

The method for this is used in Multiview Video Coding for Blu-ray 3D movies is known as the 2D plus Delta algorithm, and the MVC specification itself is part of the H.264 standard as an amendment in H.264 “Annex H” of the specification. [10]

Open source support mostly missing

As of April 2015, there is no free and open-source software that supports software decoding of the MVC video compression standard. [11] Popular open source H.264 and HEVC (H.265) decoders, such as those used in the FFmpeg and Libav libraries, simply ignore the second view and thus do not show the second view for stereoscopic views. In most cases, the reason for this support not being added is that MVC was not considered when the initial core H.264 and HEVC decoders code was written. Later amendment would as such often mean a lot of prerequisite code refactoring work and large changes its current architecture, with major work in untangling and reordering some code, and splitting different functions in existing decoder code into smaller chunks for simpler handling to in turn then make amendments such as MVC easier to add. [12]

Some proof-of-concept work has however been done downstream in the past, but never made it upstream into official releases of FFmpeg or Libav. [13] [14]

On March 8, 2016, the situation improved. Version 0.68 of the DirectShow Media Splitter and Decoders Collection LAV Filters [15] was released by developer "Nevcairiel" (who also works for Media Player Classic — Home Cinema (MPC-HC)) with support of H.264 MVC 3D demuxing and decoding. With the aid of this release and FRIM [16] written by a programmer named “videohelp3d” [17] it is possible to write an AviSynth script to pre process a H.264 MVC 3D video clip which can then be opened by free 3D video player Bino [18] and then shown as red — cyan anaglyph video for example. [11]

The usage of the FRIM AviSynth plugin (FRIMSource) is described on “videohelp3d” home page. LAV Filters can be used to get audio from H.264 MVC 3D video clip. The developer [19] posted that in a future release of, it might be possible that LAV Video renders the video as Side-by-Side directly. [20]

Patent holders

The following organizations hold patents that contributed to the development of MVC technology, listed in a patent pool by MPEG LA. [21]

Organization [22] Active patentsExpired patentsTotal patents [3]
Panasonic 60741648
LG Electronics 85287
Dolby Laboratories 77380
Fraunhofer 65065
Fujitsu 59362
Mitsubishi Electric 282149
GE 31031
Tagivan II LLC27027
Siemens 14923
Columbia University 01717
Maxell 14014
Thomson Licensing01414
Koninklijke KPN N.V.1 01313
Nippon Telegraph and Telephone 808
NTT Docomo 606
Sony 505
Electronics and Telecommunications Research Institute 213
HP Inc. 011

See also

Related Research Articles

<span class="mw-page-title-main">FFmpeg</span> Multimedia framework

FFmpeg is a free and open-source software project consisting of a suite of libraries and programs for handling video, audio, and other multimedia files and streams. At its core is the command-line ffmpeg tool itself, designed for processing of video and audio files. It is widely used for format transcoding, basic editing, video scaling, video post-production effects and standards compliance.

Advanced Audio Coding (AAC) is an audio coding standard for lossy digital audio compression. It was designed to be the successor of the MP3 format and generally achieves higher sound quality than MP3 at the same bit rate.

<span class="mw-page-title-main">Advanced Video Coding</span> Most widely used standard for video compression

Advanced Video Coding (AVC), also referred to as H.264 or MPEG-4 Part 10, is a video compression standard based on block-oriented, motion-compensated coding. It is by far the most commonly used format for the recording, compression, and distribution of video content, used by 91% of video industry developers as of September 2019. It supports a maximum resolution of 8K UHD.

ffdshow Open-source unmaintained codec library

ffdshow is an open-source unmaintained codec library that is mainly used for decoding of video in the MPEG-4 ASP and H.264/MPEG-4 AVC video formats, but it supports numerous other video and audio formats as well. It is free software released under GNU General Public License 2.0, runs on Windows, and is implemented as a Video for Windows (VFW) codec and a DirectShow filter.

SMPTE 421, informally known as VC-1, is a video coding format. Most of it was initially developed as Microsoft's proprietary video format Windows Media Video 9 in 2003. With some enhancements including the development of a new Advanced Profile, it was officially approved as a SMPTE standard on April 3, 2006. It was primarily marketed as a lower-complexity competitor to the H.264/MPEG-4 AVC standard. After its development, several companies other than Microsoft asserted that they held patents that applied to the technology, including Panasonic, LG Electronics and Samsung Electronics.

LAV or Lav may refer to:

FAAC or Freeware Advanced Audio Coder is a software project which includes the AAC encoder FAAC and decoder FAAD2. It supports MPEG-2 AAC as well as MPEG-4 AAC. It supports several MPEG-4 Audio object types, file formats, multichannel and gapless encoding/decoding and MP4 metadata tags. The encoder and decoder is compatible with standard-compliant audio applications using one or more of these object types and facilities. It also supports Digital Radio Mondiale.

libavcodec is a free and open-source library of codecs for encoding and decoding video and audio data.

Free viewpoint television (FTV) is a system for viewing natural video, allowing the user to interactively control the viewpoint and generate new views of a dynamic scene from any 3D position. The equivalent system for computer-simulated video is known as virtual reality. With FTV, the focus of attention can be controlled by the viewers rather than a director, meaning that each viewer may be observing a unique viewpoint. It remains to be seen how FTV will affect television watching as a group activity.

TDVision Systems, Inc., was a company that designed products and system architectures for stereoscopic video coding, stereoscopic video games, and head mounted displays. The company was founded by Manuel Gutierrez Novelo and Isidoro Pessah in Mexico in 2001 and moved to the United States in 2004.

Video Acceleration API (VA-API) is an open source application programming interface that allows applications such as VLC media player or GStreamer to use hardware video acceleration capabilities, usually provided by the graphics processing unit (GPU). It is implemented by the free and open-source library libva, combined with a hardware-specific driver, usually provided together with the GPU driver.

<span class="mw-page-title-main">VP8</span> Open and royalty-free video coding format released by Google in 2010

VP8 is an open and royalty-free video compression format released by On2 Technologies in 2008.

High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2, is a video compression standard designed as part of the MPEG-H project as a successor to the widely used Advanced Video Coding. In comparison to AVC, HEVC offers from 25% to 50% better data compression at the same level of video quality, or substantially improved video quality at the same bit rate. It supports resolutions up to 8192×4320, including 8K UHD, and unlike the primarily 8-bit AVC, HEVC's higher fidelity Main 10 profile has been incorporated into nearly all supporting hardware.

2D Plus Delta is a method of encoding a 3D image and is listed as a part of MPEG2 and MPEG4 standards, specifically on the H.264 implementation of the Multiview Video Coding extension. This technology originally started as a proprietary method for Stereoscopic Video Coding and content deployment that utilizes the left or right channel as the 2D version and the optimized difference or disparity (Delta) between that image channel view and a second eye image view is injected into the video stream as user data, secondary stream, independent stream, enhancement layer or NALu for deployment. The Delta data can be either a spatial stereo disparity, temporal predictive, bidirectional, or optimized motion compensation.

<span class="mw-page-title-main">DVB 3D-TV</span>

DVB 3D-TV is a new standard that partially came out at the end of 2010 which included techniques and procedures to send a three-dimensional video signal through actual DVB transmission standards. Currently there is a commercial requirement text for 3D TV broadcasters and Set-top box manufacturers, but no technical information is in there.

<span class="mw-page-title-main">Libav</span> Multimedia framework

Libav is an abandoned free software project, forked from FFmpeg in 2011, that contains libraries and programs for handling multimedia data.

Fraunhofer FDK AAC is an open-source library for encoding and decoding digital audio in the Advanced Audio Coding (AAC) format. Fraunhofer IIS, developed this library for Android 4.1. It supports several Audio Object Types including MPEG-2 and MPEG-4 AAC LC, HE-AAC, HE-AACv2 as well AAC-LD and AAC-ELD for real-time communication. The encoding library supports sample rates up to 96 kHz and up to eight channels.

Versatile Video Coding (VVC), also known as H.266, ISO/IEC 23090-3, and MPEG-I Part 3, is a video compression standard finalized on 6 July 2020, by the Joint Video Experts Team (JVET), a joint video expert team of the VCEG working group of ITU-T Study Group 16 and the MPEG working group of ISO/IEC JTC 1/SC 29. It is the successor to High Efficiency Video Coding. It was developed with two primary goals – improved compression performance and support for a very broad range of applications.

References

  1. ↑ https://research.nokia.com/page/4988 Archived 16 July 2015 at the Wayback Machine Mobile 3D Video
  2. ↑ "Archived copy" (PDF). Archived from the original (PDF) on 19 July 2011. Retrieved 13 January 2011.{{cite web}}: CS1 maint: archived copy as title (link) 3DTV AT HOME: STATUS, CHALLENGES AND SOLUTIONS FOR DELIVERING A HIGH QUALITY EXPERIENCE
  3. 1 2 "MVC Patent List" (PDF). MPEG LA . Retrieved 11 July 2019.
  4. ↑ "mpeg4-mvc - 3D Vision Blog" . Retrieved 8 March 2023.
  5. ↑ https://wiki.libav.org/Blueprint/MultiAVFrame Libav MultiAVFrame Blueprint
  6. ↑ Ijsselsteijn, Wijnand A.; Seuntiëns, Pieter J.H.; Meesters, Lydia M.J. (3 January 2006). "Human Factors of 3D Displays". 3D Videocommunication. Chichester, UK: John Wiley & Sons, Ltd. pp. 217–233. doi:10.1002/0470022736.ch12. ISBN   978-0-470-02273-3.
  7. ↑ MPEG – Technologies – Introduction to Multiview Video Coding
  8. ↑ http://vision.princeton.edu/projects/2008/ECCV_stereo/paper_low-res.pdf Learning Two-View Stereo Matching
  9. ↑ http://cs.nyu.edu/~fergus/teaching/vision_2012/6_Multiview_SfM.pdf Lecture 6: Multi-view Stereo & Structure from Motion
  10. ↑ http://www.itu.int/rec/T-REC-H.264 H.264 : Advanced video coding for generic audiovisual services
  11. 1 2 Ho, Yo-Sung; Oh, Kwan-Jung (2007). "Overview of Multi-view Video Coding". 2007 14th International Workshop on Systems, Signals and Image Processing and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services. pp. 5–12. doi:10.1109/IWSSIP.2007.4381085. ISBN   978-961-248-036-3. S2CID   453153.
  12. ↑ blogs.gentoo.org/lu_zero/2014/04/04/the-road-to-mvc/ The road to MVC
  13. ↑ http://www.nt.uni-saarland.de/fileadmin/file_uploads/theses/master/Optimized_implementation_of_a_MVC_decoder.pdf Archived 4 March 2016 at the Wayback Machine Jochen Britz – Optimized implementation of an MVC decoder – Master’s Thesis in Computer and Communication Technology
  14. ↑ https://github.com/Britz/FFmpeg Britz – In terms of my master thesis, I work on a H.264 MVC implementation in libvacodec based on H.264 Annex H.
  15. ↑ https://github.com/Nevcairiel/LAVFilters GitHub repository of LAV Filters project Retrieved 2016-03-30
  16. ↑ http://www.videohelp.com/software/FRIM Videohelp download page Retrieved 2016-03-30
  17. ↑ http://forum.doom9.org/showthread.php?t=169651 developers page of "videohelp3d" Retrieved 2016-03-30
  18. ↑ http://bino3d.org/index.html Home page of Bino - a free 3D video player Retrieved 2016-03-30
  19. ↑ http://forum.doom9.org/showthread.php?t=156191&page=1029 developers page of "Nevcairiel" (at bottom of page) Retrieved 2016-03-30
  20. ↑ CrendKing (10 February 2022), AviSynth Filter and VapourSynth Filter , retrieved 10 February 2022
  21. ↑ "Media: Licensing Programs". MPEG LA. Retrieved 10 February 2022.
  22. ↑ "Licensors Included in the MVC Patent Portfolio License". MPEG LA. Retrieved 11 July 2019.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.