Joint encoding

Not to be confused with Joint source and channel coding.

In audio engineering, joint encoding is the joining of several channels of similar information during encoding in order to obtain higher quality, a smaller file size, or both.

Joint stereo

The term joint stereo has become prominent as the Internet has allowed for the transfer of relatively low bit rate, acceptable-quality audio with modest Internet access speeds. Joint stereo refers to any number of encoding techniques used for this purpose. Two forms are described here, both of which are implemented in various ways with different codecs, such as MP3, AAC and Ogg Vorbis.

Intensity stereo coding

This form of joint stereo uses a technique known as joint frequency encoding, which functions on the principle of sound localization. Human hearing is predominantly less acute at perceiving the direction of certain audio frequencies. By exploiting this characteristic, intensity stereo coding can reduce the data rate of an audio stream with little or no perceived change in apparent quality.

More specifically, the dominance of inter-aural time differences (ITD) for sound localization by humans is only present for lower frequencies. That leaves inter-aural amplitude differences (IAD) as the dominant location indicator for higher frequencies (the cutoff being ~2 kHz). The idea of intensity stereo coding is to merge the lower spectrum into just one channel (thus reducing overall differences between channels) and to transmit a little side information about how to pan certain frequency regions to recover the IAD cues. ITD is not lost completely in this scheme, however: the shape of the ear makes it such that the ITD can be recovered from IAD if the sound comes from free space, e.g. played through loudspeakers. ^[1]

This type of coding does not perfectly reconstruct the original audio because of the loss of information which results in the simplification of the stereo image and can produce perceptible compression artifacts. However, for very low bit rates this type of coding usually yields a gain in perceived quality of the audio. It is supported by many audio compression formats (including MP3, AAC, Vorbis and Opus) but not always by every encoder.

M/S stereo coding

M/S stereo coding transforms the left and right channels into a mid channel and a side channel. The mid channel is the sum of the left and right channels, or ${\displaystyle M=L+R}$. The side channel is the difference of the left and right channels, or ${\displaystyle S=L-R}$. Unlike intensity stereo coding, M/S coding is a special case of transform coding, and retains the audio perfectly without introducing artifacts. Lossless codecs such as FLAC or Monkey's Audio use M/S stereo coding because of this characteristic.

To reconstruct the original signal, the channels are either added ${\textstyle L={\frac {M+S}{2}}}$ or subtracted ${\textstyle R={\frac {M-S}{2}}}$.

This form of coding is also sometimes known as matrix stereo ^{[lower-alpha 1]} and is used in many different forms of audio processing and recording equipment. It is not limited to digital systems and can even be created with passive audio transformers or analog amplifiers. One example of the use of M/S stereo is in FM stereo broadcasting, where ${\displaystyle L+R}$ modulates the carrier wave and ${\displaystyle L-R}$ modulates a subcarrier. This enables backwards compatibility with mono equipment, which will only require the mid channel. ^[2] Another example of M/S stereo is the stereophonic microgroove record. Lateral motions of a stylus represent the sum of two channels and the vertical motion represents the difference between the channels; two perpendicular coils mechanically decode the channels. ^[3]

M/S is also a common technique for production of stereo recordings. See Microphone practice § M/S technique.

M/S encoding does not strictly require that the left and right channels use the same weight. In Opus CELT, M/S encoding is combined with an angle parameter, so that different weights can be used to maximize de-correlation. ^{[4] : 4.5.1}

A similar form of joining multiple channels is seen in the ambisonics implementation of Opus 1.3. A matrix may be used to mix the spherical harmonic channels together, reducing redundancy. ^[5]

Parametric stereo

Parametric stereo is similar to intensity stereo, except that parameters beyond the intensity difference is used. In the MPEG-4 (HE-AAC) version, the intensity difference and time delay difference are used, allowing all bands to be used without hurting localization. HE-AAC also adds "correlation" information, which replicates ambience by synthesizing some difference between channels. ^[6]

Binaural cue coding (BCC) is the HE-AAC PS technique extended for many input channels, all downmixing to one. The very same ILD, ITD, and IC parameters were used. MPEG Surround is similar to BCC, but allows downmixing to multiple channels, and does not seem to use ITD. ^[7]

Joint frequency encoding

Joint frequency encoding is an encoding technique used in audio data compression to reduce the data rate.

The idea is to merge a given frequency range of multiple sound channels together so that the resulting encoding will preserve the sound information of that range not as a bundle of separate channels but as one homogeneous data stream. This will destroy the original channel separation permanently, as the information cannot be accurately reconstructed, but will greatly lessen the amount of required storage space. Only some forms of joint stereo use the joint frequency encoding technique, such as intensity stereo coding.

Implementations

When used within the MP3 compression process, joint stereo normally employs multiple techniques, and can switch between them for each MPEG frame. Typically, a modern encoder's joint stereo mode uses M/S stereo for some frames and L/R stereo for others, whichever method yields the best result. Encoders use different algorithms to determine when to switch and how much space to allocate to each channel; quality can suffer if the switching is too frequent or if the side channel doesn't get enough bits. With some encoding software, it is possible to force the use of M/S stereo for all frames, mimicking the joint stereo mode of some early encoders like Xing. Within the LAME encoder, this is known as forced joint stereo. ^[8]

As with MP3, Ogg Vorbis stereo files can employ either L/R stereo or joint stereo. When using joint stereo, both M/S stereo and intensity stereo methods may be used. As opposed to MP3 where M/S stereo (when used) is applied before quantization, an Ogg Vorbis encoder applies M/S stereo to samples in the frequency domain after quantization, making application of M/S stereo a lossless step. After this step, any frequency area can be converted to intensity stereo by removing the corresponding part of the M/S signal's side channel. Ogg Vorbis' floor function will take care of the required left-right panning.^{[ citation needed ]} Opus similarly has support for all three options in the CELT layer; the SILK layer is M/S-only. ^[9]

Notes

1. So named because the addition and subtraction can be represented by a matrix.

References

1. F. Baumgarte and C. Faller, “Design and evaluation of binaural cue coding,” in AES 113th Conv., Los Angeles, CA, Oct. 2002.
2. "Stereophonic Broadcasting: Technical Details of Pilot-tone System", Information Sheet 1604(4), BBC Engineering Information Service, June 1970
3. "Stereo disc recording". Archived from the original on 25 September 2006. Retrieved 4 October 2006.
4. Jean-Marc Valin; Gregory Maxwell; Timothy B. Terriberry; Koen Vos (October 17–20, 2013). "High-Quality, Low-Delay Music Coding in the Opus Codec" (PDF). www.xiph.org. New York, NY: Xiph.Org Foundation. p. 2. Archived from the original (PDF) on 14 July 2018. Retrieved 19 August 2014. CELT's look-ahead is 2.5 ms, while SILK's look-ahead is 5 ms, plus 1.5 ms for the resampling (including both encoder and decoder resampling). For this reason, the CELT path in the encoder adds a 4 ms delay. However, an application can restrict the encoder to CELT and omit that delay. This reduces the total look-ahead to 2.5 ms.
5. "Opus 1.3 Released". jmvalin.ca. For all higher-order ambisonics, channel mapping 3 provides a more efficient representation by first transforming the ambisonics signals with a designated mixing matrix before encoding. This 1.3 release provides matrices for first, second, and third order.
6. Purnhagen, Heiko (October 5–8, 2004). "LOW COMPLEXITY PARAMETRIC STEREO CODING IN MPEG-4" (PDF). 7th International Conference on Digital Audio Effects: 163–168.
7. HAN, Chih-Kang. MPEG Surround Codec Acceleration and Implementation on TI DSP Platform (PDF) (MSc).
8. "Detailed command line switches". LAME documentation. Retrieved 2013-12-13. JOINT STEREO [...] means the encoder can use (on a frame by frame basis) either L/R stereo or mid/side stereo. In mid/side stereo, [...] more bits are allocated to the mid channel than the side channel. When there isn't too much stereo separation, this effectively increases the bandwidth, so having higher quality with the same amount of bits. Using mid/side stereo inappropriately can result in audible compression artifacts. Too much switching between mid/side and regular stereo can also sound bad. To determine when to switch to mid/side stereo, LAME uses a much more sophisticated algorithm than the one described in the ISO documentation. FORCED MID/SIDE STEREO forces all frames to be encoded with mid/side stereo. It should only be used if you are sure every frame of the input file has very little stereo separation.
9. RFC   6716, §§ 4.2.1, 4.3

External links

• Jürgen Herre, Fraunhofer IIS. From Joint Stereo to Spatial Audio Coding - Recent Progress and Standardization. October 2004, Paper 157, DAFx'04 7th International Conference of Digital Audio Effects.