Phonemic restoration effect

Last updated

Phonemic restoration effect is a perceptual phenomenon where under certain conditions, sounds actually missing from a speech signal can be restored by the brain and may appear to be heard. The effect occurs when missing phonemes in an auditory signal are replaced with a noise that would have the physical properties to mask those phonemes, creating an ambiguity. In such ambiguity, the brain tends towards filling in absent phonemes. The effect can be so strong that some listeners may not even notice that there are phonemes missing. This effect is commonly observed in a conversation with heavy background noise, making it difficult to properly hear every phoneme being spoken. Different factors can change the strength of the effect, including how rich the context or linguistic cues are in speech, as well as the listener's state, such as their hearing status or age.

Contents

This effect is more important to humans than what was initially thought. Linguists have pointed out that at least the English language has many false starts and extraneous sounds. The phonemic restoration effect is the brain's way of resolving those imperfections in our speech. Without this effect interfering with our language processing, there would be a greater need for much more accurate speech signals and human speech could require much more precision. For experiments, white noise is necessary because it takes the place of these imperfections in speech. One of the most important factors in language is continuity and in turn intelligibility.

Phonemic Restoration Effect Phonemic Restoration Effect.jpg
Phonemic Restoration Effect

Background

The phonemic restoration effect was first documented in a 1970 paper by Richard M. Warren entitled "Perceptual Restoration of Missing Speech Sounds". The purpose of the experiment was to give a reason to why in background of extraneous sounds, masked individual phonemes were still comprehensible.

“The state governors met with their respective legislatures convening in the capital city.”

In his initial experiments, Warren provided the sentence shown and first replaced the first 's' phoneme in legislatures with extraneous noise, in the form of a cough. In a small group of 20 subjects, 19 did not notice a missing phoneme and one person misidentified the missing phoneme. This indicated that in the absence of a phoneme, the brain filled in the missing phoneme, through top-down processing. This was a phenomenon that was somewhat known at the time, but no one was able to pinpoint why it was occurring or had labeled it. He again did the same experiment with the sentence:

'“It was found that the wheel was on the axle.”'

He replaced the 'wh' sound in wheel and the same results were found. All people tested wrote down wheel. Warren later did much research for next several decades on the subject. [1]

Since Warren, much research has been done to test the various aspects of the effect. These aspects include how many phonemes can be removed, what noise is played in replacement of the phoneme, and how different contexts alter the effect.

Neuroanatomy

Human auditory cortex Primary auditory cortex.PNG
Human auditory cortex

Neurally, the signs of interrupted or stopped speech can be suppressed in the thalamus and auditory cortex, possibly as a consequence of top-down processing by the auditory system. [2] Key aspects of the speech signal itself are considered to be resolved somewhere in the interface between auditory and language-specific areas (an example is Wernicke's area), in order for the listener to determine what is being said. Normally, the latter is thought to be instantiated at the end stages of the language processing system, but for restorative processes, much remains unknown about whether the same stages are responsible for the ability to actually fill-in the missing phoneme.

Phonemic restoration is one of several phenomena demonstrating that prior, existing knowledge in the brain provides it with tools to attempt a guess at missing information, something in principle similar to an optical illusion. It is believed that humans and other vertebrates have evolved the ability to complete acoustic signals that are critical but communicated under naturally noisy conditions. For humans, while it is not fully known at what point in the processing hierarchy the phonemic restoration effect occurs, [3] evidence points to dynamic restorative processes already occurring with basic modulations of sound set at natural articulation rates. [4] Recent research using direct neurophysiological recordings from human epilepsy patients implanted with electrodes over auditory and language cortex has shown that the lateral superior temporal gyrus (STG; a core part of Wernicke's area) represents the missing sound that listeners perceive. [5] This research also demonstrated that perception-related neural activity in the STG is modulated by left inferior frontal cortex, which contains signals that predict what sound listeners will report hearing up to about 300 milliseconds before the sound is even presented.

Factors

Hearing impairment

People with mild and moderate hearing loss were tested for the effectiveness of phonemic restoration. Those with mild hearing loss performed at the same level of a normal listener. Those with moderate hearing loss had almost no perception and failed to identify the missing phonemes. This research is also dependent on the amount of words the observer is comfortable understanding because of the nature of top-down processing. [6]

Cochlear implants

Cochlear Implant Cochearimplants.JPG
Cochlear Implant

For people with cochlear implants, acoustic simulations of the implant indicated the importance of spectral resolution. [3] When the brain is using top-down processing, it uses as much information as it can to make a decision on if the filler signal in the gap belongs to the speech, and with lower resolution, there is less information to make a correct guess. A study [7] with actual cochlear implant users indicated that some implant users can benefit from phonemic restoration, but again they seem to need more speech information (longer duty cycle in this case) to achieve this.

Age

The age effects were studied in children and older adults, to observe if children can benefit from phonemic restoration and if so, at what capacity, and if older adults maintain the restoration capacity in the face of age-related neurophysiological changes.

Children are able to produce results comparable to adults by about the age of 5, however still not doing as well as adults. At such an early age most information is processed through bottom-up processing due to the lack of information to recall from. However, this does mean they are able to use previous knowledge of words to fill in the missing phonemes with much less of their brain developed than adults. [8] [9]

Older adults (older than 65 years) with no or minimal hearing loss show benefit from phonemic restoration. In some conditions restoration effect can be stronger in older adults than in younger adults, even when the overall speech perception scores are lower in older adults. This observation is likely due to strong linguistic and vocabulary skills that are maintained in advanced age. [10] [11]

Gender

In children, there was no effect of gender on phonemic restoration. [8]

In adults, instead of completely replacing the phonemes, researchers masked them with tones that are informative(helped the listeners pick the correct phoneme), uninformative(neither helped or hurt the listener select the correct phoneme), or misinformative (hurt the listener in picking the correct phoneme). The results showed that women were much more affected by informative and misinformative cues than men. This evidence suggests that women are influenced by top-down semantic information more than men. [12]

Female as opposed to male listeners were better able to use a delayed informative cue at the end of a long sentence to report an earlier word which was disrupted by noise. [13]

Setting

The effect reverses in a reverberation room, which echoes real life more so than the typical quiet rooms used for experimentation. This allows for echoes of the spoken phonemes to act as the replacement noise for the missing phonemes. The additional produced white noise that replaces the phoneme adds its own echo and causes listeners to not perform as well. [14]

Rate

Another study by Warren was done to determine the effect of the duration of the replacement phoneme on comprehension. Because the brain processes information optimally at a certain rate, when the gap became approximately the length of the word is when the effect started top breakdown and become ineffective. At this point the effect is no longer effective because the observer is now cognisant of the gap. [15]

Multisensory

Much like the McGurk Effect, when listeners were also able to see the words being spoken, they were much more likely to correctly identify the missing phonemes. Like every sense, the brain will use every piece of information it deems important to make a judgement about what it is perceiving. Using the visual cues of mouth movements, the brain will you both in top-down processing to make a decision about what phoneme is supposed to be heard. Vision is the primary sense for humans and for the most part assists in speech perception the most. [1]

Context

Because languages are distinctly structured, the brain has some sense of what word is to come next in a proper sentence. When listeners were listening to sentences with proper structure with missing phonemes, they performed much better than with a nonsensical sentence without a proper structure. This comes from the predictive nature of the pre-frontal cortex in determining what word should be coming next in order for the sentence to make sense. Top-down processing relies on the surrounding information in a sentence to fill in the missing information. If the sentence does not make sense to the observer then there will be little at the top of the process for the observer to go off of. If a puzzle piece of a familiar picture was missing, it would be very simple for the brain to know what that puzzle piece would look like. If the picture of something that makes no sense to the human brain and has never been seen before, the brain will have much more difficulty understanding what is missing. [16] [17]

Intensity

Only when the intensity of the noise replacing the phonemes is the same or louder as the surrounding words, does the effect properly work. This effect is made apparent when listeners hear a sentence with gaps replaced by white noise repeat over and over with the white noise volume increasing with each iteration. The sentence becomes more and more clear to the listener as the white noise is louder. [18]

Dichotic listening

When a word with the segment 's' is removed and replaced by silence and a comparable noise segment were presented dichotically. Simply put, one ear was hearing the full sentence without phoneme excision and the other ear was hearing a sentence with a 's' sound removed. This version of the phonemic restoration effect was particularly strong because the brain was doing much less guess work with the sentence, because the information was given to the observer. Observers reported hearing exactly the same sentence in both ears, regardless of one of their ears missing a phoneme. [19]

Language

The restoration effect is studied mostly in English and Dutch, where the restoration effect seemed similar between the two languages. While no research directly compared the restoration effect further for other languages, it is assumed that this effect is universal for all languages.

See also

Related Research Articles

Auditory illusions are false perceptions of a real sound or outside stimulus. These false perceptions are the equivalent of an optical illusion: the listener hears either sounds which are not present in the stimulus, or sounds that should not be possible given the circumstance on how they were created.

<span class="mw-page-title-main">Missing fundamental</span>

A harmonic sound is said to have a missing fundamental, suppressed fundamental, or phantom fundamental when its overtones suggest a fundamental frequency but the sound lacks a component at the fundamental frequency itself. The brain perceives the pitch of a tone not only by its fundamental frequency, but also by the periodicity implied by the relationship between the higher harmonics; we may perceive the same pitch even if the fundamental frequency is missing from a tone.

<span class="mw-page-title-main">Illusory continuity of tones</span> Auditory illusion

The illusory continuity of tones is the auditory illusion caused when a tone is interrupted for a short time, during which a narrow band of noise is played. The noise has to be of a sufficiently high level to effectively mask the gap, unless it is a gap transfer illusion. Whether the tone is of constant, rising or decreasing pitch, the ear perceives the tone as continuous if the discontinuity is masked by noise. Because the human ear is very sensitive to sudden changes, however, it is necessary for the success of the illusion that the amplitude of the tone in the region of the discontinuity not decrease or increase too abruptly. While the inner mechanisms of this illusion is not well understood, there is evidence that supports activation of primarily the auditory cortex is present.

Lip reading, also known as speechreading, is a technique of understanding a limited range of speech by visually interpreting the movements of the lips, face and tongue without sound. Estimates of the range of lip reading vary, with some figures as low as 30% because lip reading relies on context, language knowledge, and any residual hearing. Although lip reading is used most extensively by deaf and hard-of-hearing people, most people with normal hearing process some speech information from sight of the moving mouth.

<span class="mw-page-title-main">McGurk effect</span> Perceptual illusion

The McGurk effect is a perceptual phenomenon that demonstrates an interaction between hearing and vision in speech perception. The illusion occurs when the auditory component of one sound is paired with the visual component of another sound, leading to the perception of a third sound. The visual information a person gets from seeing a person speak changes the way they hear the sound. If a person is getting poor-quality auditory information but good-quality visual information, they may be more likely to experience the McGurk effect. Integration abilities for audio and visual information may also influence whether a person will experience the effect. People who are better at sensory integration have been shown to be more susceptible to the effect. Many people are affected differently by the McGurk effect based on many factors, including brain damage and other disorders.

In speech communication, intelligibility is a measure of how comprehensible speech is in given conditions. Intelligibility is affected by the level and quality of the speech signal, the type and level of background noise, reverberation, and, for speech over communication devices, the properties of the communication system. A common standard measurement for the quality of the intelligibility of speech is the Speech Transmission Index (STI). The concept of speech intelligibility is relevant to several fields, including phonetics, human factors, acoustical engineering, and audiometry.

Speech perception is the process by which the sounds of language are heard, interpreted, and understood. The study of speech perception is closely linked to the fields of phonology and phonetics in linguistics and cognitive psychology and perception in psychology. Research in speech perception seeks to understand how human listeners recognize speech sounds and use this information to understand spoken language. Speech perception research has applications in building computer systems that can recognize speech, in improving speech recognition for hearing- and language-impaired listeners, and in foreign-language teaching.

<span class="mw-page-title-main">Auditory scene analysis</span>

In perception and psychophysics, auditory scene analysis (ASA) is a proposed model for the basis of auditory perception. This is understood as the process by which the human auditory system organizes sound into perceptually meaningful elements. The term was coined by psychologist Albert Bregman. The related concept in machine perception is computational auditory scene analysis (CASA), which is closely related to source separation and blind signal separation.

Auditory processing disorder (APD), rarely known as King-Kopetzky syndrome or auditory disability with normal hearing (ADN), is a neurodevelopmental disorder affecting the way the brain processes sounds. Individuals with APD usually have normal structure and function of the outer, middle, and inner ear. However, they cannot process the information they hear in the same way as others do, which leads to difficulties in recognizing and interpreting sounds, especially the sounds composing speech. It is thought that these difficulties arise from dysfunction in the central nervous system. This is, in part, essentially a failure of the cocktail party effect found in most people.

TRACE is a connectionist model of speech perception, proposed by James McClelland and Jeffrey Elman in 1986. It is based on a structure called "the TRACE," a dynamic processing structure made up of a network of units, which performs as the system's working memory as well as the perceptual processing mechanism. TRACE was made into a working computer program for running perceptual simulations. These simulations are predictions about how a human mind/brain processes speech sounds and words as they are heard in real time.

<span class="mw-page-title-main">Perception of English /r/ and /l/ by Japanese speakers</span> Japanese-language speakers perception of English consonants

Japanese has one liquid phoneme, realized usually as an apico-alveolar tap and sometimes as an alveolar lateral approximant. English has two: rhotic and lateral, with varying phonetic realizations centered on the postalveolar approximant and on the alveolar lateral approximant, respectively. Japanese speakers who learn English as a second language later than childhood often have difficulty in hearing and producing the and of English accurately.

<span class="mw-page-title-main">Lombard effect</span> Involuntary tendency of speakers to increase their vocal effort when in loud environments

The Lombard effect or Lombard reflex is the involuntary tendency of speakers to increase their vocal effort when speaking in loud noise to enhance the audibility of their voice. This change includes not only loudness but also other acoustic features such as pitch, rate, and duration of syllables. This compensation effect maintains the auditory signal-to-noise ratio of the speaker's spoken words.

Psychoacoustics is the branch of psychophysics involving the scientific study of sound perception and audiology—how human auditory system perceives various sounds. More specifically, it is the branch of science studying the psychological responses associated with sound. Psychoacoustics is an interdisciplinary field of many areas, including psychology, acoustics, electronic engineering, physics, biology, physiology, and computer science.

Dichotic listening is a psychological test commonly used to investigate selective attention and the lateralization of brain function within the auditory system. It is used within the fields of cognitive psychology and neuroscience.

Monita Chatterjee is an auditory scientist and the Director of the Auditory Prostheses & Perception Laboratory at Boys Town National Research Hospital. She investigates the basic mechanisms underlying auditory processing by cochlear implant listeners.

Temporal envelope (ENV) and temporal fine structure (TFS) are changes in the amplitude and frequency of sound perceived by humans over time. These temporal changes are responsible for several aspects of auditory perception, including loudness, pitch and timbre perception and spatial hearing.

<span class="mw-page-title-main">Christian Lorenzi</span>

Christian Lorenzi is Professor of Experimental Psychology at École Normale Supérieure in Paris, France, where he has been Director of the Department of Cognitive Studies and Director of Scientific Studies until. Lorenzi works on auditory perception.

Deniz Başkent is a Turkish-born Dutch auditory scientist who works on auditory perception. As of 2018, she is Professor of Audiology at the University Medical Center Groningen, Netherlands.

Robert V. Shannon is Research Professor of Otolaryngology-Head & Neck Surgery and Affiliated Research Professor of Biomedical Engineering at University of Southern California, CA, USA. Shannon investigates the basic mechanisms underlying auditory neural processing by users of cochlear implants, auditory brainstem implants, and midbrain implants.

The speech-to-song illusion is an auditory illusion discovered by Diana Deutsch in 1995. A spoken phrase is repeated several times, without altering it in any way, and without providing any context. This repetition causes the phrase to transform perceptually from speech into song.Though mostly notable with languages that are non-tone, like English and German, it is possible to happen with tone languages, like Thai and Mandarin.

References

  1. 1 2 Warren, Richard M. (1970). "Perceptual Restoration of Missing Speech Sounds". Science. 167 (3917): 392–3. Bibcode:1970Sci...167..392W. doi:10.1126/science.167.3917.392. PMID   5409744.
  2. ↑ Riecke, L.; Vanbussel, M.; Hausfeld, L.; Baskent, D.; Formisano, E.; Esposito, F. (2012). "Hearing an Illusory Vowel in Noise: Suppression of Auditory Cortical Activity". Journal of Neuroscience. 32 (23): 8024–34. doi:10.1523/JNEUROSCI.0440-12.2012. PMC   6620946 . PMID   22674277.
  3. 1 2 Başkent, Deniz (2012). "Effect of Speech Degradation on Top-Down Repair: Phonemic Restoration with Simulations of Cochlear Implants and Combined Electric–Acoustic Stimulation". Journal of the Association for Research in Otolaryngology. 13 (5): 683–92. doi:10.1007/s10162-012-0334-3. PMC   3441953 . PMID   22569838.
  4. ↑ Cervantes Constantino, F.; Simon, J.Z. (2017). "Dynamic cortical representations of perceptual fulling-in for missing acoustic rhythm". Scientific Reports. 7 (1): 17536. doi:10.1038/s41598-017-17063-0. PMC   5727537 . PMID   29235479.
  5. ↑ Leonard, Matthew K.; Baud, Maxime O.; Sjerps, Matthias J.; Chang, Edward F. (2016-12-20). "Perceptual restoration of masked speech in human cortex". Nature Communications. 7 (1): 13619. doi:10.1038/ncomms13619. ISSN   2041-1723. PMC   5187421 . PMID   27996973.
  6. ↑ Başkent, Deniz; Eiler, Cheryl L.; Edwards, Brent (2010). "Phonemic restoration by hearing-impaired listeners with mild to moderate sensorineural hearing loss". Hearing Research. 260 (1–2): 54–62. doi:10.1016/j.heares.2009.11.007. PMID   19922784.
  7. ↑ Bhargava, Pranesh; Gaudrain, Etienne; Başkent, Deniz (2014). "Top–down restoration of speech in cochlear-implant users" (PDF). Hearing Research. 309 (4): 113–123. doi:10.1016/j.heares.2013.12.003. PMID   24368138.
  8. 1 2 Newman, Rochelle S. (2004). "Perceptual restoration in children versus adults". Applied Psycholinguistics. 25 (4): 481–93. doi:10.1017/S0142716404001237.
  9. ↑ Pattison, Darcy Sue (1976). Phonemic restoration in nursery school children (Thesis). Kansas State University. hdl:2097/11418. OCLC   33842958.[ page needed ]
  10. ↑ Saija, Jefta D.; Akyürek, Elkan G.; Andringa, Tjeerd C.; Başkent, Deniz (2014). "Perceptual Restoration of Degraded Speech Is Preserved with Advancing Age". Journal of the Association for Research in Otolaryngology. 15 (1): 139–148. doi:10.1007/s10162-013-0422-z. PMC   3901857 . PMID   24198087.
  11. ↑ Jaekel, Brittany N.; Newman, Rochelle S.; Goupell, Matthew J. (2018). "Age effects on perceptual restoration of degraded interrupted sentences". The Journal of the Acoustical Society of America. 143 (1): 84–97. doi:10.1121/1.5016968. PMC   5758365 . PMID   29390768.
  12. ↑ Liederman, Jacqueline; Gilbert, Kristen; Fisher, Janet McGraw; Mathews, Geetha; Frye, Richard E.; Joshi, Pallavi (2011). "Are Women More Influenced than Men by Top-down Semantic Information when Listening to Disrupted Speech?". Language and Speech. 54 (1): 33–48. doi:10.1177/0023830910388000. PMID   21524011.
  13. ↑ Liederman, Jacqueline; Fisher, Janet McGraw; Coty, Alexis; Matthews, Geetha; Frye, Richard E.; Lincoln, Alexis; Alexander, Rebecca (February 2013). "Sex differences in the use of delayed semantic context when listening to disrupted speech". Archives of Sexual Behavior. 42 (2): 197–201. doi:10.1007/s10508-012-9941-7. ISSN   1573-2800. PMID   22434397.
  14. ↑ Srinivasan, Nirmal Kumar; Zahorik, Pavel (2012). "Phonemic restoration effect reversed in a reverberant room". The Journal of the Acoustical Society of America. 131 (1): EL28–34. Bibcode:2012ASAJ..131L..28S. doi:10.1121/1.3665120. PMC   3261052 . PMID   22280726.
  15. ↑ Bashford, James A.; Meyers, MD; Brubaker, BS; Warren, RM (1988). "Illusory continuity of interrupted speech: Speech rate determines durational limits". The Journal of the Acoustical Society of America. 84 (5): 1635–8. Bibcode:1988ASAJ...84.1635B. doi:10.1121/1.397178. PMID   3209768.
  16. ↑ Groppe, David M.; Choi, Marvin; Huang, Tiffany; Schilz, Joseph; Topkins, Ben; Urbach, Thomas P.; Kutas, Marta (2010). "The phonemic restoration effect reveals pre-N400 effect of supportive sentence context in speech perception". Brain Research. 1361: 54–66. doi:10.1016/j.brainres.2010.09.003. PMC   2963680 . PMID   20831863.
  17. ↑ Sivonen, Päivi; Maess, Burkhard; Lattner, Sonja; Friederici, Angela D. (2006). "Phonemic restoration in a sentence context: Evidence from early and late ERP effects". Brain Research. 1121 (1): 177–89. doi:10.1016/j.brainres.2006.08.123. PMID   17027933.
  18. ↑ Shinn-Cunningham, Barbara G.; Wang, Dali (2008). "Influences of auditory object formation on phonemic restoration". The Journal of the Acoustical Society of America. 123 (1): 295–301. Bibcode:2008ASAJ..123..295S. doi:10.1121/1.2804701. PMID   18177159.
  19. ↑ Eimas, Peter D.; Tajchman, G; Nygaard, LC; Marcus, DJ (1996). "Phonemic restoration and integration during dichotic listening". The Journal of the Acoustical Society of America. 99 (2): 1141–7. Bibcode:1996ASAJ...99.1141E. doi:10.1121/1.414598. PMID   8609298.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.