Audio-visual integration in normal and elderly hearing impaired listeners using ventriloquism illusion

Hearing-impaired listeners rely heavily on lip-reading for communication.
Integrating the visual and auditory cues results in a higher understanding of
speech compared to linearly combining the sole modalities. A continuing
hypothesis is that (elderly) hearing-impaired listeners are better audio-visual
integrators because they rely more on visual cues than audio during
communication. We hypothesize that this can be shown using the ventriloquism
effect. As interpretation of the ventriloquism experiment is straightforward,
it provides a useful tool to study audio-visual integration, and is also
ideally suited to compare different groups (e.g. normal and elderly hearing
impaired listeners) and stimuli. For better audio-visual integrators we expect
the illusory effect to occur more strongly. Furthermore, we can directly
compare speech and non speech stimuli to ascertain the difference between both
types of audio-visual integration.

Our research exploring this hypothesis will provide us a better understanding
of audio-visual integration of speech, a topic especially important for
hearing-impaired (and usually also elderly) listeners as many rely on visual
speech cues to supplement poor auditory speech perception due to deafness. It
would give answers to which factors are key in processing multi modal speech
information. E.g. an increased performance with a combined speech and
non-speech stimulus would indicate that visual integration of speech
incorporates many visual cues other than speech reading alone. This would
contribute to e.g. functional brain research and computational
speech-perception models, as well as better designs of communication devices.
In practice, if this hypothesis is proven to be true, hearing-impaired
listeners can benefit from these newly gained insights. For instance, if the
hearing-impaired individual is not proficient in traditional speech-reading,
simplified strategies or other non-speech signals could enhance speech
understanding in day-to-day communication.

In the ventriloquism illusion different sources of audio and visual stimuli
interact and therefore make it more difficult to lateralize the individual
audio source. Hence, the strength of the illusion gives a measure of
audio-visual integration. In the present study, the stimuli will be presented
as speech and non-speech, and both as audio and/or video for the purpose of
quantifying audio-visual integration. To test if the illusionary effect is
greater in better audio-visual integrators, testing will be performed with
normal and elderly hearing-impaired listeners.

Behavioural study. The participants are provided with audio-visual stimuli with
conflicting source locations and are asked to perform a lateralization task.

There are no risks associated with the experiment. The study is designed to
keep burden to a minimum. The maximum duration of the experiment is 4 hours
with adequate breaks build in.