Multisensory perception in schizophrenia, A comparative study between healthy and schizophrenic individuals

One of the principal functions of the brain is to integrate signals from
multiple modalities (e.g. face and voice) into coherent multisensory
representations of objects and events. Although there are relatively few
studies on multisensory integration in schizophrenia it is reasonably well
established that schizophrenics have deficits in several aspects of
multisensory integration. Relevant for the purpose of our study is that
schizophrenic patients show impairments in audiovisual speech integration. This
was demonstrated with the use of the mcGurk effect (referring to the illusion
that observers report to *hear* /ada/ when presented with auditory /aba/ and
visual /aga/, which is taken as evidence that at some point during stimulus
processing visual and auditory signals are fused). Schizophrenics are less
susceptible to the McGurk effect than normal controls. It has been argued that
the neural substrate of this deficit is likely to be a higher-order speech
integration region such as the superior temporal sulcus/gyrus (STS/G). A
problem is that such claims are mostly based on data gathered at the
behavioural level. What is needed are studies investigating more directly the
possible neural substrates of impaired audiovisual perception.

Schizophrenia may affect the basic brain processing for time phenomenology as
patients with schizophrenia show an abnormally low time resolution. A
behavioral experiment is conducted to investigate audiovisual time perception.
Abnormalities in audiovisual integration (enlarged temporal window of
integration) are assed using the temporal ventriloquist effect.

The goal of the present study is to gain understanding of the brain mechanisms
underlying the impairment of multisensory integration processes in
schizophrenia. The study will support the development of more precise criteria
for the diagnosis of schizophrenia and more efficient treatment programs for
schizophrenia.

We will use experimental paradigms that in previous experiments with healthy
subjects have yielded robust neural markers of audiovisual integration using
Electroencephalography (EEG) .

1) Mismatch negativity evoked by McGurk effect.
The Mismatch negativity (MMN) is an auditory brain potential (ERP) generated in
the primary and secondary auditory cortex which is evoked to an infrequent
discernible change in an acoustic feature (e.g. change in pitch) among sequence
of identical sounds and reflects pre-attentive auditory deviance detection. The
MMN has been successfully used to probe the neural mechanisms underlying
audiovisual integration. Typically, in these studies audiovisual conflict
situations are created such as the McGurk effect in which the visual signal
affects the auditory percept thereby either evoking the MMN. In the MMN
experiment study we present infrequent incongruent audiovisual utterances
inducing the McGurk effect (e.g. visual /ga/ paired with auditory /pa/) among
congruent audiovisual standards (auditory and visual /pa/). In normal subjects
this evokes the MMN. The MMN can thus be evoked in spite of the fact that there
os no acoustical difference between the deviants and standards. If
schizophrenics show a diminished McGurk effect (i.e. less audiovisual
integration) it is expected that the MMN in the schizophrenic group is smaller
or even absent compared to the healthy control group.

2) Additive model approach*
In the additive model approach experimentnatural speech stimuli, human actions
and artificial stimuli are presented in three different modes: audiovisual
(AV), auditory-only (A), and visual-only (V). As customary in cross-modal ERP
studies AV interactions are determined by the following equation: AV = A + V.
Super-additive (AV > A + V) and sub-additive (AV < A + V) effects at a given
time in the ERP are considered to be the neural consequence of multisensory
integration. The use of the additive model provides the exact moment at which
auditory and visual signals are integrated and which brain areas are involved.
By comparing the two groups we can find out whether the same neural networks
are involved in audiovisual perception and whether this is depended on the type
of presented stimuli. Here, the interest is specially focused on the contrast
between audiovisual processing of speech and non-speech events.

3)Temporal ventriloquism effect
Schizophrenic patients have abnormalities in temporal and multisensory
processing which may lead to deficits in audiovisual perception. In the third
experiment we want to extend our knowledge on multisensory and temporal
processing in schizophrenia by examining the *temporal ventriloquism effect*
with schizophrenic individuals. The basic phenomenon is that when a sound and a
light are presented at slightly different onset times (usually in the order of
~100 ms), it appears as if the light is shifted in time towards the sound.
Previous studies with healthy participants have shown a so-called *temporal
ventriloquist effect* . In the normal population, clicks at ~100 ms and ~200 ms
audiovisual delays have been shown to cause perceived visuel temporal shifts,
but not at larger delays because the sounds then fall outside the *temporal
window of integration*. The critical question we address here is whether the
same phenomenon can be observed in the schizophrenic population and if so,
whether the effective audiovisual delays may be different (i.e. larger) as a
result of a possible enlarged temporal window for multisensory integration.

There are no known risks related to this study. Therefore we do not foresee any
difficulties that could lead to any medical, mental or physical problems. As
participants constantly watch a screen there may be temporary and reversible
occurrences of heightened fatigue. However, we reduced the occurrence of
fatigue conducting the experiments on two different days (with a minimum of at
least 2 days between the sessions) and by dividing the experiments into blocks
of short duration (about six minutes) with breaks of sufficient duration
between them.

Participation will result in a better fundamental understanding of the
mechanisms of multisen-sory perception and will provide a better insight in the
disordered brain function of schizophrenic patients. Specifically, the studies
will uncover impairments in (functional) connectivity between auditory and
visual processing areas. Second, the study will support the development of more
precise criteria for the diagnosis of schizophrenia and more efficient
treatment programs for schizophrenia.

This study is group-related, because it cannot be conducted without the
participation of schizophrenic patients (all belonging to one group).