Crossmodal integration: an emerging field in psychopathology

In our daily life, we are constantly immerged in a stream of stimulations coming from different sensorial modalities, and these stimulations have to be integrated in a coherent and unitary percept (Campanella & Belin, 2007). This integration ability, which offers us the opportunity to understand our social and perceptual environment and to produce adapted behavioural responses, relies on crossmodal processing (Driver & Spence, 2000). For instance, we are able to integrate the auditory information of what is said and the visual information of who is saying it, so that we can attribute a particular speech to a particular person (Kerlin et al., 2010) and thus take part in a conversation. In view of the obvious importance of these crossmodal processes, many studies have investigated their behavioural and cerebral correlates among healthy participants, notably leading to the identification of several brain areas dedicated to multisensory integration (Joassin et al., 2011a, 2011b; Love et al., 2011). The exploration of crossmodal mechanisms thus constitutes an established field in the experimental psychology and neuroscience domains (Amedi et al., 2005; Calvert et al., 2001; De Gelder & Bertelson, 2003) and has now come to maturity, as illustrated by the proposal of integrative models (e.g. Campanella & Belin, 2007). It is now well established that the auditory–visual integration of human faces and voices during the multimodal processing of identity and gender is associated with the activation of a specific network of cortical and subcortical regions. This network includes several regions devoted to the different cognitive processing implied in face and voice categorization task, notably (a) the unimodal visual and auditory regions processing the perceived faces and voices, which are inter-connected via a subcortical relay located in the striatum, (b) the left superior parietal gyrus, part of a larger parieto-motor network dispatching the attentional resources to the visual and auditory modalities, and (c) the right inferior frontal gyrus sustaining the integration of the semantically congruent information into a coherent multimodal representation (Joassin et al., 2011a; 2011b).

However, these flourishing explorations of normal crossmodal processing strikingly contrast with the scarcity of studies exploring these crossmodal abilities in clinical populations, and particularly in psychiatric states. Indeed, while several studies have been conducted in schizophrenia (De Gelder et al., 2005; Ross et al., 2007; de Jong et al., 2009; Pearl et al., 2009; Szycik et al., 2009; Seubert et al., 2010a; Van den Stock et al., 2011), autism (Foss-Feig et al., 2010; Kwakye et al., 2011; Mongillo et al., 2008; van der Smagt et al., 2007) and Alzheimer’s disease (Delbeuck et al., 2007), suggesting wide-range crossmodal impairments in these populations, crossmodality has not been explored in other psychiatric states. Many questions thus remain to be addressed concerning crossmodal integration in psychiatry.

A main limitation for the understanding of crossmodal processing is thus, in our view, the paucity of the available results concerning impaired integration in clinical populations. Indeed, pathological states are classically used in neuropsychology and neuroscience to test and extend the observations made in healthy populations (Laurienti et al., 2005) but this research perspective has not yet been followed for multisensory integration. A central aim of this paper is to underline the usefulness of exploring crossmodal processing in pathological states, as this might offer a better understanding of crossmodal impairments in these populations, but also renew the knowledge on healthy crossmodal integration. Following this objective, the next sections will first propose a description of recent results proposing the first exploration of emotional crossmodal processing in alcohol-dependence, in order to exemplify the perspectives offered by this research field. Then, these preliminary results will be discussed to illustrate how they might, together with other results obtained in other psychiatric states, end up in the raising of an ambitious research program using various psychiatric populations and sensory modalities.

Alcohol-dependence: the importance of emotional deficits

Alcohol abuse and dependence are directly involved in more than 200.000 deaths per year in Europe, and alcohol-dependence is the most frequent psychiatric diagnosis, being among the more important public health problems worldwide (Harper & Matsumoto, 2005). The extent of the negative effects of excessive alcohol consumption led many researchers to explore alcohol-dependence at clinical and fundamental levels during the last decades, particularly concerning the physiological, cognitive and cerebral impairments. Alcohol-dependence is known to have deleterious impact on many body systems, and crucially on the central nervous system. It has indeed been largely shown that alcohol-dependent individuals present major cerebral damage (see Bülher & Mann, 2011 for a review) in various brain regions (Harper, 2007; McIntosh & Chick, 2004), including white matter (Oscar-Berman & Marinkovic, 2003), limbic (Cowen et al., 2004; De Bellis et al., 2005) and fronto-temporal regions (Chanraud et al., 2007; Harper & Matsumoto, 2005). The behavioural correlates of these cerebral impairments are also well established as cognitive deficits have been repeatedly shown (see Stavro et al., 2013 for a review) for perceptive-motor (Kramer, et al., 1989; Spitzer, 1981), attentional (Noël et al., 2001), memory (Pitel et al., 2007) and executive abilities (Bechara et al., 2001). Nevertheless, this extensive exploration of the cognitive consequences of alcohol-dependence contrasts with the poor knowledge on other abilities which are also crucial in this pathology, and notably emotional disturbances.

Indeed, while alcohol-dependence has classically been considered as related to impaired cognitive functions, and particularly to reduced inhibition abilities, the most recent and influential theoretical proposals on alcohol-dependence, namely the dual-process models, renewed this perspective. Indeed, these models (Noël et al., 2010; Wiers et al., 2013) postulate the existence of two cerebral systems: (1) The “reflective system”, involved in the cognitive evaluation of the stimuli relying on memory and executive functions, initiating the controlled-deliberate responses and mainly relying on frontal areas; (2) The “affective-automatic system”, involved in the emotional evaluation of the stimuli, initiating automatic-appetitive responses and mainly relying on limbic regions. Following this view, addictive behaviours are not only due to impaired reflective system (i.e. reduced cognitive abilities), but rather to an imbalance between systems due to a combined hypo-activity of the reflective system and hyper-activity of the affective-automatic one. Importantly, these models underline the importance to go beyond cognitive functions in alcohol-dependence, and singularly to understand this automatic-affective system. It thus seems urgent to explore the affective deficits in alcohol-dependence.

Affective disturbances indeed constitute a crucial factor in most mental diseases, and singularly in alcohol-dependence as it has been shown that emotional problems are the main reason for more than 50% of the relapse appearing during the six months after detoxification (Zywiak et al., 2003). Despite this obvious importance of emotions in clinical settings, the experimental exploration of affective impairments in alcohol-dependence has only been developed recently. These studies clearly confirmed that alcohol-dependence is associated with major impairments in various emotional abilities, from emotional intelligence (Cordovil de Susa Uva et al., 2010) to alexithymia (Taieb et al., 2002), empathy (Maurage et al., 2011a) and irony comprehension (Amenta et al., 2013). Centrally, several studies explored the emotion decoding abilities among alcohol-dependent individuals, and converged to the proposal that alcohol-dependence is linked with massive decoding impairments of the affective content of faces (Frigerio et al., 2002; Marinkovic et al., 2009; Maurage et al., 2009; 2011b) and voices (Monnot et al., 2002; Monnot et al., 2001; Uekermann et al., 2005). More precisely, recently detoxified alcohol-dependent individuals overestimate the intensity of the negative emotions conveyed by visual and auditory stimuli, have an erroneous interpretation of these emotions and are not conscious of their deficit (Kornreich et al., 2001; 2002; Philippot et al., 1999). This emotional decoding deficit has been observed among individuals with various abstinence durations and in various paradigms (e.g. morphed or ambiguous faces, more complex emotional stimuli, Maurage et al., 2011a; b), and is now strongly established (Foisy et al., 2007; Montagne et al., 2006; Townshend & Duka, 2003). In summary, alcohol-dependence is associated with a global emotional stimulations decoding deficit, present for faces, voices, but also for music (Kornreich et al., 2013) or body postures (Maurage et al., 2011c).

Offering a deeper exploration of these emotional disabilities thus appears crucial at theoretical level but also for clinical practice, as they play a critical role in the emergence and persistence of alcohol-dependence. Indeed, affective impairments have been shown to influence interpersonal relations and to reinforce social isolation, which is a key factor participating in the maintenance of the pathological state (Schomerus et al., 2011; Walitzer & Dearing, 2006). Emotion decoding impairment is now clearly identified in alcohol-dependence and has a high clinical importance, notably in view of its links with interpersonal problems. However, this deficit has up to now been exclusively explored using paradigms with low ecological validity, namely using unimodal stimuli (faces or voices presented separately). It is thus unclear whether this deficit is maintained, reduced or increased in experimental designs that are closer to real life, specifically when crossmodal stimuli are used.

Crossmodal emotional deficits in alcohol-dependence

The omnipresence of crossmodal phenomena in everyday life underlines the importance of exploring the integration processes in psychological states to obtain a more precise and ecological description of the deficits presented in real-life situations. This is particularly true for affective stimulations, as the perception, decoding and production of emotional states are most often based on various sensory stimulations, combining visual (e.g. face, posture, movements) and auditory (e.g. onomatopoeia, prosody) aspects. As underlined above, many studies have recently explored emotional deficits in alcohol-dependence but, while they offered a very interesting first description of emotional disturbances, they were all focused on unimodal stimulations and are thus unable to fully describe the complexity of emotion processing in this population. In order to fill in this gap and to extend this research field towards ecological crossmodal designs, we conducted three studies to explore for the first time the crossmodal emotional processing in recently detoxified alcohol-dependent participants, compared with paired healthy controls. As these studies were based on a multidisciplinary method combining neuropsychology and neuroscience to offer different perspectives on this topic, this section will also offer the opportunity to insist on the usefulness of a complementary approach combining psychology and neuroscience tools.

(1) Behavioural study (Maurage et al., 2007): this initial exploration centrally aimed at measuring a first index of crossmodal processing in alcohol-dependence, namely the “crossmodal facilitation effect”, based on the simple principle that, in an emotion decoding task, the performance will be increased (i.e., reduced error rates and/or shorter reaction times) when presenting crossmodal stimuli (emotionally congruent face-voice association) compared to unimodal ones (isolated face or voice), as the face-voice combination will bring more information, thus facilitating the emotional identification. This facilitation effect has been repeatedly described in healthy populations and is considered as a reliable index of efficient crossmodal integration (Calvert et al., 2001; Teder-Sälejärvi et al., 2002). Conversely, the reduction or absence of this effect is considered as reflecting impaired crossmodal processing. In the study, we proposed an emotion decoding task in which participants had to decide whether faces, voices or face-voice combinations presented angry or happy emotion. The central result of this study was that alcohol-dependent individuals did not present the crossmodal facilitation effect present among controls. In other words, while healthy participants presented shorter reaction times in crossmodal situations, no significant difference was observed in alcohol-dependence between unimodal and crossmodal experimental conditions, which indexes an impaired integration of complex ecological stimuli. Capitalizing on this first behavioural description of a crossmodal integration deficit in alcohol-dependence, we then extended this exploration by investigating the brain correlates of these impaired integration abilities.

(2) Electrophysiological study (Maurage et al., 2008): electrophysiological tools, and particularly event-related potentials (ERP) have a high temporal resolution allowing to precisely spot the successive processing stages related to a cognitive function (e.g. perceptual, attentional or decisional steps), and thus to identify the stage at which a potential deficit begins (Rugg and Coles, 1995). In this perspective, ERP recording was performed during a crossmodal emotional task among alcohol-dependent individuals to determine the specific stage at which the crossmodal deficit occurs. Behavioural results first confirmed the results obtained in the first study by showing impaired crossmodal integration in alcohol-dependence. More centrally, the specific ERP components associated with crossmodal integration were identified using a classical subtraction technique (Joassin et al., 2004; Teder-Sälejärvi et al., 2002), and a clear alteration of the cerebral activity related to these integrative processes was identified among alcohol-dependent individuals compared to controls. Indeed, a reduced amplitude (indexing reduced brain activity) and a delayed latency (indexing slower brain functioning) were found for the crossmodal electrophysiological components in alcohol-dependence, particularly for anger stimulations, which were associated with a strong reduction of crossmodal processing in frontal regions. However, due to the low spatial resolution of these electrophysiological tools, this study did not allow to identify the precise brain areas involved in this deficit, and these results had thus to be extended by neuroimaging explorations.

(3) Neuroimaging study (Maurage et al., 2013a): functional magnetic resonance imaging (fMRI) was here used to specifically spot the brain regions involved in the crossmodal deficit observed in alcohol-dependence, and particularly to determine whether alcohol-dependent individuals presented an impaired activation of the specific crossmodal regions classically described in healthy controls (Joassin et al., 2011a). A similar emotion detection task than the one described in the two first studies was used while brain activation was recorded. The main result of this study was that, while alcohol-dependent individuals presented relatively preserved brain activations during the separate processing of faces and voices, they showed a massive reduction of the brain activations in the regions specifically involved in the crossmodal face-voice integration. Indeed, healthy controls showed specific crossmodal activations in classical multimodal regions (i.e. middle frontal gyrus, superior parietal lobule and superior parietal gyrus), but these regions were not significantly activated during crossmodal integration in the alcohol-dependent group. This study went one step further by exploring the potential origin of this specific hypo-activation in the integration regions. Indeed, to determine whether the reduced activation in crossmodal areas was due to impaired functioning of these regions per se or rather to reduced input coming to these areas from unimodal ones, psycho-physiological interactions were performed, measuring the functional connectivity between unimodal and crossmodal areas. While healthy controls presented a reliable connectivity pattern with strong functional connectivity between unimodal and crossmodal areas in crossmodal conditions, underlining the efficient functioning of the integration network, alcohol-dependence was associated with a total disconnection between unimodal and crossmodal regions. The crossmodal integration impairments observed in alcohol-dependence might thus be partly linked to altered connections within the crossmodal network, reducing the information transfer between unimodal and crossmodal regions.

These three studies are of course very preliminary and should be considered as a first exploratory step showing the potential interest of further exploring crossmodal processing in alcohol-dependence and in other psychiatric states. Indeed, while we only focused on a specific clinical population and on a very simple paradigm using a limited variety of emotions, our hope is that, combined with other data obtained among other psychiatric populations, these results might constitute a promising background for the expansion towards an ambitious research program able to determine the cognitive and brain correlates of impaired crossmodal integration in psychiatry. The next section will identify the main avenues for future research, and we will end this paper by underlining the potential clinical and fundamental implications that can be expected from the development of this research field.

What should be done in future research?

In view of the very few studies related to the exploration of crossmodal processing in psychiatry, nearly everything remains to be done in this research field, but we will try to identify four main questions that should be addressed in future studies to deepen the understanding of these integration deficits:

(1) Is the crossmodal deficit specific for emotional stimuli? The three studies presented above used emotional stimuli and showed a deficit in emotional crossmodal integration. However, as they did not use a control experimental condition exploring the integration of non-emotional stimulations in alcohol-dependence, it cannot be concluded that the deficit observed is specific for emotions, as it could alternatively be part of a more general crossmodal deficit present whatever the stimulation type. Further exploration is thus urgently needed to directly compare the integration of emotional and non-emotional stimuli in alcohol-dependence in order to clarify this question. Similar works have been recently conducted in schizophrenia, where it has been shown that the crossmodal deficit repeatedly described for complex emotional stimuli was at least partly the consequence of a more general integration deficit also observed for non-emotional social stimuli (Pearl et al., 2009). Moreover, it has been suggested that the deficit observed for emotional face decoding among schizophrenic patients might only be part of a global perceptive deficit leading to impaired processing of every stimulation, including very perceptually basic ones (Norton et al., 2009). Concerning alcohol-dependence, it has been shown that the deficit presented by alcohol-dependent individuals to identify emotional facial expressions was absent for other complex face processing (Maurage et al., 2009), leading to the proposal that emotional processing was specifically impaired in alcohol-dependence. However, the specificity of the crossmodal deficit for emotional integration has not been tested yet, and future studies should thus directly compare the ability shown by alcohol-dependent participants in the integration of basic stimulations (e.g. geometric shapes and blank sounds), non-emotional social stimuli (e.g. gender perception) and emotional stimuli.

(2) Does the crossmodal deficit vary across emotions? The use of a very small number of emotions is a main limitation of the three studies presented above, and a crucial aim of future studies should be to use a wider range of emotional states in order to explore the differential crossmodal deficit across emotional states in alcohol-dependence. Centrally, it has been shown in face decoding studies that alcohol-dependent individuals present massive alterations in the processing of anger stimulations as compared to other negative emotions, and it would be useful to explore whether this unimodal deficit is generalizable to crossmodal situations. It is thus urgent to develop crossmodal paradigms using a broader set of emotions, and particularly of negative ones (e.g. disgust, fear, sadness), to clarify the proposal of an anger-specific deficit. More globally, several studies have shown that the emotional deficits in alcohol-dependence are not restricted to basic emotion decoding but are also found for more complex emotional abilities (e.g. empathy, emotional intelligence), and this should encourage future studies to develop new paradigms using crossmodal stimulations to explore these emotional and interpersonal processes. Indeed, crossmodal designs might be used as an efficient tool to renew the exploration of these processes by developing more ecological explorations and thus by offering a more valid description of alcohol-related affective disorders.

(3) Is the crossmodal deficit also present for other sensorial modalities? Our previous studies are in line with the general trend followed in crossmodal literature, namely the focus on visuo-auditory integration. While this emphasis on vision and audition appears justified in view of their dominance in everyday life, the total absence of data concerning the other senses is in our view a central limit of the current knowledge. Indeed, olfaction and taste are known to play a crucial role in the daily life of healthy and clinical populations (Schiffman, 1997) and carry important emotional information (e.g. Greimel et al., 2006; Sheperd, 2006; Winston et al., 2005). Exploring the crossmodal integration of stimuli coming from these underexplored modalities and their interactions with visual and auditory ones could thus renew crossmodal integration knowledge, and particularly for alcohol-dependence. Indeed, while olfactory function has been very little explored in this pathology, it has been shown that olfactory cues are strongly involved in the appearance and persistence of alcohol-related disorders (e.g. Kareken et al., 2004; Little et al., 2005) and that olfactory deficits might be associated with cognitive alterations (Maurage et al., 2011c). Olfaction research might thus become a blooming research field in alcohol-dependence and more globally in psychiatric states. But again, all the studies performed up to now have exclusively used unimodal stimulation while in real life situations, olfactory stimulations frequently occur in interaction with cues coming from other sensory modalities. This is particularly true for emotional contexts, as illustrated by recent results among healthy controls showing a strong influence of olfactory cues on the decoding of facial expressions (Leppänen & Hietanen, 2003). However, while olfactory-visual integration impairments have recently been described in schizophrenia, suggesting that the crossmodal deficit might be independent of the modalities involved (Seubert et al., 2010a, 2010b), these vision-olfaction interactions have not been tested in alcohol-dependence. Future studies should thus explore, among healthy as well as clinical populations, the correlates of the crossmodal integration between the “chemical senses” and visual or auditory stimulations.

(4) Is the crossmodal deficit also present in other alcohol-related problems and psychiatric states? The crossmodal studies described above focused on recently detoxified alcohol-dependent patients, and these explorations should thus be extended to other populations presenting excessive alcohol consumption, and beyond to other addictions and psychiatric states. Concerning alcohol-related problems, while the literature classically focused on installed alcohol-dependence, a new research field progressively rose during the last decade, aiming at exploring the deficits associated with earlier stages of alcohol abuse. Particularly, the cognitive and cerebral correlates of binge drinking, globally defined as an excessive but episodic consumption pattern very frequently observed in youth, have been recently explored, showing that this habit rapidly leads to strong cognitive (e.g. Townshend & Duka, 2005; Zeigler et al., 2005) and brain negative effects (Campanella et al., 2013; Maurage et al., 2009b; Schweinsburg et al., 2010; Petit et al., in press, for a review). Interestingly, it has recently been shown that binge drinkers present impaired processing of facial emotional stimuli (Maurage et al., 2013b), but the generalization of this deficit towards more ecological stimulations, and particularly crossmodal ones, has not been tested yet. Exploring emotional audio-visual integration in binge drinking would thus clarify the origin of the crossmodal deficit in alcohol-dependence and give some crucial insights on the implication of these deficits in the development of alcohol-dependence, as binge drinking is now considered as a first step towards alcohol-dependence (Enoch, 2006). Concerning the exploration of emotional crossmodal processing in psychiatry, while several studies have been conducted in schizophrenia and autism, the other psychiatric states have been totally neglected. In view of the repeated description of unimodal emotional decoding deficits in depression, anxiety or anorexia nervosa (e.g. Bhatara et al., 2010; Mendlewicz et al., 2005; Rossignol et al., 2005), it would be interesting to explore the evolution of these deficits in crossmodal settings. Indeed, it might be hypothesized that these psychiatric states will lead, as it has been observed in alcohol-dependence, to increased deficits in crossmodal stimulations. But conversely, some of these patients might also beneficiate from crossmodal stimulations to partly compensate their emotional decoding deficits.

Conclusion: fundamental and clinical implications

As underlined above, the exploration of crossmodal processes in alcohol-dependence is still in its infancy and the crucial questions presented in the previous section should be addressed to strengthen and develop this research field. However, in view of the results observed in our three studies, we believe that several implications might be underlined at fundamental and clinical levels, in order to already clarify what can be expected from this research field concerning potential therapeutic interventions and future experimental investigations of these integrative processes.

At the fundamental level, the description of an increased emotional decoding deficit in crossmodal situations, which are more the rule than the exception in everyday life, suggests that earlier experimental studies which were focused on unimodal stimulations might have underestimated the affective deficits presented by alcohol-dependent individuals. In other words, alcohol-dependence might lead to stronger emotional impairments in real-life than usually described in unimodal studies, which should encourage future studies to use crossmodal stimulations to more accurately evaluate the emotional deficits. Moreover, the ubiquity of crossmodal stimulations in everyday life argues for the development of more ecological paradigms, which is allowed by the recent emergence of new techniques like virtual reality environments. At the theoretical level, a better understanding of crossmodal processing in alcohol-dependence and other clinical states could complement the results obtained among healthy participants and renew the current models of crossmodality. For example, our fMRI results described above and showing hypo-activation in specific integration regions among alcohol-dependent individuals confirm that these areas are necessary for efficient crossmodal integration, thus reinforcing the results obtained among healthy populations. These first data thus illustrate the proposal that a better understanding of impaired crossmodal processing in psychiatry might also lead to an improved comprehension of normal integration processes.

At the therapeutic level, our results clearly confirm earlier ones underlining the role played by emotional deficits in alcohol-dependence, and extend them by suggesting that these deficits are even increased in more ecological paradigms. This claims for the inclusion of these emotional disturbances in the current models of alcohol-dependence. Indeed, affective variables have been largely neglected in the classical models of alcohol-dependence, focusing on cognitive and behavioural aspects (e.g. coping strategies, motivation to change), and thus also in the therapeutic proposals linked to these models. As underlined above, more recent models, and particularly the dual-process models, underlined the importance of affective variables in alcohol-dependence (Noël et al., 2010; Wiers et al., 2013), giving the opportunity to develop therapeutic proposals focusing on these emotional abilities. However, while emotional rehabilitation programs have recently been developed and tested in other clinical populations, they have not yet been applied to alcohol-dependence. In view of the importance of emotional deficits in relapse risk, future studies should test the efficiency of these programs in alcohol-dependence, and a further step might be done by developing crossmodal emotional rehabilitation programs in order to propose more realistic emotion decoding therapeutic programs. Another therapeutic perspective is the use of crossmodal stimulations to improve the diagnosis of psychiatric states. Indeed, ERP have been largely used during the last decades in clinical settings to complement the psychiatric diagnosis. The classical paradigm is based on the elicitation of the P300 component by visual or auditory stimuli. However, the effective usefulness of such exploration remains weak, as it has been repeatedly shown that the P300 is impaired in a wide range of psychiatric states and is thus not reliable enough to be useful for producing a precise diagnosis at the individual level (Pogarell et al., 2007). Interestingly, recent results (Campanella et al., 2010; 2012; Delle-Vigne et al., 2014) have shown that using crossmodal audio-visual stimulations instead of unimodal ones might improve the sensitivity of this ERP investigation. Indeed, it has been observed that the P300 elicited by crossmodal stimulations was far more efficient to discriminate healthy controls from subclinical anxious patients. This has not been tested in alcohol-dependence yet, but using crossmodal stimuli to refine ERP explorations in clinical settings and improve their diagnostic power could constitute an interesting perspective in the years to come.

In conclusion, the main aim of this paper was to underline the huge gap between the blooming of crossmodal processing explorations in healthy controls, which led to crucial advances in the last decade, and the surprising paucity of available explorations of these integration processes in clinical populations, and singularly in psychiatry. Using the illustration of our recent studies in alcohol-dependence, we tried to explain the fundamental and clinical usefulness of a thorough exploration of crossmodal integration abilities in clinical populations. What has been presented here should only be considered as a very modest first step towards a coherent and ambitious research program allowing a precise description of the crossmodal processing abilities in psychiatry. As everything remains to be done in this research field, the initial data described in the present paper might encourage researchers in psychiatry and neuroscience to develop this exploration of crossmodal processes in clinical populations.