Cross-modal Integration and Value-Based Decision-Making
Key Research Question
- Does the acquisition of expected values through reward learning affect the cross-modal integration of two multi-modal stimuli that serve as decision cues?
- Does the strength or the efficiency of cross-modal integration affect the learning of the expected values of these multi-sensory cues?
Can the expected values of cues that we learn through repeated associations with a reward change the way how we perceptually integrate these stimuli? Does the efficiency with which we integrate stimuli from different modalities influence how we learn about rewards that are associated with them? These question lie at the heart of a new research project that is investigating the interaction of cross-modal integration and value-based decision-making in great detail.
Each experience that we encounter every day - e.g. sitting in an espresso bar at the harbor of a small Mediterranean fishing village on a sunny day - is comprised of different sensory inputs: images, sounds, smells, tastes, touches. Each of this sensory input is processed in a different primary brain region. Cross-modal integration is the awe-inspiring capability of the brain to bind information from these different sensory modalities together to form a coherent percept.
Cross-modal integration is governed by at least three principles: the spatial rule, which states that if information form different modalities originates from the same spatial location, cross-modal integration is stronger. Similarly, the temporal rule states, if stimuli from different modalities are present closely together in time, cross-modal integration is stronger and more efficient. Finally, the principle of inverse effectiveness, states that cross-modal integration will be stronger, if the uni-modal representations are weak or diffuse. We intend to created stimuli according to the first two rules and investigate, whether stronger or more efficient integration will benefit value-based learning and decision-making.
There are specific areas in the brain that are (among other things) dedicated to the integration of stimuli from different modalities (e.g. temporal parietal junction (TPJ) and superior temporal sulcus (STS)). The primary sensory cortices of different modalities are connected with these integration areas and feed the processed stimulus information to them. In these integration areas this stimulus information is combined. We think that depending for instance on the saliency or the expected value of these different stimuli, one modality will be weighted more heavily during integration and will come to dominate the resulting percept. We intend to manipulate the balance of different modalities during integration by associating the different stimuli with rewards or punishments. We expect that the modality with a higher expected value will come to dominate the cross-modal integration, whereas the modality with a negative expected value will be de-weighted during integration.
Influence of Value-based Decision-Making on Cross-Modal Integration
In this project we are changing cross-modal integration by endowing the constituting stimuli with different expected values through learning. We are using visual and auditory stimuli, which - after an initial measurement of cross-modal activity - will be associated with a reward, a punishment, or no outcome. During learning and afterwards, we will measure cross-modal BOLD activity again and determine, if the connectivity strength and the composition of cross-modal BOLD activity has changed. We intend to demonstrate this using model-based fMRI analysis (Gläscher & O'Doherty, 2010), connectivity analyses (non-linear Dynamic Causal Modeling, Stephan et al, 2008) and through Pattern Component Modeling (Diedrichsen et al., 2011), a novel form of multivariate pattern analysis for fMRI data.
Influence of Integration principles on Value-Based Decision-Making
Here, we investigate the reverse side of the interaction of learning and cross-modal integration. We are using the spatial and temporal rule to create stimuli that will be integrated more or less strongly and associate them with different rewards. We are testing whether reward associations between highly integrated cross-modal stimuli are learned more easily or quicker (estimated via a learning rate) than non-rewarded multi-sensory stimuli.