Audiovisual integration in depth: Modeling the effect of distance and stimulus effectiveness using the TWIN model

Abstract

Integrating information across our various sensory modalities results in striking behavioral benefits. This integration depends on a variety of factors, among which are the effectiveness of the stimuli and the relative timing between them. Both of these factors physically vary as a function of the distance between stimuli and the observer: intensity decreases as a function of distance for both auditory and visual stimuli, while the relative timing of energy arriving at the sensory organs differs due to differing transmission speeds. As a result, the depth at which multisensory stimuli are presented is likely to be an important factor in the gain that is derived from integrating them. Here, we use a computational approach – the Time-Window-of Integration (TWIN) framework – to examine differences in simultaneity judgments and reaction times to audiovisual stimuli presented at two depths. Using the TWIN model, we tested whether the observed behavior could be explained solely on the basis of differences in peripheral processing times, on the basis of changes in the temporal binding window (TBW), or by a combination of both factors. The results indicated that a model allowing for different TBWs for near and far space best accounts for the observed data in the majority of participants. However, the best overall model (regardless of the number of parameters) was a model containing both distance-dependent peripheral processing times and TBWs. Interestingly, TBWs were found not to expand from near to far space, but rather to get smaller. Taken together, the results indicate that distance is an additional factor in multisensory integration, above its impact on relative timing and intensity.