Benefits of multisensory presentation on perception, memory and navigation

Abstract

Navigation is the process of planning and following routes to travel from the current location to a target location. In comparison with real world navigation, we have considerable difficulty with navigation in virtual environments. An important cause is that less information is presented in a virtual environment than in the real world. For example, virtual environments are often limited to visual presentation only. Currently, little is known how adding auditory and tactile information to virtual environments affects navigation performance. The presentation of information about a single object or event to multiple sensory modalities can be advantageous to perception and memory. Because navigation largely relies on both perception and memory, multisensory benefits may also apply to navigation. Adding auditory and tactile information may therefore improve navigation in (visual) virtual environments. In the first chapter of this thesis, we introduced the Trisensory Orientation Model (TOM) to discuss the (potential) benefits for the presentation of congruent visual, auditory and tactile information on navigation. Congruent refers to the presentation of information in multiple sensory modalities at the same time, from the same place and with the same meaning. The model assumes a sequence of processes at the perceptual, memory and navigation level, and with it in mind we formulate several research questions and hypotheses which are investigated in subsequent chapters. In chapter 2, we employed electroencephalography (EEG) measurements to study early visuotactile integration processes of the visuotactile flash illusion. We found that visuotactile interactions modulate activity along the visual cortex in a similar vein as audiovisual interactions. These results indicate that the effects of multisensory presentation are likely to be rooted in early sensory interaction processes. In chapters 3 and 4, we studied the effects of congruent bi- and trisensory presentation in a temporal numerosity judgment task. We found that congruent multisensory presentation improved an observer’s ability to estimate numerosity, which complements earlier studies employing incongruent multisensory presentation. The known benefits of multisensory presentation on memory are limited to the effects of audiovisual presentation on encoding of object identity memory. In chapter 5, we presented visuotactile Morse codes and show that these effects also apply to visuotactile presentation, to retrieval and to object location memory. In chapter 6, we compare multisensory memory effects for congruent semantic and congruent non-semantic items. The obtained results indicate that multisensory congruency by itself is not sufficient for multisensory memory benefits; the congruent components need to carry actually semantic information. Finally, in chapter 7, we show that exploring a virtual environment with audiovisual landmarks results in higher navigation performance than exploration with either auditory or visual landmarks. In the final chapter, we discuss the above findings and their implications on TOM. Overall, our experiments show that multisensory presentation benefits perceptual, memory and navigation performance and provide further insight in the mechanisms underlying multisensory interactions. They imply that the presentation of multisensory information in virtual environments may help to overcome some of the difficulties associated with navigating virtual environments. Keywords Multisensory integration, Vision, Audition, Touch, Haptics, Memory, Perception, Navigation, Crossmodal interaction, EEG Main conclusions Overall, our experiments show that multisensory presentation benefits perceptual, memory and navigation performance. They imply that the presentation of multisensory information in virtual environments may help to overcome some of the current difficulties associated with navigating virtual environments.