Supernumerary Embodiment in Multimodal Augmented Reality Games

Abstract

Augmented reality (AR) is a new and upcoming medium that promises innovations in many application areas, ranging from entertainment games to serious teleoperations. Whereas virtual reality (VR) uses completely virtual environments, AR uses a combination of real and virtual to create one seamlessly mixed environment. This makes AR a technologically, perceptually, and cognitively complex area, which coincides with a lack of research on theoretical and experiential aspects of AR. With an eye towards the growing AR gaming market, this dissertation makes first steps in those less explored directions, namely in the area of multimodal game experiences that rely on the body for interaction. Specifically, I examine how supernumerary virtual avatars are experienced when presented alongside the player's real body.

This research had two main starting points. The first was to define multimodal AR, since nearly all previous AR research has solely focused on visual AR, and the corresponding definitions could not easily be translated to multimodality. I defined the components of multimodal AR to disambiguate the terms 'real' and 'virtual' and created a classification system for multimodal AR based on the combination of relevant modalities. The second starting point was to create an overview of player-avatar links in games from an interdisciplinary perspective. The presented works from Humanities, Natural and Social Sciences, showed a clear split in views: those that argue that the link must be related to embodiment, and those who argue it is more than embodiment. Focusing on embodiment views specifically, it became clear that the common reasoning did not encompass all forms of avatar presentation possible in AR, for example when the own player's body was present alongside (part of) the avatar's body. As a result, it was not clear whether embodiment of these avatars was even feasible.

Based on this foundation, a series of experiments was conducted to study the feasibility and function of this supernumerary embodiment in AR. In particular, I found that there were certain requirements for the experience of embodiment with respect to multimodality and realism that were related to the dissonance between real and virtual in AR, and that this experience may have been influenced by participants' capability to become immersed in media in general. Taking these findings into account, I presented a two-level processing model for this specific form of media-related embodiment that suggests that such media-related variables are involved in a processing level after top-down and bottom-up signal processing. In a final experiment, I studied the practical benefits of embodiment in AR by investigating the relation between embodiment and task performance, in a task involving a third virtual arm. The results showed that there was an indirect positive relation between the feeling of owning the third arm and performance that was mediated by the feeling of agency over the third arm.