Picturing meaning: an ERP study on the integration of left or right-handed first-person perspective pictures into a sentence context

Abstract

Verbal and pictorial information are often processed together. Therefore, knowing how and when information from these modalities is integrated is important. In this ERP study we investigated integration of pictorial information into a sentence context. Right-handed participants heard sentences containing manual action verbs (e.g. “You are slicing the tomato”), while seeing a picture of a manual action. Pictures matched or mismatched the sentence content and the participants’ handedness (i.e. pictures showed a left or right-handed perspective). Results showed a larger N400-amplitude for content-mismatching than for content-matching sentence-picture pairs. The N400-amplitude was not larger when the picture mismatched the participants’ handedness. However, participants responded faster to right than to left-handed perspective pictures. This study suggests that with a sentence context, pictures are integrated with verbal information, but mental simulations either do not play a role in this process or this role might be too small to be visualised in the N400.