When eyes beat lips

speaker gaze affects audiovisual integration in the McGurk illusion

verfasst von: Basil Wahn, Laura Schmitz, Alan Kingstone, Anne Böckler-Raettig
Abstract: Eye contact is a dynamic social signal that captures attention and plays a critical role in human communication. In particular, direct gaze often accompanies communicative acts in an ostensive function: a speaker directs her gaze towards the addressee to highlight the fact that this message is being intentionally communicated to her. The addressee, in turn, integrates the speaker’s auditory and visual speech signals (i.e., her vocal sounds and lip movements) into a unitary percept. It is an open question whether the speaker’s gaze affects how the addressee integrates the speaker’s multisensory speech signals. We investigated this question using the classic McGurk illusion, an illusory percept created by presenting mismatching auditory (vocal sounds) and visual information (speaker’s lip movements). Specifically, we manipulated whether the speaker (a) moved his eyelids up/down (i.e., open/closed his eyes) prior to speaking or did not show any eye motion, and (b) spoke with open or closed eyes. When the speaker’s eyes moved (i.e., opened or closed) before an utterance, and when the speaker spoke with closed eyes, the McGurk illusion was weakened (i.e., addressees reported significantly fewer illusory percepts). In line with previous research, this suggests that motion (opening or closing), as well as the closed state of the speaker’s eyes, captured addressees’ attention, thereby reducing the influence of the speaker’s lip movements on the addressees’ audiovisual integration process. Our findings reaffirm the power of speaker gaze to guide attention, showing that its dynamics can modulate low-level processes such as the integration of multisensory speech signals.
Organisationseinheit(en): Institut für Psychologie
Institut für Sportwissenschaft
Externe Organisation(en): University of British Columbia
Typ: Artikel
Journal: Psychological research
Band: 86
Seiten: 1930-1943
Anzahl der Seiten: 14
ISSN: 0340-0727
Publikationsdatum: 09.2022
Publikationsstatus: Veröffentlicht
Peer-reviewed: Ja
ASJC Scopus Sachgebiete: Experimentelle und kognitive Psychologie, Pädagogische und Entwicklungspsychologie, Geisteswissenschaftliche Fächer (sonstige)
Elektronische Version(en): https://doi.org/10.1007/s00426-021-01618-y (Zugang: Offen)

BibTeX

@article{3c889cb9ec934f83a4c4195323367b2e,
title = "When eyes beat lips: speaker gaze affects audiovisual integration in the McGurk illusion",
abstract = "Eye contact is a dynamic social signal that captures attention and plays a critical role in human communication. In particular, direct gaze often accompanies communicative acts in an ostensive function: a speaker directs her gaze towards the addressee to highlight the fact that this message is being intentionally communicated to her. The addressee, in turn, integrates the speaker{\textquoteright}s auditory and visual speech signals (i.e., her vocal sounds and lip movements) into a unitary percept. It is an open question whether the speaker{\textquoteright}s gaze affects how the addressee integrates the speaker{\textquoteright}s multisensory speech signals. We investigated this question using the classic McGurk illusion, an illusory percept created by presenting mismatching auditory (vocal sounds) and visual information (speaker{\textquoteright}s lip movements). Specifically, we manipulated whether the speaker (a) moved his eyelids up/down (i.e., open/closed his eyes) prior to speaking or did not show any eye motion, and (b) spoke with open or closed eyes. When the speaker{\textquoteright}s eyes moved (i.e., opened or closed) before an utterance, and when the speaker spoke with closed eyes, the McGurk illusion was weakened (i.e., addressees reported significantly fewer illusory percepts). In line with previous research, this suggests that motion (opening or closing), as well as the closed state of the speaker{\textquoteright}s eyes, captured addressees{\textquoteright} attention, thereby reducing the influence of the speaker{\textquoteright}s lip movements on the addressees{\textquoteright} audiovisual integration process. Our findings reaffirm the power of speaker gaze to guide attention, showing that its dynamics can modulate low-level processes such as the integration of multisensory speech signals.",
author = "Basil Wahn and Laura Schmitz and Alan Kingstone and Anne B{\"o}ckler-Raettig",
note = "Funding Information: Open Access funding enabled and organized by Projekt DEAL. This work was supported by a DFG-funded Emmy Noether grant [Grant Number BO4962/1-1] awarded to AB-R. BW and LS would like to thank Carolin and Pauline for their help in the piloting phase. ",
year = "2022",
month = sep,
doi = "10.1007/s00426-021-01618-y",
language = "English",
volume = "86",
pages = "1930--1943",
journal = "Psychological research",
issn = "0340-0727",
publisher = "Springer Verlag",
number = "6",
}