Situating language in a minimal social context: how seeing a picture of the speaker’s face affects language comprehension
Loading...
Official URL
Full text at PDC
Publication date
2021
Advisors (or tutors)
Editors
Journal Title
Journal ISSN
Volume Title
Publisher
Oxford University Press
Citations
Exportar
Citation
Hernández-Gutiérrez, D., Muñoz, F., Sánchez-García, J., Sommer, W., Abdel Rahman, R., Casado, P., Jiménez-Ortega, L., Espuny, J., Fondevila, S., & Martín-Loeches, M. (2021). Situating language in a minimal social context: how seeing a picture of the speaker’s face affects language comprehension. Social cognitive and affective neuroscience, 16(5), 502-511. https://doi.org/10.1093/SCAN/NSAB009
Abstract
Natural use of language involves at least two individuals. Some studies have focused on the interaction between senders in communicative situations and how the knowledge about the speaker can bias language comprehension. However, the mere effect of a face as a social context on language processing remains unknown. In the present study, we used event-related potentials to investigate the semantic and morphosyntactic processing of speech in the presence of a photographic portrait of the speaker. In Experiment 1, we show that the N400, a component related to semantic comprehension, increased its amplitude when processed within this minimal social context compared to a scrambled face control condition. Hence, the semantic neural processing of speech is sensitive to the concomitant perception of a picture of the speaker's face, even if irrelevant to the content of the sentences. Moreover, a late posterior negativity effect was found to the presentation of the speaker's face compared to control stimuli. In contrast, in Experiment 2, we found that morphosyntactic processing, as reflected in left anterior negativity and P600 effects, is not notably affected by the presence of the speaker's portrait. Overall, the present findings suggest that the mere presence of the speaker's image seems to trigger a minimal communicative context, increasing processing resources for language comprehension at the semantic level.
Description
This research was supported by the Spanish Ministerio de Economia y Competitividad [grant codes PSI2013-43107-P and PSI2017-82357-P] (MINECO, Programa Estatal de Investigacion Cientifica y Tecnica de Excelencia, Subprograma Estatal de Generacion de Conocimiento y Proyectos de I+D).
Referencias bibliográficas:
• Adams, R.B., Rule, N.O., Franklin Jr, R.G., et al. (2010). Crosscultural reading the mind in the eyes: an fMRI investigation. Journal of Cognitive Neuroscience, 22(1), 97–108.
• Alday, P.M., Schlesewsky, M., Bornkessel-Schlesewsky, I. (2017). Electrophysiology reveals the neural dynamics of naturalistic auditory language processing: event-related potentials reflect continuous model updates. eNeuro, 4(6), ENEURO. 0311- 16.2017.
• Bindemann, M., Burton, A.M., Langton, S.R., Schweinberger, S.R., Doherty, M.J. (2007). The control of attention to faces. Journal of Vision, 7(10), 15–15.
• Bornkessel-Schlesewsky, I., Krauspenhaar, S., Schlesewsky, M. (2013). Yes, you can? A speaker’s potency to act upon his words orchestrates early neural responses to message-level meaning. PLoS One, 8(7), e69173.
• Bornkessel-Schlesewsky, I., Schlesewsky, M. (2019). Towards a neurobiologically plausible model of language-related, negative event-related potentials. Frontiers in Psychology, 10, 298.
• Brouwer, H., Fitz, H., Hoeks, J.C.J. (2012). Getting real about semantic illusions: rethinking the functional role of the P600 in language comprehension. Brain Research, 1446, 127–43.
• Brunet, E., Sarfati, Y., Hardy-Baylé, M.C., Decety, J. (2000). A PET investigation of the attribution of intentions with a nonverbal task. Neuroimage, 11, 157–66.
• Buchwald, A.B., Winters, S.J., Pisoni, D.B. (2009). Visual speech primes open-set recognition of spoken words. Language and Cognitive Processes, 24(4), 580–610.
• Camblin, C.C., Gordon, P.C., Swaab, T.Y. (2007). The interplay of discourse congruence and lexical association during sentence processing: evidence from ERPs and eye tracking. Journal of Memory and Language, 56(1), 103–28.
• Casado, P., Martín-Loeches, M., Leon, I., et al. (2018). When ´ syntax meets action: brain potential evidence of overlapping between language and motor sequencing. Cortex, 100, 40–51.
• Cecchini, M., Aceto, P., Altavilla, D., Palumbo, L., Lai, C. (2013). The role of the eyes in processing an intact face and its scrambled image: A dense array ERP and low-resolution electromagnetic tomography (sLORETA) study. Social Neuroscience, 8(4), 314–25.
• Coulson, S., King, J.W., Kutas, M. (1998). Expect the unexpected: event-related brain response to morphosyntactic violations. Language and Cognitive Processes, 13(1), 21–58.
• Craik, F.I., Lockhart, R.S. (1972). Levels of processing: a framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6), 671–84.
• Crosse, M.J., Butler, J.S., Lalor, E.C. (2015). Congruent visual speech enhances cortical entrainment to continuous auditory speech in noise-free conditions. The Journal of Neuroscience, 35(42), 14195–204.
• Dale, R., Fusaroli, R., Duran, N.D., Richardson, D.C. (2013). The self-organization of human interaction. Psychology of Learning and Motivation, 59, 43–95.
• Donchin, E. (1981). Surprise! Surprise? Psychophysiology, 18(5), 493–513.
• Friederici, A.D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6(2), 78–84.
• Friederici, A.D. (2011). The brain basis of language processing: from structure to function. Physiological Reviews, 91(4), 1357–92.
• Hagoort, P. (2017). The core and beyond in the language-ready brain. Neuroscience and Biobehavioral Reviews, 81, 194–204.
• Hanna, J.E., Brennan, S.E. (2007). Speakers’ eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language, 57(4), 596–615.
• Hauser, M.D., Chomsky, N., Fitch, W.T. (2002). The faculty of language: what is it, who has it, and how did it evolve? Science, 298(5598), 1569–79.
• Haxby, J.V., Hoffman, E.A., Gobbini, M.I. (2000). The distributed human neural system for face perception. Trends in Cognitive Sciences, 4(6), 223–33.
• Hernandez-Gutiérrez, D., Abdel Rahman, R.A., Martín-Loeches, ´ M., Muñoz, F., Schacht, A., Sommer, W. (2018). Does dynamic information about the speaker’s face contribute to semantic speech processing? ERP evidence. Cortex, 104, 12–25.
• Holler, J., Schubotz, L., Kelly, S., Hagoort, P., Schuetze, M., Özyurek, A. (2014). Social eye gaze modulates processing of ¨ speech and co-speech gesture. Cognition, 133(3), 692–7.
• Hutzler, F., Braun, M., Võ, M.L.H., et al. (2007). Welcome to the real world: validating fixation-related brain potentials for ecologically valid settings. Brain Research, 1172, 124–9.
• Jachmann, T.K., Drenhaus, H., Staudte, M., Crocker, M.W. (2019). Influence of speakers’ gaze on situated language comprehension: evidence from event-related potentials. Brain and Cognition, 135, 103571.
• Johansson, M., Mecklinger, A. (2003). The late posterior negativity in ERP studies of episodic memory: action monitoring and retrieval of attribute conjunctions. Biological Psychology, 64(1–2), 91–117.
• Jouravlev, O., Schwartz, R., Ayyash, D., Mineroff, Z., Gibson, E., Fedorenko, E. (2018). Tracking colisteners’ knowledge states during language comprehension. Psychological Science, 30(1), 3–19.
• Kampe, K.K., Frith, C.D., Frith, U. (2003). “Hey John”: signals conveying communicative intention toward the self activate brain regions associated with “mentalizing,” regardless of modality. Journal of Neuroscience, 23(12), 5258–63.
• Knoeferle, P. (2016). Characterising visual context effects. Visually Situated Language Comprehension, 93, 227.
• Knoeferle, P., Kreysa, H. (2012). Can speaker gaze modulate syntactic structuring and thematic role assignment during spoken sentence comprehension? Frontiers in Psychology, 3, 538.
• Kolk, H.H., Chwilla, D.J., Van Herten, M., Oor, P.J. (2003). Structure and limited capacity in verbal working memory: a study with event-related potentials. Brain and Language, 85(1), 1–36.
• Kutas, M., Federmeier, K.D. (2011). Thirty years and counting: finding meaning in the N400 component of the eventrelated brain potential (ERP). Annual Review of Psychology, 62, 621–47.
• Langton, S.R., Law, A.S., Burton, A.M., Schweinberger, S.R. (2008). Attention capture by faces. Cognition, 107(1), 330–42.
• Lau, E., Namyst, A., Fogel, A., Delgado, T. (2016). A direct comparison of N400 effects of predictability and incongruity in adjective-noun combination. Collabra Psychology, 2(1), 13.
• Leckey, M., Federmeier, K.D. (2020). The P3b and P600 (s): positive contributions to language comprehension. Psychophysiology, 57(7), e13351.
• Li, X., Hagoort, P., Yang, Y. (2008). Event-related potential evidence on the influence of accentuation in spoken discourse comprehension in Chinese. Journal of Cognitive Neuroscience, 20(5), 906–15.
• Martín-Loeches, M., Fernandez, A., Schacht, A., et al. (2012). ´ The influence of emotional words on sentence processing: electrophysiological and behavioral evidence. Neuropsychologia, 50(14), 3262–72.
• McCarthy, G., Nobre, A.C. (1993). Modulation of semantic processing by spatial selective attention. Electroencephalography and Clinical Neurophysiology/Evoked Potentials Section, 88(3), 210–9.
• Mecklinger, A., Rosburg, T., Johansson, M. (2016). Reconstructing the past: the late posterior negativity (LPN) in episodic memory studies. Neuroscience and Biobehavioral Reviews, 68, 621–38.
• Molinaro, N., Barber, H.A., Carreiras, M. (2011). Grammatical agreement processing in reading: ERP findings and future directions. Cortex, 47(8), 908–30.
• Munster, K., Knoeferle, P. (2018). Extending situated language ¨ comprehension (accounts) with speaker and comprehender characteristics: toward socially situated interpretation. Frontiers in Psychology, 8, 2267.
• Myllyneva, A., Hietanen, J.K. (2015). There is more to eye contact than meets the eye. Cognition, 134, 100–9.
• Payne, B.R., Lee, C.L., Federmeier, K.D. (2015). Revisiting the incremental effects of context on word processing: evidence from single-word event-related brain potentials. Psychophysiology, 52(11), 1456–69.
• Peelle, J.E., Sommers, M.S. (2015). Prediction and constraint in audiovisual speech perception. Cortex, 68, 169–81.
• Redcay, E. (2008). The superior temporal sulcus performs a common function for social and speech perception: implications for the emergence of autism. Neuroscience and Biobehavioral Reviews, 32(1), 123–42.
• Richardson, D.C., Dale, R. (2005). Looking to understand: the coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cognitive Science, 29(6), 1045–60.
• Roehm, D., Bornkessel-Schlesewsky, I., Rosler, F., Schlesewsky, ¨ M. (2007). To predict or not to predict: influences of task and strategy on the processing of semantic relations. Journal of Cognitive Neuroscience, 19(8), 1259–74.
• Rohr, L., Abdel Rahman, R. (2015). Affective responses to emotional words are boosted in communicative situations. Neuroimage, 109, 273–82.
• Rohr, L., Abdel Rahman, R. (2018). Loser! On the combined impact of emotional and person-descriptive word meanings in communicative situations. Psychophysiology, 55(7), e13067.
• Sassenhagen, J., Schlesewsky, M., Bornkessel-Schlesewsky, I. (2014). The P600-as-P3 hypothesis revisited: single-trial analyses reveal that the late EEG positivity following linguistically deviant material is reaction time aligned. Brain and Language, 137, 29–39.
• Schindler, S., Wegrzyn, M., Steppacher, I., Kissler, J. (2015). Perceived communicative context and emotional content amplify visual word processing in the fusiform gyrus. Journal of Neuroscience, 35(15), 6010–9.
• Schindler, S., Kissler, J. (2016). People matter: perceived sender identity modulates cerebral processing of socio-emotional language feedback. NeuroImage, 134, 160–9.
• Schindler, S., Kissler, J. (2017). Language-based social feedback processing with randomized ‘senders’: an ERP study. Social Neuroscience, 13(2), 202–13.
• Schone, B., K ¨ oster, M., Gruber, T. (2018). Coherence in ¨ general and personal semantic knowledge: functional differences of the posterior and centro-parietal N400 ERP component. Experimental Brain Research, 236(10), 2649–60.
• Senju, A., Hasegawa, T. (2005). Direct gaze captures visuospatial attention. Visual Cognition, 12(1), 127–44.
• Senju, A., Johnson, M.H. (2009). The eye contact effect: mechanisms and development. Trends in Cognitive Sciences, 13(3), 127–34.
• Sommer, K., Vita, S., De Pascalis, V. (2018). The late posterior negativity in episodic memory: a correlate of stimulus retrieval? Biological Psychology, 133, 44–53.
• Van Berkum, J.J., Van den Brink, D., Tesink, C.M., Kos, M., Hagoort, P. (2008). The neural integration of speaker and message. Journal of Cognitive Neuroscience, 20(4), 580–91.
• Van Petten, C. (2014). Selective attention, processing load, and semantics: insights from human electrophysiology. In Mangun G. R., editors. Cognitive Electrophysiology of Attention: Signals of the Mind, pp. 236–53. Amsterdam, Netherlands: Academic Press.
• Van Petten, C., Luka, B.J. (2012). Prediction during language comprehension: benefits, costs, and ERP components. International Journal of Psychophysiology, 83(2), 176–90
• van Wassenhove, V. (2013). Speech through ears and eyes: interfacing the senses with the supramodal brain. Frontiers in Psychology, 4, 388.
• Vertegaal, R., Slagter, R., Van der Veer, G., Nijholt, A. (2001). Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes. In: Proceedings of the SIGCHI conference on Human factors in computing systems, Seattle, WA. New York, NY; ACM Press. 301–8.
• Vissers, C.T.W., Chwilla, D.J., Kolk, H.H. (2007). The interplay of heuristics and parsing routines in sentence comprehension: evidence from ERPs and reaction times. Biological Psychology, 75(1), 8–18.
• Wang, L., Bastiaansen, M., Yang, Y., Hagoort, P. (2011). The influence of information structure on the depth of semantic processing: how focus and pitch accent determine the size of the N400 effect. Neuropsychologia, 49(5), 813–20. Wilding, E.L., Rugg, M.D. (1997). Event-related potentials and the recognition memory exclusion task. Neuropsychologia, 35(2), 119–28.