What the course is about

Recent years have seen an increased interest of formal linguistics into so-called non-verbal communication signals, in particular manual co-speech gestures. The ecological niche of gestures (and speech), however, is face-to-face interaction, or multimodal dialogue. Accordingly, there are many gestures which are related to the previous (inter-speaker) or to simultaneous (intra-speaker) dialogue moves. Hence, for an analysis of multimodal dialogue at least two kinds of formal tools are needed: a dialogue theory and a means to represent gestures in addition to speech. The course provides both: it introduces an existing dialogue framework and shows how to adapt it to multimodal interaction. The resulting multimodal analyses involve phenomena from various provenance such as laughter, iconic gesture, frowns, and head shaking.

In most instances, communication involves more than words. From cradle to final days, humans accompany verbal signals with various non-verbal social signals, including laughs, smiles, sighs, frowns, hand-and-arm movements, and a variety of other gestures. Understanding the import of such non-verbal social signals (NVSS), particularly laughter, has interested philosophers for millennia. More recently, NVSS have become an object of study for researchers including psychologists, biologists, neuroscientists, and linguists.

In the first part of the course we will introduce the notion(s) of multimodality (Lücking & Pfeiffer 2012) and basic semiotic gesture taxonomies (Lücking 2021), and present the KoS framework (Ginzburg 2012), which is formally underpinned by a Type Theory with Records (TTR; Cooper & Ginzburg 2015, Cooper forthc.). KoS synthesizes: speech act theory, Wittgensteinian language games, formal semantics, emotion representation from cognitive psychology and AI, and conversational analysis to yield a detailed theory of dialogical relevance from the micro-level (self-repairs, interjections; Ginzburg, Fernández & Schlangen 2014) to the macro-level (the structure of complete conversations; Wong 2018). This provides a theory of context that can underpin the analysis of a variety of dialogical phenomena such as non-sentential utterances, repair, and quotation (Fernández, 2006; Ginzburg, Fernández, and Schlangen, 2014; Purver, 2004; Ginzburg and Cooper, 2014), which are also part and parcel of multimodal interaction.

Non-verbal gestures have been observed to contribute on various semantic levels: reference (most notably pointing gestures), property exemplification (most notably iconic gestures), signalling attitudes (e.g. facial expressions), or performing complete speech acts (typically emblematic gestures). In the second part of the course, the background developed in the first part is applied to multimodal phenomena from the various levels. In particular, we cover co-speech iconic gestures (Lücking 2016) and laughter (Ginzburg, Mazzocconi and Tian 2020) in some detail, but also look at sighs (Cash and Ginzburg 2019), frowns and head shakes. With partiturs (Lücking and Ginzburg 2020), we introduce a generalized format for multimodal integration and discuss several representational end empirical problems that still have to be solved. In particular, we consider how to extend a theory of relevance developed to relate utterances that follow each other temporally (horizontal relevance) to utterances/multimodal signals that occur simultaneously (vertical relevance).

Participants acquire a basic background of notions and challenges of multimodal interaction. They get acquainted with a comprehensive, uniform formal framework for studying speech and gesture in dialogue. Analyses of selected phenomena introduces participants to recent advances in the field. The course is likewise of interest for students interested in dialogue and formal linguists interested in meaning beyond single sentences.

• Cash, Christopher and Jonathan Ginzburg (2019). "A Sigh of Positivity: an Annotation scheme for sighs in dialogue". In: Proceedings of SemDial 2019 (Londonlogue), pp. 35–43. url: http://semdial.org/anthology/Z19-Cash_semdial_0007.pdf.
• Cooper, Robin (forthc). From Perception to Communication: a Theory of Types for Action and Meaning. Oxford University Press.
• Cooper, Robin and Jonathan Ginzburg (2015). "Type theory with records for natural language semantics". In S. Lappin and C. Fox, eds., The Handbook of Contemporary Semantic Theory, chap. 12, pages 375–407. Oxford, UK: Wiley-Blackwell, 2nd edn. (Preprint)
• Fernández, Raquel (2006). "Non-Sentential Utterances in Dialogue: Classification, Resolution and Use". PhD thesis. King's College, London.
• Ginzburg, Jonathan and Robin Cooper (2014). "Quotation via Dialogical Interaction". In: Journal of Logic, Language, and Information 23.3, pp. 287–311.
• Ginzburg, Jonathan, Raquel Fernández, and David Schlangen (2014). "Disfluencies as Intra-Utterance Dialogue Moves". In: Semantics and Pragmatics 7.9, pp. 1–64
• Ginzburg, Jonathan, Chiara Mazzocconi, and Ye Tian (2020). "Laughter as Language". Glossa: A Journal of General Linguistics, 5(1), 104, DOI: 10.5334/gjgl.1152
• Lücking, Andy (2016). "Modeling Co-Verbal Gesture Perception in Type Theory with Records". In: Proceedings of the 2016 Federated Conference on Computer Science and Information Systems, pp. 383–392.
• Lücking, Andy (2021). "Gesture". In: Head Driven Phrase Structure Grammar: The handbook. Ed. by Stefan Müller, Anne Abeillé, Robert D. Borsley and Jean-Pierre Koenig. Empirically Oriented Theoretical Morphology and Syntax 9. Berlin: Language Science Press. Chap. 27, pp. 1201&ndsdh;1250. doi: 10.5281/zenodo.5543318. url: https://langsci-press.org/catalog/book/259
• Lücking, Andy and Jonathan Ginzburg (2020). "Towards the score of communication". In: Proceedings of the 24th Workshop on the Semantics and Pragmatics of Dialogue. SemDial/WatchDial. Brandeis University, Waltham (Watch City), MA
• Lücking, Andy and Thies Pfeiffer (2012). "Framing Multimodal Technical Communication. With Focal Points in Speech-Gesture-Integration and Gaze Recognition". In: Handbook of Technical Communication. Ed. by Alexander Mehler and Laurent Romary. In collab. with Dafydd Gibbon. Handbooks of Applied Linguistics 8. Berlin and Boston: De Gruyter Mouton. Chap. 18, pp. 591–644.
• Purver, Matthew (2004). "The Theory and Use of Clarification in Dialogue". PhD thesis. King’s College, London.
• Wong, Kwong-Cheong (2018). "Classifying Conversations". PhD thesis. Université Paris-Diderot (Paris 7).