Multimodal Reference: Data and Analysis

Refnet Summer School, University of Edinburgh, August 2014

Albert Gatt (University of Malta), Paul Piwek (Open University), Ielka van der Sluis (University of Groningen)

Course description

Gesture -- especially pointing -- is an important part of referential actions in communication. This course will focus on multimodal reference, from both a psycholinguistic and a computational perspective.

Lectures and slides

Note: Lecture slides will be available soon

Thursday, 28 August: Introductory session

1. Introduction to gesture and reference: Semiotic classification, psycholinguistic models and computational models.
2. Multimodal Corpora: Bielefeld studies (to Appear), van der Sluis and Krahmer (2007) and Beun and Cremers (1998)
3. lecture notes for day 1

Friday, 29 August: The MREDI (Multimodal REference in DIalogue) Corpus

1. MREDI Experimental Setup (van der Sluis et al 2008), data collection and annotation
2. Using MREDI in Machine-learning experiments (Gatt and Paggio, 2014)
3. lecture notes for day 2

Saturday, 30 August: Practical session and wrap-up

• Practical task sheet (pdf format)
• Dialogue transcript (xlsx format)
• Map for this dialogue (pdf format)
• A short version of the annotation manual used for MREDI, in case you want to take a look at it. (pdf format)

Background reading

Here are some readings which give background to the material covered in the course. Most of these are available online. The background reading isn't a requirement for participants to be able to follow the sessions; however, it will probably be useful for follow-up work.

• De Ruiter, J. P. (2000). The production of gesture and speech. In McNeill, D. (Ed.), Language and Gesture (pp. 248-311). Cambridge: Cambridge University Press.
• McNeill, D. (2005). Gesture and Thought. University of Chicago Press, Chicago and London.
• van der Sluis, I. and E. Krahmer (2007). Generating Multimodal Referring Expressions. Discourse Processes 44(3): 145-174.
• Beun, R. and A. Cremers (1998). Object reference in a shared domain of conversation. Pragmatics & Cognition, 6(1/2): 121–152.
• Piwek, P., R.J. Beun and A. Cremers (2008). 'Proximal' and 'Distal' in language and cognition: evidence from deictic demonstratives in Dutch. Journal of Pragmatics, 40(4): 694-718.
• Piwek, P. (2009). Salience in the Generation of Multimodal Referring Acts.s In: D. Gatica-Perez, M. Johnston and R. Stiefelhagen (Eds.), Proceedings of the 2009 International Conference on Multimodal Interfaces (ICMI-MLMI), ACM Press, pp. 207-210, November 2-6, Cambridge, MA, USA.
• van der Sluis, I., P. Piwek, A. Gatt and A. Bangerter (2008). Towards a Balanced Corpus of Multimodal Referring Expressions in Dialogue. Proceedings of the Symposium on Multimodal Output Generation (MOG 2008) at AISB 2008 Convention, April 3 - 4, 2008, Aberdeen, Scotland.
• Gatt, A. and P. Paggio (2014). Learning when to point: A data-driven approach. Proceedings of the 25th International Conference on Computational Linguistics (COLING’14), Dublin, Ireland.