Multimodal Reference: Data and Analysis

Refnet Summer School, University of Edinburgh, August 2014

Albert Gatt (University of Malta), Paul Piwek (Open University), Ielka van der Sluis (University of Groningen)

Course description

Gesture -- especially pointing -- is an important part of referential actions in communication. This course will focus on multimodal reference, from both a psycholinguistic and a computational perspective.

Lectures and slides

Note: Lecture slides will be available soon

Thursday, 28 August: Introductory session

  1. Introduction to gesture and reference: Semiotic classification, psycholinguistic models and computational models.
  2. Multimodal Corpora: Bielefeld studies (to Appear), van der Sluis and Krahmer (2007) and Beun and Cremers (1998)
  3. lecture notes for day 1

Friday, 29 August: The MREDI (Multimodal REference in DIalogue) Corpus

  1. MREDI Experimental Setup (van der Sluis et al 2008), data collection and annotation
  2. Using MREDI in Machine-learning experiments (Gatt and Paggio, 2014)
  3. lecture notes for day 2

Saturday, 30 August: Practical session and wrap-up

Background reading

Here are some readings which give background to the material covered in the course. Most of these are available online. The background reading isn't a requirement for participants to be able to follow the sessions; however, it will probably be useful for follow-up work.