Multimodal Semantic Representation (MMSR)

If you are interested in enrolling in this course, please send me an email by April 11, 2022 with the following information:

  • Name
  • Area of study (BSc, MSc)
  • Research interests and background in semantics
  • Why do you want to take this course?
  • Would you like a virtual option to be possible for the course?

Meeting time: Thursdays, 12:15-13:45, 1.05 (start date: 21.4.2022)

The demand for more sophisticated natural human-computer and human-robot interactions is rapidly increasing as users become more accustomed to conversation-like interactions with AI and NLP systems. Such interactions require not only the robust recognition and generation of expressions through multiple modalities (language, gesture, vision, action, etc.), but also the encoding of situated meaning.

When communications become multimodal, each modality in operation provides an orthogonal angle through which to probe the computational model of the other modalities, including the behaviors and communicative capabilities afforded by each. Multimodal interactions thus require a unified framework and control language through which systems interpret inputs and behaviors and generate informative outputs.

This seminar will look at the semantics of different modalities necessary for understanding meaning in situated dialogue and ask the question of how best to represent these modalities formally and for machine understanding. We will begin with language and move on to prosody and disfluencies in speech, gesture, body posture, gaze, and facial expression. In exploring each modality, we will look at how the expression and encoding of meaning changes compared to semantics for language. Part of this investigation will look at how meaning as conveyed by distinct modalities impacts memory and emotion in humans. We will consider various, recent formalisms for representing the interaction of multiple modalities and discuss their applicability to NLP tasks.

Prerequisites: Background in formal semantics is recommended.