Content

•  
   Introduction
•  
   Tasks
•  
   Thesis outline
•  
  Related work
•  
   Speech and gestures in natural communications
•  
   Typologies of hand gestures
•  
   Abstract deictics in spatial descriptions
•  
   Describe objects with iconic gestures/sketches
•  
  Relations between speech and co-verbal hand gestures
•  
   Semantic coordinations between co-verbal gestures and verbal content
•  
   Temporal alignment between gestures and speech
•  
   Multimodal human-computer interfaces
•  
   Natural language processing
•  
   Gesture recognition and interpretation
•  
   Multimodal fusion
•  
   Representation of multimodal content
•  
   Existing multimodal datasets
•  
   Summary
•  
  Multimodal corpora
•  
   Multimodal spatial scene description corpus
•  
   The scene description experiment
•  
   The spatial description experiment
•  
   Multimodal object description corpus
•  
   The Sketchy dataset
•  
   Augmenting sketches with verbal descriptions
•  
   Data statistics
•  
   The SAGA corpus
•  
   Summary
•  
  A system of understanding multimodal spatial descriptions
•  
   Modelling the interpretation of multimodal spatial descriptions
•  
   System overview
•  
   Utterance segmentation
•  
   Representing scene descriptions
•  
   Applying gestural information
•  
  Learning knowledge from prior experience
•  
   The take corpus
•  
   Learning mappings to logical forms
•  
   Learning perceptual groundings
•  
   Applying the represented knowledge
•  
  Experiment
•  
   A scene description corpus
•  
   Evaluation
•  
   Results
•  
   Summary
•  
  Towards real-time understanding of multimodal spatial descriptions
•  
   Real-time understanding of spatial scene descriptions
•  
   System overview
•  
   Gesture detection
•  
   Gesture interpretation
•  
   Utterance segmentation
•  
   Natural language understanding
•  
   Multimodal fusion & application
•  
  System evaluation
•  
   Gesture detector evaluation
•  
   Gesture interpretation evaluation
•  
   Utterance segmentation evaluation
•  
   Whole system evaluation
•  
   Incremental evaluation
•  
   Human understanding
•  
   Summary
•  
  Investigate symbolic and iconic modes in object descriptions
•  
   Draw and Tell: iconic and symbolic modes in object descriptions
•  
  Model the meaning of multimodal object descriptions
•  
   Grounding verbal descriptions
•  
   Comparing sketches with images
•  
   Fusion
•  
  Experiments
•  
   The image retrieving task
•  
   Metrics
•  
   Experiment 1: Mono-modal models
•  
   Experiment 2: multimodal models
•  
   Experiment 3: reduced sketch details
•  
   Discussion
•  
   Summary
•  
  Learning semantic categories of multimodal descriptions
•  
   Represent multimodal utterances with semantic concepts
•  
   Task formulation
•  
   Modelling the learning of multimodal semantics
•  
   Experiments
•  
   Language semantics
•  
   Gesture semantics
•  
   Multimodal semantics
•  
   Summary
•  
  Conclusion and future work
•  
   Overview of the dissertation
•  
   Future work
•  
   References