The Minimalist Grammar of Action

Sat, July 09, 2011 10:30-11:30am

Katerina Pastra
Cognitive Systems Research Institute, Greece

Language and action have been found to share a common neural basis and in particular a common “syntax”, an analogous hierarchical and compositional organization. While language structure analysis has led to the formulation of different grammatical formalisms and associated discriminative or generative computational models, the structure of action is still elusive and so are the related computational models. However, structuring action has important implications on action learning and generalisation, in both human cognition research and computation. In this talk, we present a biologically inspired generative grammar of action, which employs the structure-building operations and principles of Chomsky’s Minimalist Programme as a reference model. In this grammar, action terminals combine hierarchically into temporal sequences of actions of increasing complexity; the actions are bound with the involved tools and affected objects and are governed by certain goals. We show, how the tool-role and the affected-object role of an entity within an action drives the derivation of the action syntax in this grammar and controls recursion, merge and move, the latter being mechanisms that manifest themselves not only in human language, but in human action too. We will present how the minimalist grammar of action can be applied through both (a) a visual action/scene parser to build action syntax trees bottom up and (b) a language parser to build action syntax trees top down. The corresponding tools open new directions to visual scene analysis and embodied language processing respectively.