Human Cognition: Decoding Perceived, Attended, Imagined Acoustic Events and Human-Robot Interfaces

Members: Adriano Claro Monteiro, Alain de Cheveign, Anahita Mehta, Alejandro Pasciaroni, Jesus Armando Garcia Franco, Byron Galbraith, Christian Denk, Daniel Neil, Dimitra Emmanouilidou, Edmund Lalor, Bert Shi, Deniz Erdogmus, Greg Cohen, James OSullivan, Jonathan Tapson, Mehmet Ozdas, Amir Khosrowshahi, Lakshmi Krishnan, Malcolm Slaney, Mahmood Amiri, Michael Crosse, Jose L Pepe Contreras-Vidal, Qian Liu, Ryad Benjamin Benosman, Shihab Shamma, Sergio Davies, Shih-Chii Liu, Thusitha Chandrapala, Timmer Horiuchi, Will Constable

Organizers:: Shihab Shamma (Univ. of Maryland) Malcolm Slaney (Microsoft) Barbara Shinn-Cunningham (Boston University) Edmund Lalor (Trinity College, Dublin)

Final Report

Final Report


This topic area aims to implement projects to measure neuronal signals that reflect the perceptual, attentional and imaginative state of an individual brain. Specifically, we will seek to develop reliable on-line decoding algorithms that extract from the EEG signal the sensory cortical responses corresponding to an auditory source amongst many in a complex scene, or to an imagined music and speech signal. The goal is to understand how the perception and coding of such complex signals are represented and shaped by top-down cognitive functions (such as attention and recall). The basic scientific approaches needed are highly interdisciplinary, spanning development of signal-analysis algorithms and models of cortical function, to experimental EEG recordings during performance of challenging psychoacoustic tasks. While there is considerable research that touches upon issues addressed by this project, there are nevertheless several unique aspects to this work. For example, animal neurophysiological and imaging studies of auditory cortical activity cannot easily replicate the sophisticated behavioral tasks possible with humans, especially with speech and music. And since fMRI approaches in humans lack the temporal acuity necessary to track and extract auditory sensory responses, this leaves techniques such as MEG, EEG, and ECoG as the only scientifically feasible options. However, MEG requires expensive elaborate laboratory setups, and ECoG is obviously restricted to a few groups in the world with access to patients and interested surgical teams. Consequently, EEG is an accessible alternative for studying human auditory cognition. The biggest obstacle has been the perceived difficulty of recording clean and sustained signals that can be reliably associated with ongoing speech and music audio. This is especially important if one is to detect and interpret the relatively small response perturbations due to cognitive influences such as imagining sound or changing the attentional focus. We conducted pilot studies at the 2012 Telluride Workshop. We focused on demonstrating the feasibility of extracting the signals to which listeners attended in a complex mixture of sounds. This preliminary demonstration is described in more detail in


To summarize, we envision the following processing chain: Perception→ Attentional Filter → Imagine → Motor Planning The key two links in our proposed system are (1) to record and decode the neural signals from the human brain, interpret them correctly by identifying the imagined auditory targets and assessing the situation, and (2) to communicate the information to a robot so as to direct its actions based on the targets heard. There is much published science, technology, and engineering that already points to the feasibility of this scenario that we can harness to make rapid progress towards our goal.


Please use this section to summarize the papers we are reading. Please give a few sentences about why this paper is important for our group?

Nima's ECoG Reconstructions

 Nima's Nature Paper - This paper describes experiments to reconstruct the sound to which a subject is listening. This paper describes the reconstruction algorithm used in 2012 EEG attention decoding paper.

Malcolm's Summary of Past Telluride Efforts

 SLT Newsletter Every year's work is different, but the work we did in past years is described at this article written for the speech-recognition community.