Decoding Multi-Modal Effects on Auditory Cognition


Organizers:: Edmund Lalor (Trinity College Dublin) Alain de Cheveign (ENS/CNRS, Paris) with help from Shihab Shamma (Univ. of Maryland) and Malcolm Slaney (Google)

Focus and Goals

The main focus of our group is hearing, but we'd also like to measure the influence of other sensory modalities, expectations and action. The real world involves many sources and objects within complex scenes that need to be parsed before they can be acted upon. We seek to develop reliable on-line decoding algorithms that extract from EEG signals the sensory-cortical responses corresponding to an auditory or visual source amongst many in a complex scene. We would like to determine interactions between sensory modalities and the effects of motor actions on the encoding of the correlated sensory responses.

The scientific approaches are highly interdisciplinary, spanning signal-analysis algorithms, models of cortical function, realtime EEG recording, and psychophysics. While the center of our Telluride work will revolve around EEG measurements, we will also be involved in psychophysical experiments that will allow a wider range of people to participate in the overall project.

The biggest challege to measuring brain activity with EEG is recording clean and sustained signals, factoring out irrelevant cortical activity, and analyzing and decoding brain states and sensory responses associated with ongoing visual and audio processing. This is especially important if one is to detect and interpret the relatively small response perturbations due to cognitive priming for example.

This year we like to focus on attention, decoding the direction of a subject's "auditory gaze", the nature of the sound being attended (e.g. speech vs music), the way the past changes our perception of the present (e.g. repetition in music), how feedback can maximize decoder output, and so on.

Potential projects are:

  • Audio and Video. EEG How do images and sound combine to make an EEG signal? Can we find an auditory signature from optical stimulation?
  • Decoding localization EEG Decoding location from EEG - can we determine the direction of a sound source, or the direction of a listener's attention? Left ear vs right ear, spatial position in one of four quadrants?
  • How do actions affect auditory responses? Can we separate muscle planning from auditory perception? Auditory-motor integration.
  • Is speech decoding special? Decoding the nature of a sound being attended to from EEG. Can we tell Speech vs. Poetry vs Rap vs. Music? Are they more alike or different?
  • Encoding of expectations in EEG. The first time we hear a musical phrase it sounds different from when we hear it repeated again (and again...). Can we see EEG correlates of how the music grows on us?
  • Decoding Algorithms pushing the limits of EEG decoding with new techniques such as DNN (Deep Neural Network) and other nonlinear approaches. Can we do a better job of decoding EEG signals and reconstructing or classifying the input sound? How robust are the algorithms with respect to subject and electrode variability?

Invited Participants

Name Institution Time Website
Alain de Cheveign ENS, Paris 26 June - 9 July ??  http://audition.ens.fr/adc/
Edmund Lalor Trinity College, Dublin 26 June - 9 July ??  http://www.mee.tcd.ie/neuraleng/People/Ed
Shihab Shamma ENS, Paris and Univ. of Maryland 26 June - 16 July  http://iec-lsp.ens.fr/spip.php?rubrique30&lang=fr
Malcolm Slaney Google 26 June - 16 July  http://www.slaney.org/malcolm/pubs.html
Jonathan Victor Cornell University 10 July - 16 July  http://www-users.med.cornell.edu/~jdvicto/jdvonweb.html
Lucas Parra City College New York 27 June - 2 July  http://bme.ccny.cuny.edu/people/faculty/lparra
'Sid Kouider' Ecole normale superieure 10 July - 16 July  http://www.lscp.net/persons/sidk/
'Behtash Babadi' University of Maryland first 2 weeks  http://ece.umd.edu/~behtash/
'Erv Hafter' Berkeley University 1st week  http://ear.berkeley.edu
'John Jeka' Temple University first 2 weeks  https://cph.temple.edu/kinesiology/faculty/john-jeka-0
Elizabeth Margulis University of Arkansas to be determined  http://www.elizabethmargulis.com