Human Attention in the Machine

Members: Asha Gopinathan, Adrian KC Lee, Claire Chamers, Connie Cheung, Diana Sidtis, Edmund Lalor, Inyong Choi, James Wright, Jonathan Brumberg, Jongkil Park, Jeffrey Pompe, Jonathan Tapson, Lakshmi Krishnan, Malcolm Slaney, Mehdi Khamassi, Magdalena Kogutowska, Mathis Richter, Matthew Runchey, Nai Ding, Nils Peters, Jennifer Hasler, Ryad Benjamin Benosman, Sahar Akram, Shih-Chii Liu, Barbara Shinn-Cunningham, Siddharth Rajaram, Sudarshan Ramenahalli, Timmer Horiuchi, Thomas Murray, Tobi Delbruck, Troy Lau, Theodore Yu, Ying-Yee Kong, Yulia Sandamirskaya

Organizers:: Shihab Shamma, Barbara Shinn-Cunningham, Malcolm Slaney

Final Reports

Our final report is available here: Final Report


Confirmed Faculty: Adrian KC Lee (Univ. of Washington), Jon Brumberg (Boston University), Edmund Lalor (Trinity College Dublin), Troy Lau (BAE Systems), Nima Mesgarani (UCSF)

Faculty Dates

Shihab Shamma Univ. of Maryland 7/1/2012 7/19/2012
Barbara Shinn-Cunningham Boston University 7/5/2012 7/12/2012
Malcolm Slaney Microsoft and Stanford 7/1/2012 7/21/2012
Adrian KC Lee Univ. of Washington 7/2/2012 7/19/2012
Jon Brumberg Boston Univ. 7/4/2012 7/14/2012
Edmund Lalor Trinity College Dublin 6/30/2012 7/22/2012
Troy Lau BAE Systems 7/8/2012 7/11/2012
Nima Mesgarani UCSF 7/14/2012 7/21/2012
Inyong Choi Boston University 7/1/2012 7/8/2012

Pre Telluride Reading Assignments

We want to hit the ground running since we've only got 19 days to finish our project. Here are some papers that everybody should read before arriving in Telluride.

Software and Data

Notes about how to work with the data and code we are collecting for the big decoding experiment are at:  http://neuromorphs.net/nm/wiki/2012/att12/decoding


We will have an actiCHamp amplifier system with at least 32 active EEG electrodes. The product details are at:  http://www.brainproducts.com/productdetails.php?id=42

We appreciate the equipment support from Dr. Florian Strelzyk at Brain Vision LLC Toll Free: +1.877.EEG4MRI Numerical: +1.877.334.4674 www.brainvision.com

Big Idea

Our project wants to harnesses measurements of human attention to inform machines so they can better interact with the world around us. This would allow us to put human intelligence and attention into the machine in ways that have never been done before.

Technical Details (work in progress)

This project aims to create a human-robotic cooperative demo that integrates the intellectual abilities of the human brain with the technological prowess of machines. Specifically, we seek to harness the cortical neural signals of a humanbots that monitor the brain in real time as it makes sense of the environment. We will use measurement techniques such as EEG, eye gaze, and others, and decode the measured signals into the audio-visual correlates of the objects to which the human is focusing attention. Providing machines (i.e. robotics) with the ability to use this information without burdening the human will significantly enhance the robot’s ability to understand the environment, and to devise optimal strategies for working cooperatively with the human based on what is capturing their attention.

This project brings to Telluride researchers with experience in robotics, neuroscience of attention, auditory computations, computer vision, scene capture and analysis, and systems development. Participants will record neural activity in the brain related to percepts of attended objects/streams, with automatic auditory and visual scene and motion analysis. Researchers in many laboratories have established that EEG, MEG and MRI signals can be used to decode the state of a human subject's mind. We propose to use these signals in concert with automated means, such as signal analyzers and robotics, to perform tasks that neither agent can do on their own.

The key two links in our proposed human/robotic system are (1) to record and decode the neural signals from the human brain, interpret them correctly by identifying auditory/visual targets, enabling the robot to assess the situation, and (2) to communicate appropriate information and control signals the robot so as to direct its “attention” to the same targets on which the human is focused and allow the robot to act upon these targets appropriately. There is much published science, technology, and engineering that already points to the feasibility of this scenario; we will harness these results to make rapid progress towards our goal.

In the context of our human/robot interactions, when the human collaborator selects an object or a source to attend to, the robot can only act upon this information if it has a “perceptual” scene that is organized in roughly the same way that the human understands the scene; only then can the robot interpret the human-supplied data correctly. For example, perceptual features in the scene must be algorithmically extracted and grouped together into various “objects” even if the analysis is not very accurate or selective. In Telluride we shall (1) describe the overall theoretical framework of the importance of temporal coherence that we shall pursue to help gather and integrate sensory data from a variety of modalities for the robotic link. Also, we will (2) design experiments to decode the neural auditory and visual signals from human subjects to advance machine perception, analysis, and action.

The adaptive temporal coherence algorithm that we will use is focused on both auditory and visual input. Most importantly, it gives a natural role for attentional signals to select and bind disparate features from the scene and hence be integrated seamlessly into source segregation algorithms, as we shall elaborate.

Why Telluride

The goal of this work project is to use temporal coherence and the adaptive mind to help robotics-- and visa versa! The research proposed here is fundamentally inter-disciplinary, at the interface between cognitive sciences, neurophysiology, and computational neuroscience. It promotes a theoretical framework that spells out how perception might arise from interactions between cognitive influences and stimulus. This framework guides all modeling and experimental data interpretation, and has clear implications to sensory perception and multimodal interactions. The fundamental hypothesis of this research is that a rapid-adaptive process alters neuronal circuits and selectivity during perception of sound, and that this plasticity is enhanced by the existence of a temporally coherent structure in the acoustic stimuli.

In past years we have had successful experiments in Telluride to both collect EEG data, and do eye-tracking experiments. These are the key technologies for our experiments. With careful planning and staffing we can have these technologies ready to run when we arrive in Telluride. During the workshop we will learn what is possible with these recording technologies, and design experiments which show their effectiveness in helping guide machines to do what they do best.

This is a very broad area for new research. We hope to bring to Telluride a world-class team of researchers to attack this problem from many different directions. Our goal is to provide at least one demonstration, at the end of the three week workshop, that demonstrates better performance with cooperative human and robotic agents than either agent can do on their own. We live in a very complex world. Humans are good at somethings, including focussing attention, and machines are better at other tasks, such as recognizing an object.