Attention in the Machine - Final Report

Note: This is unpublished work in progress.

This year the attention project had two large sub-projects, both using EEG signals to study how auditory attention works. The first project aimed to decode the EEG signals and determine which of two auditory objects the subject is attending. The second project measured the EEG signals generated by surprising events, and events that change the subject's attention.

Our projects met with great success. Most importantly, we showed for the first time, that it is possible with EEG signals to correctly determine to which auditory stream a subject is attending. Previously, this has been demonstrated with cortical recordings (requiring surgery) and with MEG (requiring an expensive cryo-magnetic system). But neither one of these prior systems demonstrated the results in real time, as was done in Telluride this summer.

We were interested in both the scientific question---how is attention reflected in EEG signals---and the practical applications of attentional signals. On the practical side there are both brain-computer interface applications, and new applications based on augmenting human perception. Thus one might want a machine to pay attention to the signal that is being attended, pay extra attention to signals a human subject is not attending, or perform different kinds of processing for whichever signal the subject wishes. One commercially important application is automatically performing different kinds of processing, i.e. for speech or music, depending on the user's attention.

Over the three week workshop we had 8 faculty with experience in ECog (Electrocorticography), MEG (Magnetoencephalography), EEG (Electroencephalography) and analysis. These faculty led a team of approximately 14 students and post-docs to collect the EEG data, analyzing it, and building the real-time demo. We had two sub-projects looking at related attentional signals: decoding and markers.

The more audacious project endeavored to determine to which sound a sound is attending. A subject listened to two simultaneous speech streams. Each stream consisted of a male speaker reading a story. The subject was told to attend to one of the stories, but perform no other actions. This is a difficult task for a human subject, but all subjects indicated they could attend to either speaker.

We used a 32-electrode EEG array to measure the subject's responses. In training sessions we characterized the subject's response to speech signals in two different ways. In the first case, we estimated an inverse filter that mapped an EEG signal back into the speech signal that could have generated the received EEG signals (a system-identification solution). By doing this we could estimate the attended signal and then correlate this estimate with the possible speech sources. In the second case we projected a training speech signal and the received EEG signals into a joint subspace (using CCA) that provided the maximum correlation. By this method we could more directly compare the received signals to the speech sources and determine to which signal the subject is attending.

We got the best results using the inverse-signal approach. Using the entire electrode array, and all possible response times, we measured a low correlation (r=0.08) between the attended speech and the EEG response at each frame. But by combining multiple seconds of signals we can build up a large and significant prediction. Our task was to determine which of two speakers the subject is attending. In the best case, with dichotic signals presented to the subject (each speaker to one ear), we got upwards of 90% accuracy when looking at 60 seconds of speech. Looking at diotic, dichotic and HRTF (head-related transfer function) presentations we got an average of 80% accuracy.

The decoding experiment was turned into a real-time demonstration using an auditory camera, custom-modified EEG collection code, and custom decoding software. An auditory camera using 64 microphones arrayed around a sphere could generate an omni-directional signal for our subject, and individually steered auditory beams for localization. The omni-directional signal was fed to the subject, while we used two different (pre-arranged) directions for sensing the subject's attention. The subject was told to listen to one direction, and we used a 32-electrode active-sensor EEG system (from BrainVision) to collect the data.

Using the decoding techniques described above, we could look at sections of the EEG response and make a prediction as to which speaker the subject was attending. We used the audio-camera system to display a three-dimensional panorama of the room, overlay an image of where sound was coming from in the three-dimensions, and, finally, show with a colored box which the stream the subject was attending. We implemented the real-time system with custom modifications of the BrainVision pyCorder software that sent the received signals to a separate computer using UDP packets. Additional real-time Matlab software received the EEG signals, the two beams from the audio camera, and then generated the attention prediction.

The second big project looked at EEG markers of attention. We know that attention modulates (changes) the response to audio signals. In this project we looked at three specific signals that we thought could predict auditory attention: ASSR, MMN and RON. The Auditory Steady-State Response (ASSR) is a signal that is correlated with a repetitive signal and can be measured with EEG. We found that we could see differences in the ASSR, especially when the stimulus conditions were difficult, perhaps due to the extra attentional load. The mismatched-negativity (MMN) response is an EEG signal that is received when the subject perceives a change in a stream. The MMN is present whether the subject is attending to the stream or is not. We found a significant reduction in the MMN response due to attention. Finally, the reorientation negativity (RON) signal is a measure of the subject's response when they return their attention to the primary stream after an exogenous (distracting) attentional shift. We were not able to measure a response to this type of event.



We couldn't have done these projects without the assistance of  BrainVision. They loaned us an actiCHamp amplifier system with 32 active EEG electrodes. The active electrodes were great because it made it easier to setup our experiments. It was easy to modify their PyCorder software to send the EEG signals, via UDP, in real time to a Matlab process that implemented the decoding. The product details are at:  http://www.brainproducts.com/productdetails.php?id=42 Thank you!

Additional Details

We have more details on these projects on separate pages. Go to these three links for the final reports.