Results from Spike-Based Computation Topic Area

The spike-based computation workgroup examined several issues about computing with spikes in neuromorphic architectures. Several subtopics were discussed in this workgroup.

  1. What are the advantages of spike representations over conventional image and speech processing representations?
  2. What is the role of spikes for carrying specific signal information: example temporal derivative information, how do we quantify the information?
  3. Role of spikes for computation
  4. Contrasting rate coding vs. timing codes
  5. How complex a neuron model is required for basic processing tasks?
  6. How best to incorporate adaptation and learning in spiking networks when solving a task?
  7. How can more cognitive tasks be programmed in a spiking neuron environment?

GRAND CHALLENGE PROJECT: The main project we executed in this workgroup was building a machine that can play card games. Building this machine is a challenge to see if we can build a system that uses a spike-based neural architecture for computation at different levels and that is able to learn strategies based on observations of its environments. We decided on a hearts card playing system which uses the spike outputs from a webcam and DVS128 retina and the AEREAR2 cochlea for the recognition of cards to be played.

In the project, we processed the output from a spiking hardware sensor (e.g. retina or cochlea) to perform a specific task. In doing these tasks, we would like to address more fundamental issues such as Initially we are simplifying the recognition problems (making a special card deck, fixed card locations, single speaker, etc). We will start with an off-the-shelf cognitive engine to play strategically and remember what cards opponents have used. We are planning a simple speech synthesis engine for the machine to state which card to play from its hand. There were some interesting questions pursued in this project:

  1. visual card recognition from a spiking imager using simple template matching
  2. audition card recognition from spiking cochlea output
  3. Spike based learning, can the system learn to correlate between the auditory and visual representations of cards. Can it generalized? e.g. once it learns the 2 and 3 of diamonds and the 2 of clubs, can it recognize the 3 of clubs without explicitly training for it?
  4. Attention aspects, how does the system keep track of what is going on, for example, whose turn is it?
  5. Spike-based cognition, how can the system remember the cards that were played--how can it play strategically
  6. Can it learn to play better as it plays more?

The complete description and results of the various modules developed in this project are at the following links.

  1. Auditory recognition
  2. Visual recognition
  3. Audio-visual fusion
  4. Cognitive module