Spike-based Computation

Group Members:

  • Jorg Conradt, TU Munich
  • David Mascarenas, Los Alamos National Labs
  • John Harris, University of Florida
  • Hynek Hermansky, Johns Hopkins University
  • Jonathan Tapson, University of Cape Town
  • Malcolm Slaney, Microsoft Conversational Systems Lab and Stanford
  • Mounya Elhilali, Johns Hopkins University
  • Michele Rucci, Boston University
  • Ryad Benjamin Benosman, University Pierre and Marie Curie
  • Ralph Etienne-Cummings, JHU/ECE Department
  • Song Hui Chon, McGill University
  • Sergio Davies, The University of Manchester
  • Shih-Chii Liu, Institute of Neuroinformatics, UNI/ETHZ
  • Steve Kalik, Toyota Research Institue
  • Tara Julia Hamilton, The University of New South Wales
  • Timmer Horiuchi, University of Maryland
  • Tobi Delbruck, Institute of Neuroinformatics, UZH/ETH Zurich
  • Andre van Schaik, The University of Western Sydney

Organized by John Harris & Shih-Chii Liu

See wiki:2010/results/spike for results of this workgroup.


  1. Marco Rodrigues (Univ of Sevilla) - 3 weeks
  2. Hynek Hermansky (JHU) - 2nd and 3rd weeks
  3. Christopher Comer (Georgia Tech) - 1st week
  4. 'Odelia Schwartz' (New York) - 2nd week
  5. 'Jon Tapson' (already part of workshop)
  6. Terrence Sejnowski (already part of workshop)

Focus and Goals

GRAND CHALLENGE PROJECT: We plan to focus our efforts on a major project that has many self-contained sub projects. Since poker has a long history at Telluride, we are planning on building a machine that can play card games. Using Tobi's retina and Shih-Chii's cochlea as spike inputs, we plan both visual recognition of cards along with speech recognition of stated cards. Initially we are simplifying the recognition problems (making a special card deck, fixed card locations, single speaker, etc). We will start with an off-the-shelf cognitive engine to play strategically and remember what cards opponents have used. We are planning a simple speech synthesis engine for the machine to state which card to play from its hand. Interesting and self-contained subprojects will include: 1. playing card recognition from a spiking imager using simple template matching 2. playing card recognition from spiking cochlea output 3. Spike based learning, can the system learn to correlate between the auditory and visual representations of cards. Can it generalized? e.g. once it learns the 2 and 3 of diamonds and the 2 of clubs, can it recognize the 3 of clubs without explicitly training for it? 4. Attention aspects, how does the system keep track of what is going on, for example, whose turn is it? 5. Spike-based cognition, how can the system remember the cards that were played--how can it play strategically 6. Can it learn to play better as it plays more? Once we develop a working testbed, various software modules can be included in the system and tested.

HISTORY: An early focus of both the neuromorphic field and the Telluride workshop has been on the design of reliable neuromorphic front-ends starting with the retina and cochlea designs from Caltech in the late 1980ʼs. These early works led to generations of “smart” sensors that tried to compute image quantities such as edges, motion, center of mass, segmentation and depth for vision applications and even more rudimentary pitch or localization types of computations for auditory inputs. These early sensors were years away from useful applications but nonetheless were employed in many student projects, often at the Telluride workshop. With the advent of the AER representation, the field has evolved further to separate the high-resolution front-end sensing/processing from the more general spike-based “cortical” processing. Indeed, a key focus of last yearʼs Telluride workshop was on how to process the output spikes from Tobi Delbruckʼs mature time-derivative imager and from Shih-Chii Liuʼs spiking cochlea/auditory nerve.

PROBLEM: While we would never claim that the front-end sensor problem is solved, the focus is now on how to process the outputs of these spiking sensors. jAER offers one promising alternative for processing spike trains but is restricted to very non-cortical types of techniques using a traditional programming language running on a sequential CPU. In the brain, it is obvious to all that these spike trains are processed by other spiking neurons. The neuromorphic VLSI group at last yearʼs workshop tried to shoehorn all of these computations onto John Arthurʼs hippocampus chip leading to mixed results after varying amounts of pain and anguish.

SOLUTION: We believe that a more general spiking neuron programming models will lead to more promising solutions of these problems. A host of environments will be available at the workshop to develop spiking neuron computation including a real-time FPGA-based simulator being developed at Florida, off-line models such as the Python-based Brian simulator, real-time C-code running on a PC (PCs are now fast enough to run 100ʼs of simple integrate-and-fire neurons in real-time), and the Spinnaker system from Manchester. The various groups and participants will trade ideas and building blocks to improve their systems. Last year the key building block was synchronization through lateral inhibition but this idea needs to be better understood and more ideas are needed.


The typical project will process the output from a spiking hardware sensor (e.g. retina or cochlea) to perform a specific task. In doing these tasks, we would like to address more fundamental issues such as

  1. What are the advantages of spike representations over conventional image and speech processing representations?
  2. What is the role of spikes for carrying specific signal information: example temporal derivative information, how do we quantify the information?
  3. Role of spikes for computation
  4. Spike arithmetic
  5. Contrasting rate coding vs. timing codes
  6. How complex a neuron model is required for basic processing tasks?
  7. How best to incorporate adaptation and learning in spiking networks when solving a task?
  8. How can more cognitive tasks be programmed in a spiking neuron environment?

Projects are cross cutting in that nearly every area involves sensing and computation.


Auditory recognition

Members: John Harris, Jonathan Tapson, Shih-Chii Liu, Tara Julia Hamilton

This workgroup will look at recognition from audio signals using spikes, frames, or spiking networks.

Visual recognition

Members: Daniel Lofaro, John Harris, Jonathan Tapson, Ryad Benjamin Benosman, Shih-Chii Liu

This workgroup will look at recognition from visual signals using spikes, frames, or spiking networks.

Audio-visual fusion

Members: Jonathan Tapson, Shih-Chii Liu, Tara Julia Hamilton

This workgroup will look at recognition using fusion of information from audio and visual signals.

Cognitive module

Members: John Harris, Ryad Benjamin Benosman, Shih-Chii Liu, Timmer Horiuchi

This workgroup will develop a cognitive module that will have features necessary for strategic game playing. There are several different projects that relate to the construction of systems that can generate game playing behavior. The details of the different projects are within the subworkgroup website.

Fast spinning card recognition

Members: David Mascarenas, Daniel Lofaro, Ryad Benjamin Benosman, Tobi Delbruck

This workgroup will develop a fast numerical card recognizer while the card is spinning under its field of view.

Equipment brought

  1. 6x DVS128 USB Silicon retinas
  2. 2x 64 channel AER-EAR USB Silicon cochleas
  3. USB-AER boards
  4. Merger/splitter boards
  5. 2x Stacks of 4 USB-AER boards (8 USB-AER)
  6. 2x PAER Retinas
  7. 1x Merger/Splitter?
  8. All of Sevilla FPGA firmware: frame-based and timestamp-based monitor and sequencer, stochastic and delay mapper, etc.
  9. USB Cables, DC adaptors, CAVIAR-AER cables.

Brian Software

Dan Goodman (  http://thesamovar.net/neuroscience ) will give a tutorial on Brian Thursday at noon. The still-evolving slides can be downloaded at:


You need to install Python version 2.6 preferably (although 2.5 will also work), and the following packages for Python: numpy 1.3.0, scipy 0.7.1, matplotlib 0.99, brian 1.2.0. On Windows, one way to install all these except for Brian is to use Python(x,y):


This also comes with some nice editors (Eclipse for a classic IDE, or Spyder for a Matlab-like environment). Brian can be installed from its webpage:


A direct link to the download page is:


You can also get the downloaded files from Tara Julia Hamilton or, if the network behaves, from the Big Files section of the Wiki,


Spikes Exploring the Neural Code by F. Rieke, D. Warland, Rob de Ruyter van Steveninck, and W. Bialek
Neural Codes and Distributed Representations: Foundations of Neural Computation edited by L. Abbott and T. Sejnowski

UDP communication standard

The sensor data sent to the controlling matlab process from the silicon retina / jAER or Apron / SCAMP system about the value of the played card will have the following format.

The syntax for the visual data sent over UDP will be as follows:

[int sequence_number] [(long)time stamp ms since 1970] [source] [ [cardstats] [count0] [count1] [count2] ... [count13+] spadeCount heartCount diamondCount clubCount <- Counts

All values will be human readable as ascii strings, one line per Datagram.

Max Datagram size is 1500 bytes.

The meanings of these fields are as follows:

int sequence number::

An int that increments with each message; used to determine if packets are dropped.

long timestamp in ms since 1970::

Used to determine the timestamp of the data. Time since 1970 in ms is a posix standard.


The source of the data, either "retina" "apron" or "cochlea".


A keyword that signals that rest of line contains data.


cardstats is the keyword indicating that the following data is card value statistics.


The histogram count or probability as int value (but unnormalized) of a particular card value, starting with ace, two, .... king. count0 is the indication of less than 1 pip, count1 is the ace and count13+ is the king; count13 is the indication of more than 13, e.g. lots of blobs or features.

S D H C U::

The likelihood of the card being a spade, diamond, heart, or club (decreasing bridge value), followed by unknown count.

Port to use

Port 12435 will be used to talk to the matlab controller process.