Integrating Perception, Cognition, and Action in Neuromorphic Hardware and Software

Members: Asha Gopinathan, Alexander Neckar, Bernabe Linares-Barranco, Bryan Tripp, Christian Brandli, Chris Eliasmith, Elisabetta Chicca, Christoph Maier, Jorg Conradt, Bert Shi, federico corradi, Gert Cauwenberghs, Giacomo Indiveri, Jasmine Berry, Jeffrey Pompe, Jonathan Tapson, Kevin Mazurek, Mehdi Khamassi, Magdalena Kogutowska, Mostafa Rahimi Azghadi, Mathis Richter, Matthew Runchey, Nai Ding, Nabil Imam, Michael Pfeiffer, Sadique Sheik, Sam Fok, Sergio Davies, Shih-Chii Liu, Samir Menon, Sudarshan Ramenahalli, Timmer Horiuchi, Thomas Murray, Tobi Delbruck, Terry Stewart, Theodore Yu, Yulia Sandamirskaya

See 'To do before arriving' to win FREE BEER!!!

Organizers: Kwabena Boahen, Chris Eliasmith

Faculty Dates

Kwaben Boahen Stanford 7/1/2012 7/21/2012
Chris Eliasmith Univ Waterloo 7/1/2012 7/11/2012
Bryan Tripp Univ Waterloo 7/11/2012 7/21/2012
Bruno Olshausen UC Berkeley 7/4/2012 7/9/2012
Thomas Serre Brown Univ 7/4/2012 7/8/2012
Bert Shi Hong Kong 7/13/2012 7/21/2012
Steve Furber Univ Manchester 7/17/2012 7/20/2012
Doug Tweed Univ Toronto 7/2/2012 7/4/2012
Jorg Conradt TU Munich 7/1/2012 7/21/2012
Sam Fok Stanford 7/10/2012 7/16/2012
Steve Scott Queens Univ 7/11/2012 7/13/2012


  1. Place Cells on Nengo on SpiNNaker on Omnibot [act12/results/Combined]
  2. Integrating SpiNNaker and Omnibots [act12/results/OmniSpiNN]
  3. Sensing and Escape [act12/results/Escape]
  4. Using Neurons to Build Asynchronous Circuits [act12/results/Asynchronous]
  5. Generating Grid Cells and Place Cells with Oscillators [act12/results/Oscillators?]
  6. Triggering Behavioural Change with Place Cells [Place Cells]
  7. Weightless Control for a Robot Arm [act12/results/Weightless?]
  8. Optimal Control on Neurogrid [act12/results/Optimal]
  9. Feed-Forward S1-C1 via the NEF [act12/results/S1C1]

Joint projects

Learning on SpiNNaker with NEF and Nengo

Lead: Sergio Davies

Interested: Chris Eliasmith, Terry Stewart

Implement a learning rule according to NEF theory on SpiNNaker. The outcome is a system which is able to learn transfer function in a supervised environment.
This project is joint between this group and the learning group.



  • We have two omni-directional robots with 4-dof arms and a camera, which have been interfaced with Nengo
  • We have one DENSO VE026 robotic arm (7DOF) as well as a pan/tilt turret with camera. These have been interfaced to Nengo
  • We have several additional robots being provided by Jorg Conradt that will be interfaced shortly.
  • A Neurogrid board with half-a-milllion neurons will be available---you can write Python scripts to map your Nengo networks onto it.
  • The SpiNNaker group will bring boards with several thousand neurons run in hardware that have already been interfaced with Nengo.
  • Tobi Delbruck's silicon retinas will be available, and have already been interfaced with Nengo.


Our main software tool will be Nengo,  http://nengo.ca/. Install the latest version by downloading  this file and unzipping it.

We recommend doing some  Tutorials. You can also watch  Videos.

To do before arriving

Three weeks of beautiful mountains will fly by, so to enjoy them as much as possible while there, we'd suggest trying any or all of:

  • Installing the nengo software and running some  Demos...
  • or better yet doing a  Tutorial or two.
  • At the very least, watch some of the  Videos, to get a sense of the environment.
  • Also peruse these slides to familiarize yourself with Neurogrid's  Hardware and  Software I  Software II
  • Think about which of the 'focus and goals' you find most interesting, and/or what part of an integrated project you'd like to tackle

I'll buy anyone a beer who can show me (on the first day) a Nengo model that they wrote that is not available anywhere on the web, and that does something vaguely interesting.

Focus and goals

The purpose of this workshop is to study the perception-cognition-action loop by building cognitive devices that interact with the real-world. Real-world interaction will be facilitated by silicon retinas for visual input, a neuromorphic computational fabric (NCF) for information processing, and robotic actuators for motor output.

NCFs are flexible hardware platforms that emulate spiking neural networks using arrays of silicon neurons in field programmable mixed-analog-digital chips. For example, Neurogrid, a hardware platform developed at Stanford, has a million silicon neurons.

For prototyping, testing, and generating network connection weights for hardware implementations, Nengo, an open-source software environment developed at Waterloo, will be employed. Nengo embodies the Neural Engineering Framework (NEF), a formal method for mapping control-theoretic algorithms on to the neural connections between highly heterogeneous populations of spiking neurons.

Through this workshop, students will be provided with hands-on experience with available hardware and software tools for constructing cognitive neuromorphic systems. This experience will provide familiarity with sensory (retina), motor (arm), and cognitive (working memory, decision making, action selection, etc.) aspects of neuromorphic engineering.

Possible Projects

  1. Sensory-motor co-ordination: The system must identify a visual target and track it (e.g., track a rolling ball with a camera and catch it). Demonstrates sensory recognition, motion processing, and motor co-ordination.
  2. Handwriting generation: The system will write the digits 0-9 using its actuators.
  3. Handwriting recognition: The system must identify the visual input as a member of a category. Drawn responses should not look like inputs. Demonstrates that input variability can be ignored when appropriate.
  4. Handwriting copying: The system must reproduce visual details of an input (i.e., drawn responses should look like the inputs). Demonstrates that low-level visual information (i.e., input variability) is accessible to the motor system.
  5. Path integration: The system will navigate arbitrarily, and be able to 'bee-line' to the starting point based on an internal representation
  6. Reinforcement learning: The system must learn environmental reward contingencies (e.g., related to motor coordination, catching artificial prey, etc.) Demonstrates reinforcement learning in the context of a cognitive model.(Software would be used for online training of weights, but input/output would be hardware.)
  7. Internal counting: The system must be able to identify two numbers, where the first indicates a starting point and the second a number of times to count. The response after internal counting will be the sum of the digits.
  8. Serial memory: The system will memorize a list and write it out, exhibiting human error patterns.
  9. Simple fluid intelligence: The system must build on its ability to categorize visual input to inductively infer structured, symbolic patterns. For instance, showing example patterns like [2 2 3 4; 3 4], [2 2 5 4; 5 4], and [2 2 9 4; 9 4] would result in [2 2 6 4] giving an output of [6 4]. It has recently been argued that such inference is a hallmark of human cognition.
  10. Mobile Robotics: Chasing a laser pointer dot. Build a robotic cat!
  11. Hardware Hacking: Work on improving the throughput of Neurogrid's daughterboard, allowing for larger NEF networks in hardware. Connection weights are interpreted probabilistically, and by taking advantage of layers of conditional probability and binary search, we should be able to achieve O(log(N)) spike routing.