SpiNNaker Robotic Results

A longer title: Interfacing Real-Time Spiking Sensors and Actuators with the SpiNNaker Neuromorphic Architecture.

Participants: 'cameron patterson', Sergio Davies, 'chang woo shin', John Doyle, 'david lester'

About SpiNNaker

SpiNNaker is a Application Specific Integrated Circuit designed with the goal of modelling massively parallel artificial neural networks in real time. (see  http://intranet.cs.man.ac.uk/apt/projects/SpiNNaker/) Multiple SpiNNaker chips may be attached to one another to scale the neural network, and a fully configured system supports up to 65,536 chips. As SpiNNaker chip cores are general purpose commodity ARM processors they are easily programmable to support multiple neuron and synaptic models. Using one such neuronal model we have successfully modelled 1,000 Izhikevich neurons in real time on each processor core, and this scales to over 1bn neurons on a fully configured real-time SpiNNaker system.

About the Project

The project undertaken was to integrate a real-time spiking retinal sensor ( http://siliconretina.ini.uzh.ch/wiki/index.php) into a closed loop system on a mobile robot. The main goal of the project was to achieve computation only by means of spike events. For this purpose the SpiNNaker board has been used to simulate various spiking artificial neural networks to test the architecture in real-time. The solution functions using only with neural spikes (with the exception of the motor control interface, which is analogue).

We began firstly by interfacing a USB Silicon retina to the SpiNNaker board, taking input events from the retina from the virtual Serial port provided at up to 4Mb/s. The implementation using the micro-controller implemented by Jorg Conradt's  http://www.lsr.ei.tum.de/professors/detail/conradt group tops out at around 120k events/s. There are 2 issues this immediately brings to the fore regarding our SpiNNaker test chip environment: they are unable to to handle the native 128x128 resolution if we map 1 pixel to 1 discreet input neuron, and also the data rate is particularly high (120keps), and this is untested together with the Ethernet interface and it's packet overheads.

SpiNNaker is currently a development test system, it has not yet been deployed into a real-time environment, and has handled only software simulations (such as the doughnut hunter - see You Tube video below): It has been an excellent opportunity at this workshop to interface with physical real-time spiking sensors, and to actuators from the SpiNNaker chips and board.

Project Undertaken

The target task chosen is a well worn robotic one - line following. We decided to approach the main goal by solving an intermediate problem - controlling the robot using gestures made to the silicon retina. This would form a good basis in order to tackle the line following problem.

Firstly we attached the USB Silicon Retina to a PC, and built software to translate between the stream of data from the virtual Serial port into standard SpiNNaker Ethernet frames. The timing information is implicit in the arrival time of the events in this model. This was achieved using C++ and the GNU g++ compiler. Secondly before approaching the design of the neural network to run on the SpiNNaker system in itself, we looked at the motor control aspect of the problem, and developed functions to convert the rate and quantity of output spikes into a directional motor 'velocity' to be transmitted to the robot. The pseudo function is as follows, calculated as each output direction spike arrives:

new_rolling_average = max_speed*(time_since_last_spike) + rolling_average*(1-time_since_last_spike).

In addition we incorporated a decay function to ensure that the robot did not run away without proper stimulus:

Every ms: new_rolling_average = rolling_average-(max_speed*0.999).

Only significant interval changes are communicated to the robot in order to adjust its speed, in L/R F/B and rotational axes.

Building the Network for Gesture Control

A network was built that had discreet areas of stimulation - to the right would send right spikes, top -> forward spikes etc. This is summarised in the following diagram:

We quickly achieved success in this approach and we posted a video to You Tube that demonstrated this gesture control with Sergio Pellis waggling his finger in the relevant field of vision on the silicon retina and the robot moving in the appropriate direction. We used this control method due to absence of a tripod - we did consider using a human in front of a projector screen - waving their arms/hands in the relevant section of the field in order for this to control the robot too. We believe this would be successful, and beats Microsoft's Kinect gesture control XBOX360 hardware device to market ;-)

Line Following


As native data rate and event uniqueness may be problematic, a relevant method of reducing the data flow must therefore be employed, without losing the significant information required for line following. We identified and studied the following methods:

  1. Sub-sampling over time – for example dropping 9 out of 10 samples. This doesn't reduce the number of input neurons required, and would drop significant numbers of relevant spikes during slower movement making line detection more difficult.
  2. Sub-sampling categorisation over space – for example blocking/pixelating – either uniformly e.g. 16*16, or non-uniformly with better resolution at the centre of the image, per the biological eye. Using this method we don't drop spikes, just categorise them. (see 2*diagrams below)
  3. Sub-sampling over space and time (s of t) – for example only using pixel coordinates which are exact multiples of 4 and discarding the others. This however loses spiking information from unused neurons that may be used to detect the line coherently over and above any noise.
  4. Windowing – instead of using the full 'image field' use a subset which is suitable to enable decision making about which vector to take. (see diagram below)

Colour Enhanced example of the uniform 16x16 sub-sampling (2a):

Colour Enhanced example of the non-uniform 16x16 foveated sub-sampling method (2b):

Colour Enhanced image of the slotted/Windowed 64x8 data (4):

Sub-Sampling Approach

In our SpiNNaker line following system we initially tried non-uniform sub-sampling (Option 2b & diagram), but found that line edge information was found more broadly than just the better resolution centre. We then tried a combination of the uniform sub-sampling (2a) and window mechanisms (4 & diagram), where we used a full-width 'slot' to determine position of the line edges in order to try to maintain the line at the centre of view, whilst moving forward. We eventually settled on the uniform sub-sampling technique (diagram 2a) as it was simpler (and to be honest, the first we got to work reliably!). We were also limited to using a single chip on our multi-core boards due to tool problems, and a board fault that occurred in transit to Telluride.

If moving perfectly forward, the line would become invisible to the retina, as no changes would be detected, so we added an element of L/R jitter, as well as making use of the inherent mechanical vibration and movement in order to maintain visibility of the edges of the line.

For the line following we decided to follow a piece of dark blue tape, 3-5cm wide, on a plain background, using the edges made apparent to the silicon retina during motion to choose a path for the robot to follow. For this purpose a uniform 16x16 sensing grid was used without dropping spikes - and this fed into a neural network developed using PyNN (see robodrive.py attachment) in order to process the sub-sampled information, by connecting a 'hidden' layer of neurons between the input and the outputs:

  • The first 2 layers are two 16x16 matrices (one for positive polarity and another negative)
  • The third layer is a 4x4 "neural" sub-sampling of the input. Its sensitivity can be tuned by changing the parameter sampler_weight. The more frequent the cells in the sub-sampling neuron "receptive field" fire, the more likely the sub-sampling neuron will fire, inhibiting the rest of the sub-sampling neurons (WTA)
  • The sub-sampler winning units project to the output layer, attempting to keep the main stimulus in the centre of the visual field. The outputs are also in competition, in order to achieve better stability.

We can see this network structure in the following diagram:


As well as the gesture control video as seen above, results of the line-following solution software and neural network/connectivity implemented, can be found in the following You Tube video:

Future Work

We did find some problems in our implementation, we experienced congestion within our Ethernet spiking data, many events having been buffered and packed into SpiNNaker Ethernet frames. We had to limit the number of events per frame in order for them to be timely processed, and this caused some additional event loss. We would also like to explore the additional packing of data trade-off: If we don't send spikes in a timely manner the images appear much more grainy and noisy and the temporal resolution is lost.

We want to implement STDP in order to enable some plasticity and 'learning' rather than using data captures of the spikes to manually tweak the networks. Here's an example of an analysis of a retina run over a line - where the most active neurons are highlighted - polarity is ignored. You can see that my run was skewed to the right in this uniform 16x16 capture/analysis!:

This can be better visualised with a surface map as follows:

We have a number of recorded runs that we are conducting further analysis of - such as in the example above.

We also had problems in getting an untethered WLAN attached retina functioning - we saw a highly reduced data rate, which colloquially can be accounted for by:

  1. The number of access points in the workshop - possible clashes in the Ether.
  2. Using TCP as the transport for the retina data (reliable, can be delayed by retransmission) compounding the above. It would have been nice to have had a UDP option.

The Robot with the wirelessly attached retina mounted to it:

We would additionally like to look at deviating the line being followed, to see how the robot copes with this and recovers to follow the deviated line.


This was a project in it's test configuration, using a fraction of SpiNNaker's capacity (in reality ignoring the input neurons, and looking at the network sub-sampling units/connections we can see that the line following is being conducted using 16+3 neurons). This is a LIF implementation, so notionally we can expect to support up to 2000 LIF real-time neurons on each core. We are expecting that even a single test chip will be able to achieve much more than this simple/trivial example, and we hope to work with our partners and colleagues in order to progress further and extend the remit in the future.

The Telluride exercise was ultimately successful, and also helped us iron out many of the real issues that we will encounter as we move further from test - and into day-to-day utilisation of the neural simulator system. We met many people and we look forward to collaborating in the future. Please get in touch any time, we'd love to hear from you:

'cameron patterson': login to see email

Sergio Davies: login to see email

More About the Silicon Retina

The silicon retina used is a DVS128 class CMOS device which rather than sending a stream of rasterised frames, sends asynchronous events as each pixel changes detected intensity in either positive or negative directions. The sensitivity of these events can be adjusted and the sensor is able to generate sub millisecond responses to changes in individual pixel – currently a matrix of 128x128 pixels. As the individual pixels 'spike' they generate Address Event Representations on their outputs, and attached micro-controllers are able to attach microsecond resolution time-stamp information to the x and y coordinates, and polarity of the event. This representation is inspired by the arrival of spike information in the visual cortex from the eye in biology.

More About the SpiNNaker Chip

Each SpiNNaker chip consists of multiple ARM 968-E processing modules, together with tightly coupled instruction and data memories, timers, interrupt and a custom DMA controller. Each processor sub-system attaches to two asynchronous interconnects: The System NOC, providing connectivity to shared chip level peripherals and the Communications NOC to access the router and a communications path to the other processing cores on and off chip. The System NoC gives access to up to 1Gb of SDRAM, Fast Ethernet connection (for management and external stimulus/activation), watchdog timer, system RAM and system control functions. The Communications NOC is also asynchronous which provides access between processor cores for spike events, and also to send/receive inter-chip events where other neurons are located. A first iteration of the SpiNNaker chip delivered a 2 core processor and this proof-of-concept has been used in this implementation. Five PCBs have been produced each consisting of 4 SpiNNaker dual core chips with relevant inter-chip interconnects and a single Ethernet connection per board (you can see a test PCB pictured in the Schematic above). A larger scale production SpiNNaker chip is currently in production which has 18 cores on die, with up-scaled on-board NOCs and resources. This is the chip that is expected to scale up to a million core machine, as well as many other smaller scale systems.


SpiNNaker is a partnership between the University of Manchester, the University of Southampton, ARM Ltd, Silistix. We also wish to say thanks to: Tobi Delbruck and Jorg Conradt for their retinas and robots, all organisers and participants in the Telluride Neuromorphic Engineering and Cognition workshop 2010, Alex Rast and Francesco Galluppi & the rest of the crew in Manchester... and our mums.