Event based Convolution Filters for feature detection

Main contributors Sadique Sheik John Chiasson



Two different spike based vision sensors were available to work with: 1) DVS (128x128) 2) ATIS(240x304). Both these sensors are based on the same working principle. Intensity chance at each pixel location (y,x) in the vision sensor generates a 'spike event'. The event is represented in AER, ie. it contains the information of the location x,y of the pixel and a binary magnitude of the change in intensity p, (1 or -1 for increase or decrease).

Feature Detectors

Two different sets of features were of interest. The first were edges of different orientations and the second were corners of different orientations. In order to detect these features a feed forward neural network architecture was implemented. The structure of the network for edge detection is shown in the below figure :

Network architecture of spike based edge detection network

Neural network and spike generation

We implemented a feed-forward neural network simulator for implementing feature detectors. The network connectivity is shown in the above figure and is as follows:

  • A spiking neuron was implemented which works as follows.
    • Each neuron has a membrane potential Vmem.
    • Each time this neuron receives a spike with a projection of weight w, Vmem is incremented by a value w.
    • At each time step if Vmem is greater than a threshold value Vth, Vmem is reset to a reset value Vreset (usually 0) and this neuron generates a spike.
    • At each time step in the simulation, the membrane potential is decremented to Vmem*decay where decay < 1.0 (this corresponds to the leak of a neuron).
  • A set of neurons corresponding to each pixel in the input is defined as a layer of neurons.
  • Each feature of interest (ie. vertical or horizontal edge) is assigned a layer of neurons.
  • Every pixel from the image sensor projects to a set of neighboring neurons (11x11 patch) in the feature detection layer(s).
  • The weights for these projections is determined by the filter used which corresponds to the feature of interest.

Orientation filter

A vertical Gabor filter (to detect vertical edges) is described by the following equation:


And is transformed to different angles through:



For more on the parameters go to  http://en.wikipedia.org/wiki/Gabor_filter

A set of 8 gabor filters tuned to different orientations is shown in the figure below.

Gabor filters tuned to 8 different orientations

- Key details -

  • Two identical orientation filters were initialized for each orientation, one for each polarity (+1 or -1) of retinal spikes. This was done in order to avoid misjudgment of alignment for thin objects.
  • The output spikes from these two filters was then combined and the information of polarity was ignored. ie. in terms of the output there is only one spiking source for each orientation.

In the demo 4 (and not 8) orientation filters were designed to detect edges at multiples of 45 from the vertical. There is a membrane potential for each Gabor filter. Thus by knowing which neuron (pixel) spikes from which filter one knows the direction of the edge.

Corner Detector

Combinations of edges leads to corners. We extend this idea to corner detectors. By combining input spikes from two orthogonal edge detectors corners can be detected. In terms of the underlying neural network architecture, this was realized by activating four corner filters that correspond to each orientation (each column in the figure below). The choice of these convolution filters was made in order to detect only corners and not intersections or T signs.

Corner filters for 16 different orientations.

The architecture of this feed forward filtering was identical to the architecture of input -> edge detectors. Technically the polarity of inputs was replaced by orientation of the edge and so its dimensions increased from 2 to N orientations.

- Key details -

  • In practice there are twice as many corners as there are orientations.
  • Unlike orientation filters all of whom receive the same input, corner filters only receive inputs from two of the orientation filters that correspond to their two orthogonal orientations.


The figure below shows the input and output activity in a short time window of the recorded data:

  • input spikes are plotted in gray-scale.
  • All orientation filter's outputs are superimposed and plotted in red.
  • All orientations of corner filter's outputs are superimposed and plotted in blue.

Snap shot of activity of the feature detection network and its inputs.

  • Analysis - 1) The edge detectors efficiently detected edges. They were more accurate when the two different polarities in the inputs were handled separately by two different filters. 2) The output of the edge detectors was sensitive to the input firing rates. A high activity at any given location was forcing the detectors to generate spikes despite the lack of true edges. 3) The effect of high activity could be clearly seen with corner detectors. More often than not, the corner detectors were detecting high activity locations (in terms of edge detectors) as opposed to corners.


- The edge detectors using Gabor based orientation filters were accurate. - The corner filters were not as accurate as edge detectors and probably require some kind of activity normalization or adaptation. - The overall simulation in python was computationally intensive and slow. For simulation purposes C or C++ would have been a better choice for the programming language. - A neuromorphic implementation of this system would have been ideal for real time application.