Auditory Visual Fusion


Suyash Joshi, Shih-Chii Liu, Terry Stewart


The aim of this project was to connect the retina and the cochlea to NENGO and create a co-incidence detector. We get information from jAER using UDP packets, and we can feed this into the coincidence detector we've created in NENGO. The detector will light up if orientations from the retina and 48 channels of the choclea are all active.


A coincidence detector was created the layout of this is shown below, as can be seen it's a tree like structure where a thresholding unit is created and then cascaded with multiplier units.

The results of the coincidence detector are below

As can be seen it is still prone to spiking due to noise, and doesn't detect some of the spikes however, if multiple events saturate the channels at the time of the clap - by reducing the sensitivity of the neurons, the saturation can be more reliably detected. A much larger set of populations is required for making this system more robust to noise. All this parameter tweaking couldn't be completed during the workshop but is still currently worked on.