Gestures Recognition

Main contributors: Luis Camunas, Michael Pfeiffer

Participants: Sadique Sheik, Kayode Sanni, Yezhou Yang, Garrick Orchard, Ryad Benjamin Benosman

This demo recognizes 6 different hand gestures recorded with a DVS retina:

1. Thumb up

2. Thumb down

3. Horizontal palm

4. Vertical palm

5. Victory

6. Pointing finger

The recording is processed using 6 event-based convolutional filters, where each filter has a kernel that matches one of the different gestures. Ideally, each filter should generate output events only when the desired gesture is present, but in practice it can happen that several filters respond to a single gesture (because within a gesture we can usually find several patterns which can match different filters). Therefore, we compare the number of output events generated by each filter in a short time window, so the one with a larger number of events corresponds to the recognized gesture.

The convolution kernels are shown in the next figure.

The following video shows the performance of this approach. On the left figure, the events generated by the DVS are presented in red, while the events generated by the different filters are presented in blue, black, cyan, magenta, green and yellow, respectively. The label inside the left figure indicates the recognized gesture dynamically, which corresponds to the filter with a larger response. The figure on the right represents the number of events generated by the different filters in each time window. (Video Download)

Future work: In this demo, the recognition is running in a matlab model of the event-based convolution filters, so the first objective for the near future would be to implement it in hardware using the 2D mesh of convolutional modules described in [1]. This way, it would be able to detect gestures on real time.


[1]Zamarreño-Ramos, C.; Linares-Barranco, A; Serrano-Gotarredona, T.; Linares-Barranco, B., "Multicasting Mesh AER: A Scalable Assembly Approach for Reconfigurable Neuromorphic Structured AER Systems. Application to ConvNets?," Biomedical Circuits and Systems, IEEE Transactions on , vol.7, no.1, pp.82,102, Feb. 2013. (PDF Download)