Event-based Implementation of the Moravec corner detector

Leader: Michael Pfeiffer

The goal of this subproject was to detect corners in streams of DVS events, which constitute reliable features, e.g. for SLAM applications. Compared to other subprojects, the goal here was to use ideas that have been successfully used in computer vision, and port them to the event-based domain. The Moravec corner detector is one of the oldest tools (from early 1980s) to detect corners in images. It relies on the fact that the local neighborhood of a corner in an image is very different from the local neighbourhoods of the same size around neighbouring pixels. This is quantified by taking a patch of fixed size (e.g. 5x5 pixels) around each potential corner, and computing the sum of squared differences (SSD) with patches of the same size around pixels nearby. If the minimum SSD in the neighbourhood is large, then the point is called a corner. The detector has several limitations (e.g. anisotropy and the tendency to detect single pixels as corners), so more advanced detectors like Harris or SUSAN were developed in Computer Vision. However, we decided to use this detector, since it lends itself nicely to event-based implementation, and corner-detection in event-streams is still a largely unsolved problem.

Illustration of the event-based Moravec corner detector

Fig. 1: Illustration of the Moravec detector: A patch around an event (red square) is compared with all patches of the same size around events in the neighbouring region. If the difference between all patches is large, then the event is considered an event. If the difference is small in at least one direction, then the event is either part of an edge, or part of the interior. Noise pixels are eliminated by requiring a minimum number of events in the neighbourhood.

For events from DVS or ATIS sensors, we adopted a similar strategy: first, only events are considered as corners (this might not always be right, e.g. if an object with a T-corner is moved in the direction of an edge, but is a good approximation and works well in practice). We next took events within a small time window around the time of the event. First, we counted the number of both spatially and temporally adjacent events. Only if there are a minimum number of events nearby, the event will be considered as a corner. This prevents the detection of single events as corners. The minimum number of events is a parameter to be set by the user, and depends e.g. on the size of the receptive field and the noise level. Finally, for each event in the neighbourhood we compared whether their neighbourhood (in terms of spatially and temporally nearby events) is similar to that of the potential corner, using the sum of square distances, where events are counted as 1, and non-events as 0. If the minimum SSD for all neighbouring events is larger than the number of nearby events times a constant (another free parameter), then the event is considered a corner.

Results of the event-based Moravec detector

Fig. 2: Results of the event-based Moravec detector: The gray points correspond to events within the last 50 ms time window. The white cross indicates a detected corner (only one corner at a time is detected). More results are shown in the video available here: Corners_new.avi Download

In practice this seems to work well, and finds all types of corners (L, T, X and diagonal types). The greatest problem at the moment is that there are quite a few false detections, mostly due to noise spikes, but also because of the low spatial resolution of the sensor. Potentially they can be removed with more parameter tuning.

In summary we have shown a successful adaptation of a computer vision operator to the event-based vision domain, with promising results and additional capabilities, e.g. the suppression of single pixels as corners.