Feature detection in events

The dv-processing library provides extensible interfaces and implementations for feature detection on event streams. Features are certain locations of interest in the data that can be detected with repeatability, such as corners. The detected features can be tracked over time in the event stream, so the library provides a reusable interface for feature detection algorithm implementations that can be used with a tracking algorithm.

Feature detectors

This section describes available feature detectors and their basic usage.

Feature detector using OpenCV

The most basic use of a feature detector is to use OpenCV feature detector implementation with the dv-processing feature detector wrapper on a dv::Frame. The feature detection wrapper base class dv::features::FeatureDetector was designed to wrap a feature detection algorithm and provide a few additional processing steps to handle image margins (ignore feature that are very close to edge of the image) and subsampling as a post-detection step (sampling only an optimal set of features).

Margins are set by a floating-point coefficient value, which expresses a relative margin size from the resolution of an input image, e.g. 0.05 would set the margins of 5% of the total width / height of the image.

Only a select number of features are useful for most applications. The dv::features::FeatureDetector simplifies feature subsampling by providing a post-processing step that sub-samples the output features. The sub-sampling can be performed by enabling one the following post-processing options:

• None: Do not perform any post-processing, all features from detection will be returned.

• TopN: Retrieve a given number of the highest scoring features.

• AdaptiveNMS: Apply the AdaptiveNMS algorithm to retrieve equally spaced features in pixel space dimensions. More information on the AdaptiveNMS here: original code and paper .

The following code sample shows how to detect the “good features to track” from OpenCV on an accumulated image from a live camera:

C++Python

#include <dv-processing/core/frame.hpp>
#include <dv-processing/features/feature_detector.hpp>
#include <dv-processing/io/camera_capture.hpp>

#include <opencv2/features2d.hpp>
#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Open any camera
    dv::io::CameraCapture capture;

    // Make sure it supports event stream output, throw an error otherwise
    if (!capture.isEventStreamAvailable()) {
        throw dv::exceptions::RuntimeError("Input camera does not provide an event stream.");
    }

    // Initialize an accumulator with camera resolution
    dv::Accumulator accumulator(*capture.getEventResolution());

    // Initialize a preview window
    cv::namedWindow("Preview", cv::WINDOW_NORMAL);

    // Let's detect 100 features
    const size_t numberOfFeatures = 100;

    // Create an image feature detector with given resolution. By default, it uses FAST feature detector
    // with AdaptiveNMS post-processing.
    auto detector = dv::features::ImageFeatureDetector(*capture.getEventResolution(), cv::GFTTDetector::create());

    // Initialize a slicer
    dv::EventStreamSlicer slicer;

    // Register a callback every 33 milliseconds
    slicer.doEveryTimeInterval(33ms, [&accumulator, &detector](const dv::EventStore &events) {
        // Pass events into the accumulator and generate a preview frame
        accumulator.accept(events);
        dv::Frame frame = accumulator.generateFrame();

        // Run the feature detection on the accumulated frame
        const auto features = detector.runDetection(frame, numberOfFeatures);

        // Create a colored preview image by converting from grayscale to BGR
        cv::Mat preview;
        cv::cvtColor(frame.image, preview, cv::COLOR_GRAY2BGR);

        // Draw detected features
        cv::drawKeypoints(preview, dv::data::fromTimedKeyPoints(features), preview);

        // Show the accumulated image
        cv::imshow("Preview", preview);
        cv::waitKey(2);
    });

    // Run the event processing while the camera is connected
    while (capture.isRunning()) {
        // Receive events, check if anything was received
        if (const auto events = capture.getNextEventBatch()) {
            // If so, pass the events into the slicer to handle them
            slicer.accept(*events);
        }
    }

    return 0;
}

Detected “good features to track” on an accumulated image.

Arc* Feature detector

The library provides an implementation of Arc* feature detector. Arc* performs corner detection on a per-event basis, so it uses dv::EventStore as an input. More details on this feature detection algorithm can be found in the original paper publication .

C++Python

#include <dv-processing/core/frame.hpp>
#include <dv-processing/features/arc_corner_detector.hpp>
#include <dv-processing/features/feature_detector.hpp>
#include <dv-processing/io/camera_capture.hpp>
#include <dv-processing/visualization/colors.hpp>

#include <opencv2/features2d.hpp>
#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Open any camera
    dv::io::CameraCapture capture;

    // Make sure it supports event stream output, throw an error otherwise
    if (!capture.isEventStreamAvailable()) {
        throw dv::exceptions::RuntimeError("Input camera does not provide an event stream.");
    }

    const auto resolution = *capture.getEventResolution();

    // Initialize an accumulator with camera resolution
    dv::Accumulator accumulator(resolution);

    // Initialize a preview window
    cv::namedWindow("Preview", cv::WINDOW_NORMAL);

    // Let's detect 100 features
    const size_t numberOfFeatures = 100;

    // Create an Arc* feature detector with given resolution. Corner range is set to 5 millisecond.
    auto detector = dv::features::FeatureDetector<dv::EventStore, dv::features::ArcCornerDetector<>>(
        resolution, std::make_shared<dv::features::ArcCornerDetector<>>(resolution, 5000, false));

    // Initialize a slicer
    dv::EventStreamSlicer slicer;

    // Register a callback every 33 milliseconds
    slicer.doEveryTimeInterval(33ms, [&accumulator, &detector](const dv::EventStore &events) {
        // Run the feature detection on the incoming events. Features are extracted on a per-event basis
        const auto features = detector.runDetection(events, numberOfFeatures);

        // Run accumulator for an image, it is going to be used only for preview
        accumulator.accept(events);
        const dv::Frame frame = accumulator.generateFrame();

        // Create a colored preview image by converting from grayscale to BGR
        cv::Mat preview;
        cv::cvtColor(frame.image, preview, cv::COLOR_GRAY2BGR);

        // Draw detected features
        cv::drawKeypoints(preview, dv::data::fromTimedKeyPoints(features), preview);

        // Show the accumulated image
        cv::imshow("Preview", preview);
        cv::waitKey(2);
    });

    // Run the event processing while the camera is connected
    while (capture.isRunning()) {
        // Receive events, check if anything was received
        if (const auto events = capture.getNextEventBatch()) {
            // If so, pass the events into the slicer to handle them
            slicer.accept(*events);
        }
    }

    return 0;
}

Note

The provided implementation is not suitable for running real-time with high event rates.

Detected Arc* features displayed on an accumulated image.

Event-based blob detector

The library provides an implementation of a simple event-based blob detector. The class makes use of cv::SimpleBlobDetector to detect blobs on an image. The image is generated using dv::EdgeMapAccumulator. A default cv::SimpleBlobDetector is provided with reasonable values to safely detect blobs and not noise in the accumulated image. It is also possible to specify the detector that should be used to find the blobs and some specific region of interests in the image plane where the blobs should be searched. In addition to this, eventually, it is also possible to specify the down sampling factor to be applied to the image before performing the detection and also specify any additional pre-processing step that should be applied before running the actual detection step. In summary, the detection steps are as following:

1. Compute accumulated image from events

2. Apply ROI and mask to the accumulated event image

3. Down sample image (optional)

4. Apply pre-process function (optional)

5. Detect blobs

6. Rescale blobs to original resolution (if down sampling was performed)

C++Python

#include <dv-processing/features/event_blob_detector.hpp>
#include <dv-processing/features/feature_detector.hpp>
#include <dv-processing/io/camera_capture.hpp>
#include <dv-processing/visualization/colors.hpp>

#include <opencv2/features2d.hpp>
#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Open any camera
    dv::io::CameraCapture capture;

    // Make sure it supports event stream output, throw an error otherwise
    if (!capture.isEventStreamAvailable()) {
        throw dv::exceptions::RuntimeError("Input camera does not provide an event stream.");
    }

    const auto resolution = *capture.getEventResolution();

    // Initialize an accumulator with camera resolution (for visualization only)
    dv::Accumulator accumulator(resolution);

    // Initialize a preview window
    cv::namedWindow("Preview", cv::WINDOW_NORMAL);

    // Let's detect up to 100 features
    const size_t numberOfFeatures = 100;

    // define preprocessing step (here we do nothing but show how it can be implemented)
    std::function<void(cv::Mat &)> preprocess = [](cv::Mat &image) {
    };

    // Define number of pyr-down to be applied to the image before detecting blobs. If pyramidLevel is set to zero, the
    // blobs will be detected on the original image resolution, if pyramid level is one, the image will be down sampled
    // by a factor of 2 (factor of 4 if pyramid level is 2 and so on...) before performing the detection. The blobs will
    // then be scaled back to the original resolution size.
    const int32_t pyramidLevel = 0;
    // Create an event-based blob detector
    const auto blobDetector = std::make_shared<dv::features::EventBlobDetector>(resolution, pyramidLevel, preprocess);
    auto detector           = dv::features::EventFeatureBlobDetector(resolution, blobDetector);

    // Initialize a slicer
    dv::EventStreamSlicer slicer;

    // Register a callback every 33 milliseconds
    slicer.doEveryTimeInterval(33ms, [&accumulator, &detector](const dv::EventStore &events) {
        // Run the feature detection on the incoming events.
        const auto features = detector.runDetection(events, numberOfFeatures);

        // Run accumulator for an image, it is going to be used only for preview
        accumulator.accept(events);
        const dv::Frame frame = accumulator.generateFrame();

        // Create a colored preview image by converting from grayscale to BGR
        cv::Mat preview;
        cv::cvtColor(frame.image, preview, cv::COLOR_GRAY2BGR);

        // Draw detected features
        cv::drawKeypoints(preview, dv::data::fromTimedKeyPoints(features), preview);

        // Show the accumulated image
        cv::imshow("Preview", preview);
        cv::waitKey(2);
    });

    // Run the event processing while the camera is connected
    while (capture.isRunning()) {
        // Receive events, check if anything was received
        if (const auto events = capture.getNextEventBatch()) {
            // If so, pass the events into the slicer to handle them
            slicer.accept(*events);
        }
    }

    return 0;
}