Filtering events

Event stream coming from a camera can contain noise, which is caused by analog electrical circuitry used to compare brightness on each pixel. The dv-processing library provides algorithms that filter noise efficiently on an event stream. Additionally, filtering can be used to subsample events in region of interest, filter by polarity, apply masks to events. This tutorial covers available filter implementations and how to use them efficiently.

Implementation of filters

The library provides two main types of filters - noise and subsampling. Subsampling filters include polarity filters, region of interest, mask filters. The library provides two algorithms for filtering noise that can be found under namespace dv::noise. Below is a class hierarchy diagram for available noise filters:

Available filters and their hierarchy in the library.

All filter have a common programming pattern:

• Events are added to the filter instance using overloaded dv::EventFilterBase::accept() method.

• Input events are filtered and returned by calling overloaded dv::EventFilterBase::generateEvents() method.

• Internally events are filtered by calling a filter retain() method on each event and discarding events if a false is returned.

Noise filtering

This chapter describes the available event noise filters in the library.

Background activity noise filter

dv::noise::BackgroundActivityNoiseFilter - events are filtered based on short-term local neighborhood activity. If an event is “supported” by another event registered at local pixel neighborhood, that event is not considered noise and is added to the output events.

Following sample code shows the usage of the dv::noise::BackgroundActivityNoiseFilter to filter noise:

C++Python

#include <dv-processing/data/generate.hpp>
#include <dv-processing/noise/background_activity_noise_filter.hpp>
#include <dv-processing/visualization/event_visualizer.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Hardcoded VGA resolution
    const cv::Size resolution(640, 480);

    // Initializing input events with uniformly distributed events which represent noise
    dv::EventStore events = dv::data::generate::uniformEventsWithinTimeRange(0, 10ms, resolution, 1000);

    // Adding additional data for drawing, this will give an idea whether the filter removes actual signal events
    events.add(dv::data::generate::dvLogoAsEvents(10000, resolution));

    // Initialize a background activity noise filter with 1-millisecond activity period
    dv::noise::BackgroundActivityNoiseFilter filter(resolution, 1ms);

    // Pass events to the filter
    filter.accept(events);

    // Call generate events to apply the noise filter
    const dv::EventStore filtered = filter.generateEvents();

    // Print out the reduction factor, which indicates the percentage of discarded events
    std::cout << "Filter reduced number of events by a factor of " << filter.getReductionFactor() << std::endl;

    // Use a visualizer instance to preview the events
    dv::visualization::EventVisualizer visualizer(resolution);

    // Generate preview images of data input and output
    const cv::Mat input  = visualizer.generateImage(events);
    const cv::Mat output = visualizer.generateImage(filtered);

    // Concatenate the images into a single image for preview
    cv::Mat preview;
    cv::hconcat(input, output, preview);

    // Display the input and output images
    cv::namedWindow("preview", cv::WINDOW_NORMAL);
    cv::imshow("preview", preview);
    cv::waitKey();

    return 0;
}

The sample code outputs such images:

Output of the sample use of background activity filter. Left is a preview of input events, right is a preview of filtered events.

Note

The image above that the filter reduces the amount of speckles on white area, but maintains the logo preview, those events are not filtered out.

Fast decay noise filter

dv::noise::FastDecayNoiseFilter - events are filtered based on lower resolution fast-decaying representation of events. Events contribute to a low-resolution accumulated image with a fast decay, which also represents local activity. Unlike the BackgroundActivityNoiseFilter, this filter uses decay instead of a hard time threshold, although the approach is very similar - an event needs to be supported by another event in a local pixel neighborhood. This filter has a lower memory footprint since the neighborhood is represented in a low resolution accumulated image.

Following sample code shows the usage of the dv::noise::FastDecayNoiseFilter to filter noise:

C++Python

#include <dv-processing/data/generate.hpp>
#include <dv-processing/noise/fast_decay_noise_filter.hpp>
#include <dv-processing/visualization/event_visualizer.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Hardcoded VGA resolution
    const cv::Size resolution(640, 480);

    // Initializing input events with uniformly distributed events which represent noise
    dv::EventStore events = dv::data::generate::uniformEventsWithinTimeRange(0, 10ms, resolution, 1000);

    // Adding additional data for drawing, this will give an idea whether the filter removes actual signal events
    events.add(dv::data::generate::dvLogoAsEvents(10000, resolution));

    // Initialize a background activity noise filter with 10-millisecond half life decay, resolution subdivision
    // factor of 4 and noise threshold of 1. Half life decay and noise threshold values controls the quality of
    // filtering, while subdivision factor is used for resolution downsizing for internal event representation.
    dv::noise::FastDecayNoiseFilter filter(resolution, 10ms, 4, 1.f);

    // Pass events to the filter
    filter.accept(events);

    // Call generate events to apply the noise filter
    const dv::EventStore filtered = filter.generateEvents();

    // Print out the reduction factor, which indicates the percentage of discarded events
    std::cout << "Filter reduced number of events by a factor of " << filter.getReductionFactor() << std::endl;

    // Use a visualizer instance to preview the events
    dv::visualization::EventVisualizer visualizer(resolution);

    // Generate preview images of data input and output
    const cv::Mat input  = visualizer.generateImage(events);
    const cv::Mat output = visualizer.generateImage(filtered);

    // Concatenate the images into a single image for preview
    cv::Mat preview;
    cv::hconcat(input, output, preview);

    // Display the input and output images
    cv::namedWindow("preview", cv::WINDOW_NORMAL);
    cv::imshow("preview", preview);
    cv::waitKey();

    return 0;
}

The sample code outputs such images:

Output of the sample use of fast decay noise filter. Left is a preview of input events, right is a preview of filtered events.

Note

The image above that the filter drastically the amount of speckles on white area, although the logo image is also affected and some true-signal events contributing to the logo are also filtered out.

Event subsampling

The same filtering approach is used to subsampling events based on their pixel location, polarity or other properties. This chapter describes the available event subsampling filters in the library.

Mask filter

dv::EventMaskFilter - filters events based on a pixel mask. Events are discarded in pixel locations where mask has zero values.

Following sample code shows the usage of the dv::EventMaskFilter to filter out selected regions of events:

C++Python

#include <dv-processing/core/filters.hpp>
#include <dv-processing/data/generate.hpp>
#include <dv-processing/visualization/event_visualizer.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Smaller resolution for previews
    const cv::Size resolution(200, 200);

    // Initializing input events with events that represent a logo
    const dv::EventStore events = dv::data::generate::dvLogoAsEvents(0, resolution);

    // Initialize a mask with all zero values
    cv::Mat mask(resolution, CV_8UC1, cv::Scalar(0));

    // Draw two rectangles to generate a similar to checkerboard mask pattern
    cv::rectangle(
        mask, cv::Point(0, 0), cv::Point(resolution.width / 2, resolution.height / 2), cv::Scalar(255), cv::FILLED);
    cv::rectangle(mask, cv::Point(resolution.width / 2, resolution.height / 2),
        cv::Point(resolution.width, resolution.height), cv::Scalar(255), cv::FILLED);

    // Initialize the mask filter with the generated mask
    dv::EventMaskFilter filter(mask);

    // Pass events to the filter
    filter.accept(events);

    // Call generate events to apply the filter
    const dv::EventStore filtered = filter.generateEvents();

    // Print out the reduction factor, which indicates the percentage of discarded events
    std::cout << "Filter reduced number of events by a factor of " << filter.getReductionFactor() << std::endl;

    // Use a visualizer instance to preview the events
    dv::visualization::EventVisualizer visualizer(resolution);

    // Generate preview images of data input and output
    const cv::Mat input  = visualizer.generateImage(events);
    const cv::Mat output = visualizer.generateImage(filtered);

    // Concatenate the images into a single image for preview
    cv::Mat preview, maskColored;
    cv::cvtColor(mask, maskColored, cv::COLOR_GRAY2BGR);
    cv::hconcat(std::vector<cv::Mat>({input, maskColored, output}), preview);

    // Display the input and output images
    cv::namedWindow("preview", cv::WINDOW_NORMAL);
    cv::imshow("preview", preview);
    cv::waitKey();

    return 0;
}

The sample code outputs such images:

Output of the sample use of event mask filter. Left is a preview of input events, middle is the mask used, and right is a preview of filtered events.

Refractory period filter

dv::RefractoryPeriodFilter - refractory period filter discards bursts of events at repeating pixel locations. Each event timestamp is compared against most recent event timestamp on the same pixel location, if the timestamp difference is less than the refractory period, the event is discarded.

Following sample code shows the usage of the dv::RefractoryPeriodFilter to filter out events that are within close time period on the same coordinate location:

C++Python

#include <dv-processing/core/filters.hpp>
#include <dv-processing/data/generate.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    const cv::Size resolution(200, 200);

    // Initializing 10000 events that are uniformly spaced in pixel area and time
    dv::EventStore events = dv::data::generate::uniformEventsWithinTimeRange(0, 10ms, resolution, 10000);

    // Initialize refractory period filter with 1-millisecond period
    dv::RefractoryPeriodFilter filter(resolution, 1ms);

    // Pass events to the filter
    filter.accept(events);

    // Call generate events to apply the filter
    const dv::EventStore filtered = filter.generateEvents();

    // Print out the number of events after filtering
    std::cout << "Filtered [" << filtered.size() << "] events out of [" << events.size() << "]" << std::endl;

    return 0;
}

Polarity filter

dv::EventPolarityFilter - filter events based on polarity.

Following sample code shows the usage of the dv::EventPolarityFilter to filter out events based on polarity:

C++Python

#include <dv-processing/core/filters.hpp>
#include <dv-processing/data/generate.hpp>
#include <dv-processing/visualization/event_visualizer.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    const cv::Size resolution(200, 200);

    // Initializing input events with events that represent a logo
    dv::EventStore events = dv::data::generate::dvLogoAsEvents(0, resolution);

    // Filter positive polarity events only
    dv::EventPolarityFilter filter(true);

    // Pass events to the filter
    filter.accept(events);

    // Call generate events to apply the filter
    const dv::EventStore filtered = filter.generateEvents();

    // Use a visualizer instance to preview the events
    dv::visualization::EventVisualizer visualizer(resolution);

    // Generate preview images of data input and output
    const cv::Mat input  = visualizer.generateImage(events);
    const cv::Mat output = visualizer.generateImage(filtered);

    // Concatenate the images into a single image for preview
    cv::Mat preview;
    cv::hconcat(input, output, preview);

    // Display the input and output images
    cv::namedWindow("preview", cv::WINDOW_NORMAL);
    cv::imshow("preview", preview);
    cv::waitKey();

    return 0;
}

The sample code outputs such images:

Output of the sample use of polarity filter. Left is a preview of input events and right is a preview of filtered events.

Note

The generated DV logo image comes from event representation, white area means no events are there, dark grey area are negative events and pixels are blue on coordinates where positive polarity events are provided. The filtered image only contains the letters “DV” since the background circle is represented by negative polarity events.

Event region filter

dv::EventRegionFilter - filter events based on given region of interest.

Following sample code shows the usage of the dv::EventRegionFilter to filter out specific area of events:

C++Python

#include <dv-processing/core/filters.hpp>
#include <dv-processing/data/generate.hpp>
#include <dv-processing/visualization/event_visualizer.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    const cv::Size resolution(200, 200);

    // Initializing input events with events that represent a logo
    dv::EventStore events = dv::data::generate::dvLogoAsEvents(0, resolution);

    // Initialize region filter using hardcoded coordinates
    dv::EventRegionFilter filter(cv::Rect(50, 50, 100, 100));

    // Pass events to the filter
    filter.accept(events);

    // Call generate events to apply the filter
    const dv::EventStore filtered = filter.generateEvents();

    // Use a visualizer instance to preview the events
    dv::visualization::EventVisualizer visualizer(resolution);

    // Generate preview images of data input and output
    const cv::Mat input  = visualizer.generateImage(events);
    const cv::Mat output = visualizer.generateImage(filtered);

    // Concatenate the images into a single image for preview
    cv::Mat preview;
    cv::hconcat(input, output, preview);

    // Display the input and output images
    cv::namedWindow("preview", cv::WINDOW_NORMAL);
    cv::imshow("preview", preview);
    cv::waitKey();

    return 0;
}

The sample code outputs such images:

Output of the sample use of region filter. Left is a preview of input events and right is a preview of filtered events.

Filter chain

Multiple filters can be combined into a single filter chain, which optimizes memory operations to increase the performance of applying multiple filters. This is achieved by using the dv::EventFilterChain class. Multiple filters can be added using dv::EventFilterChain::addFilter() method, it accepts filter wrapped in std::shared_pointer, the shared pointer is used to be able to modify the parameters of filters after they are added to the filter chain.

Following sample code shows the usage of the dv::EventFilterChain to apply multiple types of filters in a single chain:

C++Python

#include <dv-processing/core/filters.hpp>
#include <dv-processing/data/generate.hpp>
#include <dv-processing/noise/background_activity_noise_filter.hpp>
#include <dv-processing/visualization/event_visualizer.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    const cv::Size resolution(200, 200);

    // Initializing input events with events that represent a logo
    dv::EventStore events = dv::data::generate::dvLogoAsEvents(0, resolution);

    // Initialize event filter chain, it contains no filters
    dv::EventFilterChain filter;

    // Now let's add filters
    // First, add a region filter with hardcoded coordinates
    filter.addFilter(std::make_shared<dv::EventRegionFilter<>>(cv::Rect(50, 50, 100, 100)));

    // Second, add a positive polarity filter
    filter.addFilter(std::make_shared<dv::EventPolarityFilter<>>(true));

    // Third, add a background activity noise filter
    filter.addFilter(std::make_shared<dv::noise::BackgroundActivityNoiseFilter<>>(resolution));

    // Pass events to the filter
    filter.accept(events);

    // Call generate events to apply the filter chain, it will apply all three filters
    const dv::EventStore filtered = filter.generateEvents();

    // Use a visualizer instance to preview the events
    dv::visualization::EventVisualizer visualizer(resolution);

    // Generate preview images of data input and output
    const cv::Mat input  = visualizer.generateImage(events);
    const cv::Mat output = visualizer.generateImage(filtered);

    // Concatenate the images into a single image for preview
    cv::Mat preview;
    cv::hconcat(input, output, preview);

    // Display the input and output images
    cv::namedWindow("preview", cv::WINDOW_NORMAL);
    cv::imshow("preview", preview);
    cv::waitKey();

    return 0;
}

The sample code outputs such images:

Output of the sample use of multiple filters in a filter chain. Left is a preview of input events and right is a preview of filtered events.

Filtering performance

The provided event filters performance is measured using benchmarks, the benchmarks for filters can be found under directory benchmarks/noise in the project repository. These are sample benchmarking results, performance of filters is measured in throughput of mega-events per second. These measurements were capture on an AMD Ryzen 5 3600 6-Core processor.

Filter	Event count per iteration	Throughput, MegaEvents / second
FastDecayNoiseFilter	1000	82.3
FastDecayNoiseFilter	4096	82.9
FastDecayNoiseFilter	32768	80.1
FastDecayNoiseFilter	262144	67.5
FastDecayNoiseFilter	1000000	67.2
BackgroundActivityNoiseFilter	1000	141.5
BackgroundActivityNoiseFilter	4096	139.5
BackgroundActivityNoiseFilter	32768	105.4
BackgroundActivityNoiseFilter	262144	134.6
BackgroundActivityNoiseFilter	1000000	135.0
RefractoryPeriodFilter	1000	268.7
RefractoryPeriodFilter	4096	278.9
RefractoryPeriodFilter	32768	254.9
RefractoryPeriodFilter	262144	255.1
RefractoryPeriodFilter	1000000	167.5
PolarityFilter	1000	503.7
PolarityFilter	4096	345.0
PolarityFilter	32768	165.4
PolarityFilter	262144	157.7
PolarityFilter	1000000	156.6
RegionFilter	1000	446.9
RegionFilter	4096	371.7
RegionFilter	32768	166.8
RegionFilter	262144	153.6
RegionFilter	1000000	151.0
MaskFilter	1000	421.5
MaskFilter	4096	264.8
MaskFilter	32768	123.6
MaskFilter	262144	118.0
MaskFilter	1000000	117.1
ThreeFiltersNoChain	1000	89.3
ThreeFiltersNoChain	4096	88.0
ThreeFiltersNoChain	32768	76.2
ThreeFiltersNoChain	262144	75.0
ThreeFiltersNoChain	1000000	72.6
ThreeFiltersWithChain	1000	170.4
ThreeFiltersWithChain	4096	99.1
ThreeFiltersWithChain	32768	69.4
ThreeFiltersWithChain	262144	70.3
ThreeFiltersWithChain	1000000	70.1