Event accumulation

Events are a sparse representation of brightness changes measured by the pixels on a camera sensor. Accumulation is a method applied to events to generate a frame representation of events. The dv-processing library provides a few highly optimized algorithm implementations to perform event accumulation and achieve different frame-like representation of the events.

Definitions

Certain definitions are used in this chapter and within accumulator API. Following is a list of specific definitions and their meaning within context of the accumulator:

• Potential - normalized pixel brightness value in some floating point range; it is an internal representation range for brightness that can be scaled into other brightness representations;

• Contribution - a numeric value that an event contributes to the brightness of pixel;

• Decay - pixel brightness correction applied over time when no events contribute to its brightness;

• Neutral potential - a default pixel brightness without any contribution;

• Minimum / maximum potential - limits for potential value representation.

Accumulator

The dv::Accumulator is a generalized implementation of a few accumulation algorithms that can be configured using class methods. Following chapter will describe the configuration options available for the accumulator. Final chapter of this section will provide a code sample that shows the use of all these configuration options for accumulation of events from a live camera.

Decay function and decay param

One of None, Linear, Exponential, Step. Defines the data degradation function that should be applied to the image. For each function, the Decay param setting assumes a different function:

Function	Enum value	Decay param function	Explanation
None	Decay::NONE	No function	Does not apply any decay
Linear	Decay::LINEAR	The slope a of the linear function, in intensity per microsecond	Assume intensity of a pixel is I0 at time 0, this function applies a linear decay of the form of I0 - (t * decayparam) or I0 + (t * decayparam)until the value hits the value specified in neutral potential
Exponential	Decay::EXPONENTIAL	The time constant tau of the exponential function in microseconds	Assume intensity of a pixel is I0 at time 0, this function applies an exponential decay of the form I0 * exp(-(t/decayparam)). The decay approaches a value of neutralPotential over time.
Step	Decay::STEP	No function	Set all pixel values to neutral potential after a frame is extracted.

The decay function can be set using dv::Accumulator::setDecayFunction() method.

Event Contribution

The contribution an event has onto the image. If an event arrives at a position x, y, the pixel value in the frame at x, y gets increased / decreased by the value of Event contribution, based on the events polarity.

Except:

• The resulting pixel value would be higher than Max potential, the value gets set to Max potential instead

• The resulting pixel value would be lower than Min potential, the value gets set to Min potential instead

• The event polarity is negative, and Ignore polarity is enabled, then the event is counted positively

Event contribution can be set using the dv::Accumulator::setEventContribution() method.

Min potential / Max potential

Sets the minimum and maximum values a pixel can achieve. If the value of the pixel would reach higher or lower, it is capped at these values. These values are also used for normalization at the output. The frame the module generates is an unsigned 8-bit grayscale image, normalized between Min potential and Max potential. A pixel with the value Min potential corresponds to a pixel with the value 0 in the output frame. A pixel with the value Max potential corresponds to a pixel with the value 255 in the output frame.

Min potential and Max potential can be set using dv::Accumulator::setMinPotential() and dv::Accumulator::setMaxPotential() methods.

Neutral potential

This setting has different effects depending on the decay function:

Function	Neutral potential function
None	No function.
Linear	Pixel brightness value decays linearly into the neutral potential value.
Exponential	No function.
Step	Each pixel value is set to neutral potential after generating a frame.

Neutral potential can be set using dv::Accumulator::setNeutralPotential().

Ignore polarity

If this value is set, all events act as if they had positive polarity. In this case, Event contribution is always taken positively. This can be used to generate edge images instead of an actual image reconstruction. This can be done by setting Neutral potential and Min potential to zeros.

This feature can be enabled using the dv::Accumulator::setIgnorePolarity() method.

Synchronous Decay

If this value is set, decay happens in continuous time for all pixels. In every frame, each pixel will be eagerly decayed to the time the image gets generated. If this value is not set, decay at the individual pixel only happens when the pixel receives an event. Decay is lazily evaluated at the pixel.

Note

Both decay regimes yield the same overall decay over time, just the time at which it is applied changes. This parameter does not have an effect for Step decay. Step decay is always synchronous at generation time.

Synchronous decay can be enabled using the dv::Accumulator::setSynchronousDecay() method.

Accumulating event from a camera

The following sample code show how to use dv::Accumulator together with dv::EventStreamSlicer and dv::io::CameraCapture to implement a pipeline that generates continuous stream of accumulated frames:

C++Python

#include <dv-processing/core/frame.hpp>
#include <dv-processing/io/camera_capture.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Open any camera
    dv::io::CameraCapture capture;

    // Make sure it supports event stream output, throw an error otherwise
    if (!capture.isEventStreamAvailable()) {
        throw dv::exceptions::RuntimeError("Input camera does not provide an event stream.");
    }

    // Initialize an accumulator with some resolution
    dv::Accumulator accumulator(*capture.getEventResolution());

    // Apply configuration, these values can be modified to taste
    accumulator.setMinPotential(0.f);
    accumulator.setMaxPotential(1.f);
    accumulator.setNeutralPotential(0.5f);
    accumulator.setEventContribution(0.15f);
    accumulator.setDecayFunction(dv::Accumulator::Decay::EXPONENTIAL);
    accumulator.setDecayParam(1e+6);
    accumulator.setIgnorePolarity(false);
    accumulator.setSynchronousDecay(false);

    // Initialize a preview window
    cv::namedWindow("Preview", cv::WINDOW_NORMAL);

    // Initialize a slicer
    dv::EventStreamSlicer slicer;

    // Register a callback every 33 milliseconds
    slicer.doEveryTimeInterval(33ms, [&accumulator](const dv::EventStore &events) {
        // Pass events into the accumulator and generate a preview frame
        accumulator.accept(events);
        dv::Frame frame = accumulator.generateFrame();

        // Show the accumulated image
        cv::imshow("Preview", frame.image);
        cv::waitKey(2);
    });

    // Run the event processing while the camera is connected
    while (capture.isRunning()) {
        // Receive events, check if anything was received
        if (const auto events = capture.getNextEventBatch()) {
            // If so, pass the events into the slicer to handle them
            slicer.accept(*events);
        }
    }

    return 0;
}

Frame generated using dv::Accumulator class.

Edge accumulation

The dv-processing library provides a highly-optimized variant of accumulator for generating edge maps - dv::EdgeMapAccumulator. It was specifically optimizes for speed of execution, so it has only a minimal set of settings and supported features compared to dv::Accumulator.

Below is a table providing available parameters for the dv::EdgeMapAccumulator:

Parameter	Default value	Accepted values	Comment
Contribution	0.25	[0.0; 1.0]	Contribution potential for a single event.
Ignore polarity	true	boolean	All events are considered positive if enabled.
Neutral potential	0.0	[0.0; 1.0]	Neutral potential is the default pixel value when decay is disabled and the value that pixels decay into when decay is enabled.
Decay param	1.0	[0.0; 1.0]	This value defines how fast pixel values decay to neutral value. The bigger the value the faster the pixel value will reach neutral value. Decay is applied before each frame generation. The range for decay value is [0.0; 1.0], where 0.0 will not apply any decay and 1.0 will apply maximum decay value resetting a pixel to neutral potential at each generation (default behavior).

Following sample show the use of dv::EdgeMapAccumulator with dv::EventStreamSlicer and dv::io::CameraCapture to generate stream of edge images:

C++Python

#include <dv-processing/core/frame.hpp>
#include <dv-processing/io/camera_capture.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Open any camera
    dv::io::CameraCapture capture;

    // Make sure it supports event stream output, throw an error otherwise
    if (!capture.isEventStreamAvailable()) {
        throw dv::exceptions::RuntimeError("Input camera does not provide an event stream.");
    }

    // Initialize an accumulator with some resolution
    dv::EdgeMapAccumulator accumulator(*capture.getEventResolution());

    // Apply configuration, these values can be modified to taste
    accumulator.setNeutralPotential(0.0f);
    accumulator.setEventContribution(0.25f);
    accumulator.setDecay(1.0);
    accumulator.setIgnorePolarity(true);

    // Initialize a preview window
    cv::namedWindow("Preview", cv::WINDOW_NORMAL);

    // Initialize a slicer
    dv::EventStreamSlicer slicer;

    // Register a callback every 33 milliseconds
    slicer.doEveryTimeInterval(33ms, [&accumulator](const dv::EventStore &events) {
        // Pass events into the accumulator and generate a preview frame
        accumulator.accept(events);
        dv::Frame frame = accumulator.generateFrame();

        // Show the accumulated image
        cv::imshow("Preview", frame.image);
        cv::waitKey(2);
    });

    // Run the event processing while the camera is connected
    while (capture.isRunning()) {
        // Receive events, check if anything was received
        if (const auto events = capture.getNextEventBatch()) {
            // If so, pass the events into the slicer to handle them
            slicer.accept(*events);
        }
    }

    return 0;
}

Frame generated using dv::EdgeMapAccumulator class.

Event visualization

Accumulators, described in previous chapters are useful when image or edge representation of events is needed, and they are mostly useful to process events using typical image processing algorithms. dv::visualization::EventVisualizer class serves a purpose to perform simple event visualization. Instead of increasing or decreasing pixel brightness, the dv::visualization::EventVisualizer just performs color coding of pixel coordinates where an event was registered.

Following sample show the use of dv::visualization::EventVisualizer class to generate colored previews of events using dv::EventStreamSlicer and dv::io::CameraCapture:

C++Python

#include <dv-processing/io/camera_capture.hpp>
#include <dv-processing/visualization/event_visualizer.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Open any camera
    dv::io::CameraCapture capture;

    // Make sure it supports event stream output, throw an error otherwise
    if (!capture.isEventStreamAvailable()) {
        throw dv::exceptions::RuntimeError("Input camera does not provide an event stream.");
    }

    // Initialize an accumulator with some resolution
    dv::visualization::EventVisualizer visualizer(*capture.getEventResolution());

    // Apply color scheme configuration, these values can be modified to taste
    visualizer.setBackgroundColor(dv::visualization::colors::white);
    visualizer.setPositiveColor(dv::visualization::colors::iniBlue);
    visualizer.setNegativeColor(dv::visualization::colors::darkGrey);

    // Initialize a preview window
    cv::namedWindow("Preview", cv::WINDOW_NORMAL);

    // Initialize a slicer
    dv::EventStreamSlicer slicer;

    // Register a callback every 33 milliseconds
    slicer.doEveryTimeInterval(33ms, [&visualizer](const dv::EventStore &events) {
        // Generate a preview frame
        cv::Mat image = visualizer.generateImage(events);

        // Show the accumulated image
        cv::imshow("Preview", image);
        cv::waitKey(2);
    });

    // Run the event processing while the camera is connected
    while (capture.isRunning()) {
        // Receive events, check if anything was received
        if (const auto events = capture.getNextEventBatch()) {
            // If so, pass the events into the slicer to handle them
            slicer.accept(*events);
        }
    }

    return 0;
}

Frame generated using dv::visualization::EventVisualizer class.

Time surface

Time surface is an event representation in an image frame, except instead of representing event in pixel brightness, it represents event in a 2D image structure, but pixel value contains the latest event timestamp.

The timestamps representations can be normalized to retrieve an image representation of the time surface. It will represent the latest timestamps with the brightest pixel values.

Following sample show the use of dv::TimeSurface class to generate time surface previews of events using dv::EventStreamSlicer and dv::io::CameraCapture:

C++Python

#include <dv-processing/core/frame.hpp>
#include <dv-processing/io/camera_capture.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Open any camera
    dv::io::CameraCapture capture;

    // Make sure it supports event stream output, throw an error otherwise
    if (!capture.isEventStreamAvailable()) {
        throw dv::exceptions::RuntimeError("Input camera does not provide an event stream.");
    }

    // Initialize an accumulator with camera sensor resolution
    dv::TimeSurface surface(*capture.getEventResolution());

    // Initialize a preview window
    cv::namedWindow("Preview", cv::WINDOW_NORMAL);

    // Initialize a slicer
    dv::EventStreamSlicer slicer;

    // Register a callback every 33 milliseconds
    slicer.doEveryTimeInterval(33ms, [&surface](const dv::EventStore &events) {
        // Pass the events to update the time surface
        surface.accept(events);

        // Generate a preview frame
        dv::Frame frame = surface.generateFrame();

        // Show the accumulated image
        cv::imshow("Preview", frame.image);
        cv::waitKey(2);
    });

    // Run the event processing while the camera is connected
    while (capture.isRunning()) {
        // Receive events, check if anything was received
        if (const auto events = capture.getNextEventBatch()) {
            // If so, pass the events into the slicer to handle them
            slicer.accept(*events);
        }
    }

    return 0;
}

Frame generated using dv::TimeSurface class.

Speed invariant time surface

Speed invariant time surface is a specific time surface variant that is more suitable for feature extraction, the implementation follows this paper: https://arxiv.org/pdf/1903.11332.pdf.

Following sample show the use of dv::SpeedInvariantTimeSurface class to generate time surface previews of events using dv::EventStreamSlicer and dv::io::CameraCapture:

C++Python

#include <dv-processing/io/camera_capture.hpp>

#include <opencv2/highgui.hpp>

int main() {
    using namespace std::chrono_literals;

    // Open any camera
    dv::io::CameraCapture capture;

    // Make sure it supports event stream output, throw an error otherwise
    if (!capture.isEventStreamAvailable()) {
        throw dv::exceptions::RuntimeError("Input camera does not provide an event stream.");
    }

    // Initialize an accumulator with camera sensor resolution
    dv::SpeedInvariantTimeSurface surface(*capture.getEventResolution());

    // Initialize a preview window
    cv::namedWindow("Preview", cv::WINDOW_NORMAL);

    // Initialize a slicer
    dv::EventStreamSlicer slicer;

    // Register a callback every 33 milliseconds
    slicer.doEveryTimeInterval(33ms, [&surface](const dv::EventStore &events) {
        // Pass the events to update the time surface
        surface.accept(events);

        // Generate a preview frame
        dv::Frame frame = surface.generateFrame();

        // Show the accumulated image
        cv::imshow("Preview", frame.image);
        cv::waitKey(2);
    });

    // Run the event processing while the camera is connected
    while (capture.isRunning()) {
        // Receive events, check if anything was received
        if (const auto events = capture.getNextEventBatch()) {
            // If so, pass the events into the slicer to handle them
            slicer.accept(*events);
        }
    }

    return 0;
}

Frame generated using dv::SpeedInvariantTimeSurface class.

Performance of available accumulators

The library performs benchmarking of available accumulation algorithms to ensure their best performance. Accumulators are also benchmarked on two metrics:

• Event throughput - measured in millions of (mega) events per second;

• Framerate - measured in generated frames per second.

The benchmarks are performed by generating a batch of events at uniformly random pixel coordinates on a VGA (640x480) resolution. Below are the results of running the benchmark on AMD Ryzen 7 3800X 8-Core Processor:

Accumulator type	Framerate (FPS)	Throughput (MegaEvent/s)
Accumulator	668	66.5
EdgeMapAccumulator	1767	149.1
EventVisualizer	785	78.3
TimeSurface	910	91.5
SpeedInvariantTimeSurface	370	36.8