Event-based computer vision and processing internship

Overview

About PROPHESEE. Prophesee is the inventor of the world’s most advanced neuromorphic vision systems. The company developed a breakthrough event-based vision approach to machine vision that enables dramatic reductions in power consumption, latency, and data processing requirements. By mimicking how the human eye and brain work, Prophesee’s patented Metavision® sensors and algorithms reveal information that is invisible to traditional frame-based sensors.

Prophesee’s technology is transforming applications across industrial automation, aerospace and defense, autonomous systems, IoT, AR/VR, and mobile. Headquartered in Paris, Prophesee has offices in Grenoble and Shanghai.

Job background: Prophesee designs and produces a new type of cameras that are bio-inspired and do not gather information with a fixed frame-rate; instead each pixel is captured asynchronously when needed. This is called event-based image processing. The output is extremely sparse and enables real-time treatment of the information at an equivalent frequency of a kHz or more. Prophesee also advances the algorithmic and machine learning side of this new kind of machine vision to enable its clients to build new applications mainly in automotive, virtual reality and industrial automation.

Internship details

The intern will be part of the Event Signal Processing (ESP) team, whose main objective is to design algorithms close to the pixel array. Noise filtering, flicker detection and mitigation, or bandwidth control are some of the ESP features already improving the data generated by the Prophesee sensors.

Topics

1. Topic 1: Event-based focusing in the wild
2. Topic 2: Event-based biasing in active light conditions
3. Topic 3: Event-based 2D features for drone navigation for embedded ML
4. Topic 4: Event-based stream encoding and data storage

Topic 1 description: Focusing event sensor is well established in conditions where the target is highly contrasted and stable. This introduces restrictions in applications where the object of interest is moving fast in cluttered environments. Optimizing the lens position is crucial to maintain high contrasts on objects of interest, while removing part of its background. The motion-aware data stream of event sensors contains unique information, such as depth, stability, occlusions, that ease the focusing of the sensor. Some statistics can be extracted close to the sensor in the ESP processing pipeline, and it makes sense to run the auto-focusing algorithm near the sensor to exploit the low latency of events. The internship aims to evaluate and implement a full event-based auto-focusing algorithm running next in the sensor ESP. Road-map: implement latest focusing algorithm, optimize for resource-constrained platforms, organize benchmarking and evaluation procedures of the solution and the comparison with the state of the art.

Topic 2 description: The event-based sensor shines when synchronized with an illumination source, enabling applications such as depth estimation, eye tracking, visual light communications, SLAM, etc. Light is often pulsed at high frequencies to generate unique features, and this process can be used to communicate information about the object state. The goal is to implement and improve algorithms to robustify event-based sensors against adversarial active lighting conditions. Algorithms can be programmed inside the sensor or next to it, which is mandatory where latency is constrained. Road-map: implement latest event-based algorithms for active lighting demodulation, prototype on the sensor ESP and help build a sensor filter, and improve the existing test bench for benchmarking.

Topic 3 description: In robotics, the shape is used to estimate the 3D transformation between the object and the camera (e.g., SLAM). The raw timestamp information of the event pixel can be discarded for such features, as only the spatial structure of the event is used by the algorithm. Timestamps are used to smoothly track features. Filters were designed to extract such information inside the sensor ESP, and the output can be combined with conventional ML algorithms to enhance performance. The internship aims to evaluate how these sensor filters can better use the ESP spatial filters and how including these ESP filters in training can lead to more efficient filter transfer functions. Road-map: implement machine learning pipelines using ESP 2D features, benchmark against existing algorithms with an emphasis on data/power reduction, design next-generation filters inside the sensor ESP and implement processing on an embedded platform.

Topic 4 description: Event-based streams are sparse and asynchronous, creating challenges for data exchange, storage, and decoding. Defining generic and efficient encoding schemes is a key enabler for interoperability and high-performance processing pipelines. The internship will investigate and benchmark new encoder/decoder techniques, explore data representations and compression strategies, and identify trade-offs across bandwidth, latency, and storage. Road-map: propose and implement generic encoding/decoding techniques, design benchmarking methodologies, evaluate compression ratio, bandwidth, latency, complexity, and robustness, and optimize encoding technologies and parameters to guide future standardization.

A large part of the work will involve proposing new encoding techniques, implementing prototypes, and benchmarking on representative datasets. The internship may involve using machine learning or optimization frameworks to tune encoding parameters or guide design space exploration. This work helps define future data formats for event-based data exchange and storage.

Responsibilities and requirements

• Required Qualifications, Experience, And Skills

• Good programming skills (C/C++)
• Mathematical and/or Computer Science
• Computer Vision, ML, or Robotics
• Benchmarking and algorithm evaluation
• Digital/Embedded platforms/Microcontroller is a plus
• English C1 minimum

Education

Master 1 or 2

Soft skills

Strong problem-solving skills, strong analytical skills. Flexible to dynamic environments and fast changing technologies. Passionate about technology. Team player. Good sense of autonomy. Must be pragmatic and self-motivated to complete a task even if it is outside of just the “well known” realm. “Can Do Attitude” is preferred.