ROBOTCORE Perception is an optimized robotic perception stack that leverages hardware acceleration to provide a speedup in your perception computations. API-compatible with the ROS 2 perception stack, ROBOTCORE Perception delivers high performance, real-time and reliability to your robots' perception.
Get ROBOTCORE® Perception Perception consulting
Data obtained in a robot from its sensors like cameras and lidar is typically fed into the perception turning into something useful for decision making and planning physical actions. Perception helps sense the static and dynamic objects, and build a reliable and detailed representation of the robot's environment using computer vision and machine learning techniques. The perception layer in a robot is thereby responsible for object detection, segmentation and tracking. Traditionally, a perception pipeline starts with image pre-processing, followed by a region of interest detector and then a classifier that outputs detected objects.
ROBOTCORE Perception is a hardware accelerated perception stack served as either source code or IP that allows your perception pipelines to reduce their runtime latency and increase throughput. Delivering a perception speedup, it's API-compatible with the ROS 2 perception stack simplifying its integration in existing robots, so that you don't spend time reinventing the wheel and re-developing what already works.
Benchmarks
A simple Rectify-Resize-Harris ROS perception pipeline is demonstrated which performs pre-processing (Rectify and Resize) and region of interest detection (Harris corners detector). This simple pipeline consisting of only 3 ROS Nodes takes on average 1266.32 milliseconds to execute for a single pass, which leads to less than 1 Hz when running in an idle embedded Quad-core arm Cortex-A53.
Faster robots require faster perception. ROBOTCORE Perception delivers exactly this upgrading perception pipelines with a 4.5x speedup, impacting pipeline runtime and throughput in embedded. More complex perception graphs can further leverage acceleration, achieving 10x speedups and above.
Moving faster (or with more dexterity) requires faster perception computations. Edge perception is key to navigate under environments that change, which are most in the human world. The robot below is generating information from its sensors which needs to be processed at the edge (directly in the robot) for smooth behaviors, reducing any latency from the sensors and all the way into the actuators. By leveraging hardware acceleration, robots’ perception extracts information from these sensor readings which is then used to plan actions in the order of tenths of milliseconds. All happening on the edge and responding to environmental changes.
ROBOTCORE Perception delivers faster perception capabilities to ROS computational graphs. So that your robots can move faster and better.
ROBOTCORE Perception is the result of joint research and co-development with computer system architects from Harvard University.
Read the paper
ROBOTCORE Perception is served by seasoned ROS developers for ROS development. It includes documentation, examples, reference designs and the possibility of various levels of support.
Ask about support levels(plots are interactive)
(Measurements discard ROS 2 message-passing infrastructure overhead and host-device (GPU or FPGA) data transfer overheads)
Resize - speedup
2.61x
Resize - kernel runtime latency (ms) (ROBOTCORE Perception running on an AMD KV260, NVIDIA Isaac ROS running in a Jetson Nano 2GB. Measurements present the kernel runtime in milliseconds (ms) and discard ROS 2 message-passing infrastructure overhead and host-device (GPU or FPGA) data transfer overhead)
Resize - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
Rectify - speedup
7.34x
Rectify - kernel runtime latency (ms)
Rectify - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
Harris - speedup
30.27x
Harris - kernel runtime latency (ms)
Harris - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
Histogram of Oriented Gradients - speedup
509.52x
Histogram of Oriented Gradients - kernel runtime latency (ms)
Histogram of Oriented Gradients - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
Canny Edge Tracing - speedup
3.26x
Canny Edge Tracing - kernel runtime latency (ms)
Canny Edge Tracing - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
Fast Corner Detection - speedup
8.43x
Fast Corner Detection - kernel runtime latency (ms)
Fast Corner Detection - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
Gaussian Difference - speedup
11.94x
Gaussian Difference - kernel runtime latency (ms)
Gaussian Difference - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
Bilateral Filter - speedup
9.33x
Bilateral Filter - kernel runtime latency (ms)
Bilateral Filter - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
Stereo LBM - speedup
5.19x
Stereo LBM - kernel runtime latency (ms)
Stereo LBM - % resource consumption (LUT, FF, DSP, BRAM)
(Considering an AMD KV260 board for resource estimation and an image with 4K resolution as input.
Additional details available in the ROBOTCORE Perception documentation.)
2-Node pre-processing perception graph latency (ms)
(Simple graph with 2 Nodes (Rectify-Resize) demonstrating perception
pre-processing with the image_pipeline ROS 2
package. AMD's KV260 and NVIDIA's Jetson Nano 2GB boards are used for benchmarking, the
former featuring a Quad-core arm Cortex-A53 and the latter a Quad-core arm Cortex-A57.
Source code used for the benchmark is available at the perception_2nodes
ROS 2 package)
2-Node pre-processing perception graph performance-per-watt (Hz/W)
Graph speedup - 2-Node pre-processing perception graph latency
3.6x
Performance-per-watt improvement - 2-Node pre-processing perception graph
6x
3-Node pre-processing and region of interest detector perception graph latency
(ms)
(3-Nodes graph (Rectify-Resize-Harris) demonstrating perception
pre-processing and region of interest detection with the image_pipeline ROS 2
package. AMD's KV260 featuring a Quad-core arm Cortex-A53 is used for benchmarking.
Source code used for the benchmark available at the perception_3nodes
ROS 2 package)
Graph speedup - 3-Node pre-processing and region of interest detector
perception
graph
4.5x