CUDA support in Intel RealSense library

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

We’re working on a project involving some computer vision. Our target platform includes multiple Intel RealSense cameras connected to an NVIDIA Jetson TX2.

The performance issue we’re running into is when we need to register the depth image of one camera to its RGB image, or transform the image of multiple cameras to a common world space. The Intel RealSense library has a function that performs the intra-camera registration (they call it “align”), but it’s simply too slow on the Jetson (200ms per frame). We’re looking for a CUDA implementation of the registration algorithm, either as part of that library or as a standalone component. Intel plans on supporting this in the future, but it would be beyond our preferred timeframe.

Has anyone looked at this, or have any pointers to something we might be able to leverage?

#2

Hi,

We don’t have experience of this camera.

But suppose you can open an onboard/USB camera with GStreamer.
You can check our MMAPI sample to see if helps.

Thanks.

#3

Acquiring the data is not an issue. The issue is the depth/color registration. The camera itself is not important - I only mention the camera because it’s a common camera for Jetson users, and Intel has C++ and SSE implementations, but no CUDA implementation of this important method.

#4

Hi,

We don’t have registration implementation for Jetson.(But there is one for DIRVE platform)

Suppose you can achieve it with feature point and warping.
Check NPP and VisionWorks SDK to see if helps.

NPP: https://developer.nvidia.com/npp
VisionWorks: https://developer.nvidia.com/embedded/visionworks

Thanks.

#5

My request to access the DRIVE platform was rejected. How can we get this implementation?

#6

Hi,

Please noticed that the DRIVE libraries are not supported by Jetson.

May I know more about your use case?
One possible alternative is to generate transform matrix offline.

You can use the RealSense library to generate the extrinsic matrix. (suppose this is an ONE time job)
And apply the warping operation with VisionWorks at runtime.

Thanks.

#7

We’re working on a project for a client that involves some computer vision. Our target platform includes multiple Intel RealSense cameras connected to an NVIDIA Jetson TX2.

The performance issue we’re running into is when we need to register the depth image of one camera to its RGB image, or transform the image of multiple cameras to a common world space. The Intel RealSense library (librealsense) has a function that performs the intra-camera registration (they call it “align”), but it’s simply too slow on the Jetson (200ms per frame). We’re looking for a CUDA implementation of the registration algorithm, either as part of librealsense or as a standalone component. Intel plans on supporting this in the future, but it would be beyond our timeframe.

We have both the intrinsics and extrinsics. The only issue is speeding up the registration function. I’ll take a look at the VisionWorks functions.

Thanks.

#8

I think VisionWorks may be exactly what we need. We already use OpenCV for part of our pipeline, so we should just be able to leverage VW there instead of doing the work in the image acquisition phase.

Thanks.