Isaac ROS NV12 NitrosImage

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

Hello,

I have an application where I have an MJPEG USB webcam I am trying to integrate into a Nitros graph on a Jetson Orin Nano Super. My approach has been to use a gstreamer pipeline to take the image from a v4l2src and pass it to a nvvideodecoder set to decode MJPEG, then pass it to nvvideoconvert and lastly to a custom gstreamer sink that takes the surfacebuffer from nvvideoconvert and publishes it via a Nitros Image (doing the EGLimage thing for full zero-copy goodness).

Shockingly this works, and works fairly well, but most of the downstream ROS Isaac nodes want rgb8 or bgr8, and nvvideoconvert only does RGBA as an RGB-adjacent planar image type - I have gotten the entire thing working by throwing an Isaac image format converter shim in the way, but it takes a fair amount of processing power that probably doesn’t need to be spent. What I really want to do is use NV12, since nvvideoconverter supports it and the majority of downstream blocks do as well.

I have been working my way through the process and think I have a reasonable handle on it:

  • Nitros ImageBuilder doesn’t strictly do semi-planar types, but will treat nv12 types as contiguous with a Y plane followed by a half-height UV plane, and there is a constructor in the ImageBuilder that handles it

  • nvvideoconvert seems to put the Y and UV planes on 256-byte boundaries, so for most practical image sizes there will be a gap between Y and UV, so doing a quick cudamemcpy to make the planes contiguous is often necessary

  • Subscribing to a nv12 Nitros Image type via something like rqt_image will trigger a CPU copy and conversion to RGB8, so one can rarely actually see the nv12 image type but can see the image post-conversion.

Annoyingly, the image that I get out of rqt_image shows the Y channel perfectly well (full black-and-white frame, looking accurate) but the chroma channels appear to take up the upper half of the image (red, green, and blue are present, but the chroma channel is squished to half-height and only appears in the top half of the image) and the lower half of the image is tinted strongly green (usually associated with a zero in the chroma channel). It’s like the nv12 conversion to RGB8 isn’t properly respecting the half-height chroma, and instead assembling the other frame channels incorrectly. I have attempted simply cudamemcpying the Y and UV channels directly, with no respect for pitch, and even doubled the UV channel to see if it got rid of the green tint at the bottom half of the image, and I don’t see any change. Is there a bug in nv12 conversion with Isaac ROS 3.1 around nv12 image conversion? Barring that, is there an example somewhere of creating an nv12 image with Nitros ImageBuilder?

Thanks in advance!

2

Hello @sparks333,

Thanks for your post.

You can tried cudaMemcpy2D instead of CudaMemcpy to copy the data, respecting the source pitch (stride) of the Y and UV planes from the nvvideoconvert output while writing to a destination buffer that has zero pitch (i.e., packed).

CudaMemcpy is probably using the padded source pitch for the UV plane but only copying up to height / 2, so the total length of the UV plane in the destination buffer is not properly calculated.

3

Hello,

Good thought, but I have tried 2D memcpy as well, respecting the source pitch as reported by the eglimage and setting a contiguous destination pitch, with the same result.

After messing with it a bunch yesterday, I decided to test an assumption and briefly redirected the destination memory buffer to the host device and memcopied using the same method to a local buffer that was written to file, then opened with ffplay on raw mode specifying nv12 format. The image looked perfect. Thus, I must conclude that either rqt_image or the Nitros image bridge that converts from nv12 or rgb8 has a bug.

In any case, I ended up ditching the cudamemcpy for a vppi nv12 to rgb8 conversion, turned out it was super cheap and fast and (except for increased bandwidth requirements) made debugging the system much smoother.

4

Glad to hear you solved the problem. I’ll also report this issue to our development team to look into it. Thanks!