CSI-Camera + nvargus + pipeline issues with headless Orin NX device

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
1

Hello I have a question regarding use of gstreamer pipeline and nvargus for capturing frames on a headless jetson device. I have experience working with jetson chips on dev boards or carrier boards that have a display output and have generally had very little issue. However, most recently I am working with a Jetson Orin NX chip on a ConnectTech partner board that does NOT have a display output option and is intended to be used headless.

This board has two CSI camera ports and I am using a Raspberry PI HQ IMX477 sensor for frame capture. The issue really lies in the interaction with the camera when in a headless scenario. I am still trying to learn about this myself as there are bits and pieces all over the forum but for whatever reason if I try to do a standard OpenCV application as I would normally, using a gstreamer pipeline with nvarguscamerasrc the pipeline will fail due to their being a virtual display. Meaning that if a virtual display is detected, the entire capture pipeline will fail. I have been working on ways to bypass this.

  1. My first attempt involved going through V4L2 and capturing the frames that way and displaying them with x11 forwarding. The issue here is that with the IMX477 HQ sensor, its stuck in capturing the frames in RAW10 format and, off of additional reading, I cannot access the Jetson ISP to process these frames to make them usable.
  2. After that realization, I contacted the vendor on recommended options. They presented using udpsink to stream these frames over the network to a machine you’d want to view the camera at. This works and I am able to see a clean stream, that also surprisingly has very little latency, but the issue is that I have not figured out a method to actually work with this data. What I mean by this is that I want to capture frames from the camera, work with them on the device, and then use udpsink to send them off for viewing on another display.

So currently I am wondering if there is a proper method, or at least a recommended method for doing this. Is the headless “bug” being worked on for the Jetson platform? Hopefully this made sense and I appreciate any assistance with this problem. Thank you for your time!

2

*** Please note that this reply is generated by LLM automatically ***

I’ll provide a comprehensive answer to your question.

Overview of the issue

You’re working with a Jetson Orin NX chip on a ConnectTech partner board without a display output, and you’re trying to capture frames from a Raspberry PI HQ IMX477 sensor using a GStreamer pipeline with nvarguscamerasrc. However, you’re encountering issues due to the headless nature of the device.

Standard OpenCV application with GStreamer pipeline

In a typical scenario with a display output, you would use a GStreamer pipeline with nvarguscamerasrc to capture frames from the camera and display them using OpenCV. However, in a headless scenario, this approach fails due to the lack of a virtual display.

Alternative approach using V4L2

You attempted to use V4L2 to capture frames from the camera and display them using X11 forwarding. However, you encountered an issue with the IMX477 HQ sensor being stuck in RAW10 format, and you couldn’t access the Jetson ISP to process these frames.

Using udpsink to stream frames over the network

The vendor recommended using udpsink to stream the frames over the network to a machine for viewing. This approach works, but you’re looking for a way to capture frames from the camera, work with them on the device, and then send them over the network for viewing.

Recommended approach

To achieve your goal, you can use a GStreamer pipeline with nvarguscamerasrc to capture frames from the camera, and then use nvvidconv to convert the frames to a format that can be processed by the Jetson ISP. You can then use nvegltransform to transform the frames into a format that can be sent over the network using udpsink.

Here’s an example pipeline:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! \
  'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! \
  nvvidconv ! 'video/x-raw(memory:NVMM), format=NV12' ! \
  nvegltransform ! 'video/x-raw(memory:NVMM), format=NV12' ! \
  udpsink host=<IP address> port=<port number>

This pipeline captures frames from the camera using nvarguscamerasrc, converts them to NV12 format using nvvidconv, transforms them using nvegltransform, and sends them over the network using udpsink.

Processing frames on the device

To process the frames on the device, you can add additional elements to the pipeline, such as nvjpegenc to encode the frames as JPEG images, or nvofa to perform object detection.

For example:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! \
  'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! \
  nvvidconv ! 'video/x-raw(memory:NVMM), format=NV12' ! \
  nvjpegenc ! 'image/jpeg, width=1920, height=1080' ! \
  udpsink host=<IP address> port=<port number>

This pipeline captures frames from the camera, converts them to NV12 format, encodes them as JPEG images using nvjpegenc, and sends them over the network using udpsink.

Headless bug

Regarding the headless bug, it’s not clear if it’s being actively worked on by NVIDIA. However, there are workarounds available, such as using udpsink to stream frames over the network, or using alternative approaches like V4L2.

In summary, you can use a GStreamer pipeline with nvarguscamerasrc to capture frames from the camera, and then use nvvidconv and nvegltransform to convert and transform the frames into a format that can be sent over the network using udpsink. You can also add additional elements to the pipeline to process the frames on the device.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

1 Like
5

Hi @na258588,

I hope you are doing well.

  1. When running GStreamer pipelines with nvarguscamerasrc, the NVIDIA Argus framework creates an EGLStream to transfer frames between the camera’s ISP and GPU memory. This requires access to the Jetson’s local GPU and a valid EGLDisplay context (provided by the Jetson’s local X server).
    When using X11 forwarding over SSH, Argus attempts to create the EGLDisplay on the remote X server instead of the Jetson’s local GPU, which causes it to fail.

  2. The suggestion made by the vendor might be the right approach. You can try using in your app theappsink Gstreamer element to extract the frames coming from the camera for local processing and then use appsrc Gstreamer element to push the processed frames downstream to a udpsink so they can be viewed on another machine.

Let me know if this helps!

Best regards,
Nico
Embedded Software Engineer at ProventusNova

1 Like
6

Thank you or this information, I was able to get this working using this exact method.

1 Like