Is it possible to run a GStreamer pipeline in headless mode with the source plugin nvarguscamerasrc?

[padames]

Continuing the discussion from NvArgus Camera Issue with Xvfb and GStreamer.

I am integrating Connect Tech’s Photon NGX-003 board with ORIN NX (16GB) and e-Con Systems’ visible light camera eCAM80_CUNX/Starvis IMX415 for an AI Vision application. Our production system ships headless. The following two pipelines work fine when the Gnome desktop is enabled and active, and a computer display is physically connected to the board. They fail if the display is disconnected or if the desktop is stopped and disabled, and the system is restarted in non-graphical mode, and the unit is accessed from an Ubuntu desktop via ssh -Y <orin_host_name> such that we can serve the remote X client requests with the local X server through ssh. Our Jetpack is 5.1.2.

gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=0 ! "video/x-raw(memory:NVMM), width=1280, height=720, framerate=10/1, format=(string)NV12" ! queue ! nvvidconv flip-method=0 ! "video/x-raw, format=(string)BGRx" ! videoconvert ! "video/x-raw, format=(string)BGR" ! nv3dsink -e

gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=0 ! "video/x-raw(memory:NVMM), width=1280, height=720, framerate=10/1, format=(string)NV12" ! queue ! nvvidconv flip-method=0 ! "video/x-raw, format=(string)BGRx" ! videoconvert ! "video/x-raw, format=(string)BGR" ! nveglglessink

The errors are, for the sink nv3dsink:

libEGL warning: DRI3: failed to query the version
libEGL warning: DRI2: failed to authenticate
Setting pipeline to PAUSED ...
libEGL warning: DRI3: failed to query the version
libEGL warning: DRI2: failed to authenticate
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
X Error of failed request:  BadRequest (invalid request code or no such operation)
  Major opcode of failed request:  155 ()
  Minor opcode of failed request:  1
  Serial number of failed request:  20
  Current serial number in output stream:  20
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)

For the nveglglessink sink:

libEGL warning: DRI3: failed to query the version
libEGL warning: DRI2: failed to authenticate
Setting pipeline to PAUSED ...

Using winsys: x11 
Pipeline is live and does not need PREROLL ...
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function initialize(), line 89)
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function create(), line 44)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadInitialize:320 Failed to create FrameConsumer
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:241 (propagating)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, waitRunning:203 Invalid thread state 3
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:806 (propagating)
Got EOS from element "pipeline0".
Execution ended after 0:00:00.117963862
Setting pipeline to NULL ...
Freeing pipeline ...

(gst-launch-1.0:124008): GStreamer-WARNING **: 18:35:14.699: The default memory allocator was freed!

(gst-launch-1.0:124008): GStreamer-CRITICAL **: 18:35:14.700: gst_object_unref: assertion '((GObject *) object)->ref_count > 0' failed

We have used the virtual X server Xvfb for years to support our non-GUI C++ analytics application without issues in a headless environment. The GStreamer pipelines cannot use Xvfb, we noticed the XServer installed via gdm3 has at least two extensions: NV_CONTROL and NV_GLX that Xvfb does not have. I don’t know anything about these extensions.

This is the first time in years that we have faced difficulties capturing video from a camera in a headless system or when the desktop is enabled but the Display is not physically connected.
Is this a requirement of the Argus source plugin nvarguscamerasrc?

Any ideas on how to make this work for us will be greatly appreciated.

[DaneLLL]

Hi,
We don’t support Argus stack with virtual display. For this use-case, we would suggest set up UDP or RTSP to stream out camera frames.

[padames]

@DaneLLL Thanks for the response. Any documentation that you may suggest? The ultimate goal for the pipeline on the consuming end is to have the appsink as the receiver of raw video in BGR pixel format.
A comment about this. Doesn’t this defeat the purpose of having a MIPI camera in the first place? I mean, in an edge device, one wants the tightest loop possible between the physical sensor and the software that processes the signal. This sounds like additional latency in the loop between the sensor, the analytical processing of this input and the response to it.

[DaneLLL]

Hi,
There are examples of UDP and RTSP in FAQ:
Jetson AGX Orin FAQ

Please take a look. Since Orin Nano does not have hardware encoder, please use software encoder.

[padames]

@DaneLLL Thanks for the expanded answer. I can reproduce the examples from the first two posts of the three. However, the examples use a videotestsrc. How could I put the output of my camera, which is in pixel format NV-12 through this pipeline?

One of the examples that worked in my ORIN NX-based system is:

gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,width=1280,height=720,format=I420 ! timeoverlay valignment=4 halignment=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! h264parse ! mpegtsmux ! udpsink host=<host_PC_ip> port=5003 sync=0

I continued this discussion under Orin NX, follow the link below.