NvArgus Camera Issue with Xvfb and GStreamer

[doruk.sonmez1]

Hi All,

I’m trying to run my CSI camera pipelines under a virtual display using Xvfb. However, it seems there is some bug in the never version of NvArgus library on JetPack 5.0.2 since I also have tested the same thing on JetPack 4.6 without any errors.

This is my working terminal command on JetPack 4.6.0:

$ xvfb-run -a gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280,height=720'! nvvidconv ! 'video/x-raw,width=640,height=360,format=(string)BGRx' ! videoconvert ! 'video/x-raw,format=(string)BGR' ! appsink

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28,000001 fps Duration = 35714284 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120,000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.

So this pipeline hangs until I do Ctrl+C as expected. However the below pipeline throws error from Argus library and exits on JetPack 5.0.2:

$ xvfb-run -a gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280,height=720'! nvvidconv ! 'video/x-raw,width=640,height=360,format=(string)BGRx' ! videoconvert ! 'video/x-raw,format=(string)BGR' ! appsink

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function initialize(), line 89)
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function create(), line 44)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadInitialize:320 Failed to create FrameConsumer
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:241 (propagating)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, waitRunning:203 Invalid thread state 3
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:806 (propagating)
WARNING: from element /GstPipeline:pipeline0/GstAppSink:appsink0: Pipeline construction is invalid, please add queues.
Additional debug info:
gstbasesink.c(1209): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstAppSink:appsink0:
Not enough buffering available for  the processing deadline of 0:00:00.020000000, add enough queues to buffer  0:00:00.020000000 additional data. Shortening processing latency to 0:00:00.000000000.
Got EOS from element "pipeline0".
Execution ended after 0:00:00.294815564
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I have checked some version from /usr/src/jetson_multimedia_api/argus/RELEASE.txt on both devices and it seems there is a minor update from 0.98 to 0.99 which may cause this error.

JetPack 5.0.2

===============================================================================
Argus Camera API Release Notes
===============================================================================

-------------------------------------------------------------------------------
Release 0.99 (07/13/2022)
-------------------------------------------------------------------------------

JetPack 4.6.0

===============================================================================
Argus Camera API Release Notes
===============================================================================

-------------------------------------------------------------------------------
Release 0.98 (06/23/2021)
-------------------------------------------------------------------------------

Can someone please look into this? Thanks in advance

[doruk.sonmez1]

I got rid of the buffering warnings by adding queues to the pipeline for a clearer view:

$ xvfb-run -a gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! queue ! nvvidconv ! 'video/x-raw,width=640,height=360,format=(string)BGRx' ! queue ! videoconvert ! 'video/x-raw,format=(string)BGR' ! queue ! appsink

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function initialize(), line 89)
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function create(), line 44)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadInitialize:320 Failed to create FrameConsumer
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:241 (propagating)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, waitRunning:203 Invalid thread state 3
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:806 (propagating)
Got EOS from element "pipeline0".
Execution ended after 0:00:00.294815564
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

[ShaneCCC]

So still have problem for this topic?
What’s xvfb?

[doruk.sonmez1]

Yes, I still have it.

And in terms of Xvfb and what it is, this is the description:
" Xvfb or X virtual framebuffer is a display server implementing the X11 display server protocol. In contrast to other display servers, Xvfb performs all graphical operations in virtual memory without showing any screen output."

I’m using Xvfb to run my Electron application in headless mode on Jetsons. However the problem is not related to Electron because my pipeline doesn’t keep playing as it is expected. It just dies.

[ShaneCCC]

What’s the reason to use Xvfb?

[doruk.sonmez1]

As I described in my last comment:

I’m using Xvfb to run my Electron application in headless mode on Jetsons. However the problem is not related to Electron because my pipeline doesn’t keep playing as it is expected. It just dies.

If there is any other option that will allow me to run some commands in a virtual display, I can work with that too.

[ShaneCCC]

What the reason for “Xvfb to run my Electron application in headless mode”

@doruk.sonmez1

Xvfb to run my Electron application in headless mode

[doruk.sonmez1]

My Electron application designed in a way to run on some sort of display. If we are using any Jetson in headless mode or even a server, we run it with Xvfb to trick the application with a virtual display.

[ShaneCCC]

The r32 is much different with r35 the GPU driver also different.
The eglstream initial failed without DISPLAY.
I don’t have solution for it now.

[doruk.sonmez1]

So is this also means there is no way to get CSI camera work even with a VNC on newer JetPacks?

I just set up a VNC server following the link below:

GitHub - overclock98/Jetson_Nano_true_Headless_setup_without_hdmi_display:...

Now I can connect to the VNC display using RealVNC and my Electron application is working since it gets display even though it is a dummy one. But there is still something strange about the Argus Library. If I set DISPLAY environment variable, it throws the same error but if I unset the DISPLAY variable, I can get it working. Please check the difference from the attached screenshots.

[ShaneCCC]

I would suggest using teamvviewer for your case if you want to check the preview.
Due to VNC using virtual display that argus don’t support it.

[DaneLLL]

Hi,
You may try
Jetson AGX Orin FAQ

Q: How to configure VNC w/o monitor connected for Jetson?

And another method is to run

/usr/src/jetson_multimedia_api/samples/10_argus_camera_recording/

And check encoded h264 stream.

[doruk.sonmez1]

Hi @DaneLLL,

Thanks for the comment. I actually deployed VNC successfully with the same steps and the screenshots I have posted in my earlier comment was actually screenshots from VNC screen (RealVNC), so those are the results with the VNC.

The other method you shared results with some related error too as you can see from the screenshot (VNC screen) below. Why is the NvArgus camera library is so strictly connected to some output element (EGLImage) in JetPack 5? This kind of blocks using a Jetson with a CSI camera in headless mode.

[DaneLLL]

Hi,
For information, is xvfb based on framebuffer console? This is not supported on Jetpack 5/Orin:
Running without display manager - #4 by WayneWWW

If xvfb depends on framebuffer console, it is not supported on Jetpack 5.

[doruk.sonmez1]

So the problem is out of the xvfb scope now. You mean it is not possible to use CSI camera on any Orin/Jetpack 5 without a monitor because none of the VNC solutions or any other virtual X server options seems working. I was also thinking that I might use a CSI camera with a Jetson Orin NX on a robotics application but it seems it is off the table too.

May I ask why exactly it is not possible to get a CSI camera pipeline work without a display on Jetpack 5.x while it works perfectly on Jetpack 4.6.x? I mean why framebuffer console support is dropped on JetPack 5 since all the integrated or third party solutions are actually using it?

If there is any option that is not based on framebuffer console, I would be glad to hear it and do some testing.

[DaneLLL]

Hi,
Argus in headless mode should be working since there are similar use-cases. Please do this on developer kit and give it a try:
Jetson AGX Orin FAQ

Q: How to run developer kit in headless mode?

This is to run headless mode on developer kit.

[doruk.sonmez1]

I don’t think that thread is relevant with CSI camera usage. How about the following part from the JetPack documentation? Since the error source is EGLStream Consumer and what I understand from this specific part of the documentation, NVIDIA dropped nvbuf_utils completely and used EGLOutput Consumers instead in JetPack 5. Since we are bound to some monitor window, I suspect the preferred consumer is EGLOutput consumers. But the docs are also mentioning about two other consumer types. Maybe we could use CUDA frames as its mentioned in CUDA consumers without involving EGL output. Since these are options, I think we should be able to choose one of each depending on our needs. How we can do that? How and where exactly CUDA consumers can be used at?

[DaneLLL]

Hi,
On Jetpack 5 NvBuffer is deprecated and replaced with NvBufSurface. There are similar NvBufSurface APIs to map the buffer to EGLImage and get CUDA pointer. Please refer to cuda_postprocess() in 12 sample and apply it to 10 sample. This should also work on headless device.

And please use latest Jetpack 5.1.2. Certain issues are fixed in later releases.

[doruk.sonmez1]

Hi @ShaneCCC @DaneLLL ,

Thank you for your collaboration. I solved it with “tricking” the device hardware-wise instead software-wise. Anyone who is going to come across to this topic and looks for a solution, just use a dummy HDMI or DisplayPort plug sold online marketplaces for around $10 - $20.