IMX219-83 depth imaging freezes with Jetson Xavier NX

Hello,

currently I am trying to get a RGB-Stereo Camera (IMX219-83) to work with the Jetson Xavier NX.

I have tried several projects that should be able to stream cameras, but they tend to crash or time out.
One of them was a stereo project of my own which did work with another Ubuntu 18.04 System.
This one works just with a low resolution
Another project

The very recent test has been some code to calculate the Disparity of two camera streams. The code itself runs, but it starts with a very low framerate and after some time (~5-10 sec) no more images are being displayed, and the program freezes.

The camera itself is working (even though the Xavier just displays it up to 1280x720, it supports 3264x2464) and also with a reasonable framerate.

DISPLAY=:0.0 gst-launch-1.0 nvarguscamerasrnsor-id=0 ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=(string)NV12, framerate=(fraction)20/1' ! nvoverlaysink -e

In another topic I noticed, that several people had problems with the SDK, as it could not flash or install the software, so that I had to do it myself. Here is the link.

Summing it all up in a question: Is there a good guide (preferable with code-examples in c++) how to properly set up the Xavier for a camera stream that utilizes the GPU capacities for depth imaging?

If not, then I’d also appreciate relating topics that might help.

Stay Safe
kremerf

hello kremerf,

you’re having assign a 1280x720 sensor mode to render display according to your gstreamer commands,

you may refer to below gstreamer pipeline, which enable 3264x2464 sensor mode.
it also adding video converter (i.e. nvvidconv ) to downscale the frame resolution as 1920x1080 for display preview frames.
for example,
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=3264, height=2464, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, width=1920, height=1080' ! nvoverlaysink -ev

hello kremerf,

may I know were both of your cameras already has frame synchronize?
you may refer to Topic 111355, there’s software approach (i.e. syncSensor example) to have dual-frame synchronization.

here’s one discussion thread you may check, Topic 80215.
it works with stereo cameras and using Argus to retrieve frames; OpenCV to remap and visionworks to calculate the disparity.

BTW, there’s also Isaac SDK you could refer to, for example, Coarse-to-Fine Stereo Depth.
thanks

Oh yes, the code I posted is fixed to 1280x720, as that was the one working.

I tried your suggestion with following result:

$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=3264, height=2464, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, width=1920, height=1080' ! nvoverlaysink -ev
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3264, height=(int)2464, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)3264, height=(int)2464, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)NV12, pixel-aspect-ratio=(fraction)459/616
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)NV12, pixel-aspect-ratio=(fraction)459/616
/GstPipeline:pipeline0/GstNvOverlaySink-nvoverlaysink:nvoverlaysink-nvoverlaysink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)NV12, pixel-aspect-ratio=(fraction)459/616
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)NV12, pixel-aspect-ratio=(fraction)459/616
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3264, height=(int)2464, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)3264, height=(int)2464, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 3264 x 1848 FR = 28,000001 fps Duration = 35714284 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1920 x 1080 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1280 x 720 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;
GST_ARGUS: 1280 x 720 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;
ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 679 Frame Rate specified is greater than supported
GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 3264 H = 2464 
   seconds to Run    = 0 
   Frame Rate = 21,000000 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 899 InvalidState.
GST_ARGUS: Cleaning up
Got EOS from element "pipeline0".
Execution ended after 0:00:01.420270695
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I’ll have a look at the topics you mentioned. thank you for those. Currently I have not synchronized the cameras, as I thought still images should work either way.

hello kremerf,

please also check sensor capability shown by GST_ARGUS.
you may modify the gstreamer pipeline to 3264x2464@21-fps to match your sensor outputs.
for example,

GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;
...

Using 21 FPS results in an image. Great! So all resolutions are working, when displayed in a smaller size.

I will have a look at the other topics now and if questions arise or I get some great solutions I’ll keep this topic up to date.