Jetson Nano 2gb RPI Camera v2.1

[codydg]

I am trying to use a Raspberry Pi Camera, v2.1 with my Jetson Nano and I’ve had some success, but it’s limited.

What I’ve done which has worked:

1. Follow instructions on setting up the Jetson Nano.
2. Run this script, replacing version with “4.6.0” to get a CUDA-enabled, updated version of OpenCV.
3. Power off, plug in Raspberry Pi Camera v2.1, and power back on.
4. Capture a video to confirm functionality: nvgstcapture-1.0 --mode=2 --automate --capture-auto
5. Make a C++ Project to test everything so far.
   A. Make CMakeLists.txt which creates an executable based on main.cpp, and targets OpenCV.
   B. Make main.cpp which contains this example code for Background Subtraction
6. mkdir build && cd build && cmake .. && make
7. Run built executable, providing it --input=../video.mp4 where video.mp4 is the output from step 2.

Success! The Background Subtraction worked and I was able to watch both panes created by imshow. Next, I wanted to use the live camera feed.

At first I tried the change below:

--VideoCapture capture( samples::findFile( parser.get<String>("input") ) );
++VideoCapture capture(0);

This resulted in a solid green frame which also ran incredibly slow.

After reading this forum post, I tried changing the line to:

VideoCapture capture("nvarguscamerasrc ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink", CAP_GSTREAMER);

As I was writing this forum post, I ran the code as stated above so I could copy the error and paste it here. However, it worked… Once. I closed the executable and re-ran it and it didn’t work the second time. Eventually I rebooted and here’s the current state:

After a fresh reboot, I can run nvgstcapture-1.0 --mode=2 --automate --capture-auto as many times as I want, and I will get a successful recording. But if I run the executable which has identical source code to that linked in Step 5B, except for the line of code that now reads VideoCapture capture("nvarguscamerasrc ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink", CAP_GSTREAMER);, I get the following output:

GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 2 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
CONSUMER: ERROR OCCURRED
[ WARN:0@4.217] global /home/codydg/opencv/workspace/opencv-4.6.0/modules/videoio/src/cap_gstreamer.cpp (2402) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module nvarguscamerasrc0 reported: CANCELLED
GST_ARGUS: Cleaning up
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
[ WARN:0@124.921] global /home/codydg/opencv/workspace/opencv-4.6.0/modules/videoio/src/cap_gstreamer.cpp (1356) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@124.922] global /home/codydg/opencv/workspace/opencv-4.6.0/modules/videoio/src/cap_gstreamer.cpp (862) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Unable to open camera!
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Argus client is exiting with 4 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)

Note that there is a significant amount of time after it says “Cleaning up” where I was simply waiting for output. From then, if I keep running it, I get the following output:

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:751 Failed to create CaptureSession
[ WARN:0@0.890] global /home/codydg/opencv/workspace/opencv-4.6.0/modules/videoio/src/cap_gstreamer.cpp (1405) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1

And any attempt to run will now result in the following output:

Encoder null, cannot set bitrate!
Encoder Profile = High
bitrate = 4000000
Encoder Profile = High
Encoder control-rate = 1
Encoder EnableTwopassCBR = 0
Opening in BLOCKING MODE
** Message: 19:39:13.727: <main:4674> iterating capture loop ....
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:751 Failed to create CaptureSession

It will still generate an MP4 file, but all programs just say the MP4 is corrupted.

I’m very confused about this. I especially don’t understand why attempting to run the executable results in the camera being seemingly inoperable until I reboot again!

[codydg]

Some other things I tried that might not be relevant:

After reading this forum post, I tried the pipeline provided:

v4l2src device=/dev/video0 ! image/jpeg, width=640,height=480,framerate=30/1,format=MJPG ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=true

Which outputted:

Opening in BLOCKING MODE
[ WARN:0@1.182] global /home/codydg/opencv/workspace/opencv-4.6.0/modules/videoio/src/cap_gstreamer.cpp (2402) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Internal data stream error.
[ WARN:0@1.187] global /home/codydg/opencv/workspace/opencv-4.6.0/modules/videoio/src/cap_gstreamer.cpp (1356) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@1.187] global /home/codydg/opencv/workspace/opencv-4.6.0/modules/videoio/src/cap_gstreamer.cpp (862) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Unable to open camera!

I figured this was due to an incompatibility between v4l2src and my Raspberry Pi Camera. I found rpicamsrc which supports image/jpeg like the pipeline above used, so I tried changing the pipeline to:

rpicamsrc device=/dev/video0 ! image/jpeg, width=640,height=480,framerate=30/1,format=MJPG ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=true

Which outputted Opening in BLOCKING MODE, then hung indefinitely.

[JerryChang]

hello codydg,

could you please try convert the format as BGRx for testing?
for example, cap = cv2.VideoCapture("nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! appsink")

[codydg]

Since I’m using C++, I believe the equivalent code to what you’ve proposed is:

cv::VideoCapture capture("nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! appsink");

Should I still include cv::CAP_GSTREAMER as a second argument? Meaning should I instead use this:

cv::VideoCapture capture("nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! appsink", cv::CAP_GSTREAMER);

For my testing using your provided pipeline, I kept CAP_GSTREAMER, and it worked well! I started changing things around and I tried all of the following, which worked successfully:

nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! appsink
nvarguscamerasrc ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! appsink
nvarguscamerasrc ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink

Notably, the last one of those is one I tried previously which didn’t work before, and your pipeline didn’t work the first time I tried it, either. I have changed no software configuration, but I did end up unplugging the camera while the Jetson Nano was turned off, and after reconnecting it later, everything above worked. So I’m now wondering if this was truly a loose ribbon cable, or something similar.

I appreciate the quick response, and I’m glad it’s all working. I wish I had a more concrete answer to what was going on for the forums’ sake but it does really seem like it was a hardware issue.

Thank you for your help!
-Cody