Jetson is struggling with high resolution camera

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
,
1

My current project requires me to be have as little latency as possible, one of the main reasons i moved from NX to orin was to reduce runtime of my code, even if it will only improve on a few miliseconds.

i’ve built a small test process to check for latency on my camera, what i’ve done is:
connect a LED to the jetson GPIO.
find the pixel in the frame that the LED is on.
enter a simple open camera test file.
listen to a switch before each frame grab.
when the switch is on, start a timer and send HIGH through the GPIO pin.
test the pixel the LED is on, when its green number is more than a certain number, it means the program has seen the LED is on.

parameters i change between each test:
fps- either 30 or 60
resolution - either 1080p(1920x1080) or 480p(640x480)
camera exposure time- between 1 (almost 100% dark) to 300(about 30milisec, default camera exposure)

i’ll add my test results on jetson NX first:
fps :30
resolution - 1080p
exposure - 1-10
i can see the led on the same frame i turned it on-best result, between 0 to 30 milisec delay

fps:60
res-1080p
rexposure 1-10
i can see the LED with 3-4 frame delay, meaning over 60 milisec delay

fps :60
res-480p
exposure 1-10
i can see the led on the same frame i turned it on- between 0 to 16 milisec delay but low resolution

now my ORIN test results, which have been disappointing :
fps :30
resolution - 1080p
exposure - 1-10
i can see the led on the NEXT frame i turned it on- between 30 to 60 milisec delay

fps:60
res-1080p
rexposure 1-10
i can see the LED with 5-6 frame delay, meaning over 100 milisec delay

fps :60
res-480p
exposure 1-300
i can see the led on the same frame i turned it on- between 0 to 16 milisec delay but low resolution

conclusion- ORIN seem to struggle even more than the NX when it comes to high fps and high resolution, or even high resolution in general.
in a project which i am fighting with every 1 milisecond delay, this has been a very big problem.

notes:
gstreamer pipeline to start the camera:
open("v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480 ! nvvidconv flip-method=2 ! video/x-raw(memory:NVMM), format = I420 ! nvvidconv ! video/x-raw,format=BGRx,width=(int)640, height=(int)480, framerate= (fraction)60/1 ! videoconvert !video/x-raw,format=BGR ! appsink drop=True sync=False");

to change camera settings i use v4l2-ctl

what i am looking for:
i would appreciate some advice when it comes to using cameras, maybe i am doing something wrong, maybe its a camera problem and not a jetson problem, maybe its an Opencv problem.
if anyone has a good development camera i can use i would appreciate it as well, preferably USB3 camera but i can work with a mipi camera if the cable is long enough(>35cm).

3

Try nvv4l2camerasrc instead of v4l2src to avoid memory copy to hurt the performance.

Thanks

4

Can you please provide me with a pipeline that i can run with? I’ll be able to test it by the end of the day and it can be difficult to find a good pipeline example. Unless it is only to switch between the two sources

5

Have a try below command.

gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! ‘video/x-raw(memory:NVMM), width=(int)640, height=(int)480, framerate=(fraction)30/1, format=(string)UYVY’ ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=(string)NV12’ ! nv3dsink -e
6

although the camera does start in the terminal, i am unable to find a suitable opencv gstreamer pipeline that will start the camera.
i have used these pipeline examples:

"nvarguscamerasrc ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! nvvidconv ! \
        video/x-raw(memory:NVMM),format=BGRx ! nvvidconv ! video/x-raw ! videoconvert ! video/x-raw,format=BGR ! appsink"

"nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw, format=NV12, framerate=30/1 \
    ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"

" nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1 \
     ! nvvidconv flip-method=2 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"

" nvv4l2camerasrc  device=/dev/video0 ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)UYVY \
     ! nvvidconv  flip-method=2 ! video/x-raw(memory:NVMM), format=(string)NV12 !  videoconvert !video/x-raw(memory:NVMM),format=BGR !  appsink drop=True sync=False "

all have either given me “internal pipeline error” or “no camera available”

exact error messages:

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:740 No cameras available

[ WARN:0] global /tmp/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp (1053) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /tmp/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp (616) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

(camera:17908): GStreamer-CRITICAL **: 17:55:28.643: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed
7

Try nvv4l2camerasrc instead of nvarguscamerasrc
Maybe search for some pipeline example.

8

right i tried mostly nvarguscamerasrc by accident, should only work on mipi cameras anyway.

i found this pipeline to work with nvv4l2camerasrc :

camera_one.open("nvv4l2camerasrc device=/dev/video0 ! video/x-raw(memory:NVMM),width=1920,height=1080,format=UYVY,framerate=(fraction)60/1 \
	! nvvidconv flip-method=2 ! video/x-raw,format=BGRx,width=1920,height=1080 ,framerate=(fraction)60/1! videoconvert ! video/x-raw,format=BGR ! appsink drop=1 sync=False", cv::CAP_GSTREAMER);

something a bit weird, even after changing the framerate, the actual frame rate does not change and is stuck on 60, which made tests harder to compare to v4l2 as i can’t check 30fps tests.
regardless, as for 60fps i am only able to test on NX at the moment and as for the NX i got these results:
delay of 2-4 frames meaning 32-64 milisec delay, very inconsistent but the average was about 35~ so moving to nvv4l2camerasrc did make the 60 a bit better although inconsistent.
tomorrow i will be able to test with ORIN as well but regardless 60fps is still behaving worse than 30fps, because on 30fps i have a constant delay of 30ms, with 60fps its not constant and it is still more delay.

9

Try below command before the test.

v4l2-ctl -c low_latency_mode=1
10

the camera doesn’t have that settings it seems.

v4l2-ctl --all
Driver Info (not using libv4l2):
	Driver name   : uvcvideo
	Card type     : See3CAM_CU30
	Bus info      : usb-3610000.xhci-3.4
	Driver version: 4.9.253
	Capabilities  : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
	Width/Height      : 1920/1080
	Pixel Format      : 'UYVY'
	Field             : None
	Bytes per Line    : 3840
	Size Image        : 4147200
	Colorspace        : Default
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             : 
Crop Capability Video Capture:
	Bounds      : Left 0, Top 0, Width 1920, Height 1080
	Default     : Left 0, Top 0, Width 1920, Height 1080
	Pixel Aspect: 1/1
Selection: crop_default, Left 0, Top 0, Width 1920, Height 1080
Selection: crop_bounds, Left 0, Top 0, Width 1920, Height 1080
Streaming Parameters Video Capture:
	Capabilities     : timeperframe
	Frames per second: 30.000 (30/1)
	Read buffers     : 0
                     brightness 0x00980900 (int)    : min=-15 max=15 step=1 default=0 value=0
                       contrast 0x00980901 (int)    : min=0 max=30 step=1 default=10 value=10
                     saturation 0x00980902 (int)    : min=0 max=50 step=1 default=16 value=16
 white_balance_temperature_auto 0x0098090c (bool)   : default=0 value=1
                          gamma 0x00980910 (int)    : min=40 max=500 step=1 default=220 value=220
                           gain 0x00980913 (int)    : min=1 max=100 step=1 default=1 value=1
           power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=0 value=0
      white_balance_temperature 0x0098091a (int)    : min=1000 max=10000 step=50 default=4500 value=4500 flags=inactive
                      sharpness 0x0098091b (int)    : min=0 max=127 step=1 default=16 value=16
                  exposure_auto 0x009a0901 (menu)   : min=0 max=3 default=0 value=0
              exposure_absolute 0x009a0902 (int)    : min=1 max=10000 step=1 default=312 value=1 flags=inactive
                   pan_absolute 0x009a0908 (int)    : min=-648000 max=648000 step=3600 default=0 value=0
                  tilt_absolute 0x009a0909 (int)    : min=-648000 max=648000 step=3600 default=0 value=0
                  zoom_absolute 0x009a090d (int)    : min=100 max=800 step=1 default=100 value=100

gst-device-monitor-1.0 output:
Device found:

	name  : See3CAM_CU30
	class : Video/Source
	caps  : video/x-raw, format=(string)UYVY, width=(int)2304, height=(int)1536, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 24/1, 12/1 };
	        video/x-raw, format=(string)UYVY, width=(int)2048, height=(int)1536, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 42/1, 21/1 };
	        video/x-raw, format=(string)UYVY, width=(int)2304, height=(int)1296, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 15/1 };
	        video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1280, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 50/1, 25/1 };
	        video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1, 15/1 };
	        video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)960, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 58/1, 30/1 };
	        video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1 };
	        video/x-raw, format=(string)UYVY, width=(int)1152, height=(int)768, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1 };
	        video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1 };
	        image/jpeg, width=(int)2304, height=(int)1536, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)48/1;
	        image/jpeg, width=(int)2048, height=(int)1536, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)50/1;
	        image/jpeg, width=(int)2304, height=(int)1296, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
	        image/jpeg, width=(int)1920, height=(int)1280, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)50/1;
	        image/jpeg, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
	        image/jpeg, width=(int)1280, height=(int)960, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)58/1;
	        image/jpeg, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
	        image/jpeg, width=(int)1152, height=(int)768, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
	        image/jpeg, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
	properties:
		udev-probed = true
		device.bus_path = platform-3610000.xhci-usb-0:3.4:1.0
		sysfs.path = /sys/devices/3610000.xhci/usb2/2-3/2-3.4/2-3.4:1.0/video4linux/video0
		device.bus = usb
		device.subsystem = video4linux
		device.vendor.id = 2560
		device.vendor.name = "e-con\\x20systems"
		device.product.id = c130
		device.product.name = See3CAM_CU30
		device.serial = e-con_systems_See3CAM_CU30_152B9904
		device.capabilities = :capture:
		device.api = v4l2
		device.path = /dev/video0
		v4l2.device.driver = uvcvideo
		v4l2.device.card = See3CAM_CU30
		v4l2.device.bus_info = usb-3610000.xhci-3.4
		v4l2.device.version = 264701 (0x000409fd)
		v4l2.device.capabilities = 2216689665 (0x84200001)
		v4l2.device.device_caps = 69206017 (0x04200001)
	gst-launch-1.0 v4l2src ! ...
11

OK, it’s USB camera instead of CSI camera to enable the low_latency_mode=1
Try boost the system as performance to try.

sudo nvpmodel -m 0
sudo jetson_clocks
sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vic/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vic/state
cat /sys/kernel/debug/bpmp/debug/clk/vic/max_rate > /sys/kernel/debug/bpmp/debug/clk/vic/rate
13

it seems to have made it better, it is almost consistent 1 frame delay, meaning 30~ miliseconds delay, which is about the same as 30fps when using v4l2src. even sometimes less. maybe having a better camera would lower delay even more. if i’m working with 120fps camera and i am 1 frame delay it is something i can work with. still, using sudo su without disabling password requirement isn’t something i can do.
so when it comes to delay it seems to be about the same between the frame rates but i would still like to lower delay with the 60fps to 0 frames if that is even possible

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.