How much cameras AXG Xavier can processed?

[kuskov.stanislav]

Hi! my hw is AXG xavier with gmsl board and 8 cameras leopard imx390.
When I’m truing get stream from 4 cameras with full HD I have decreasing fps of them to 0.057

My question is:
It’s ok? Maybe I must use something for speedup it.

Powermode is MAXN, jetson_clocks is on
My command is:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! xvimagesink & gst-launch-1.0 nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! xvimagesink & gst-launch-1.0 nvarguscamerasrc sensor-id=2 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! xvimagesink & gst-launch-1.0 nvarguscamerasrc sensor-id=3 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! xvimagesink &

PS. when i decrease resofution FPS increased

[ShaneCCC]

Could you try below sink for 4 HD cameras.

gst-launch-1.0 nvarguscamerasrc ! ‘video/x-raw(memory:NVMM),width=1920,height=1080,format=NV12’ ! nvvidconv ! fpsdisplaysink video-sink=fakesink

[kuskov.stanislav]

Hi! I’m trying it for each camera in a different terminal and its give me a 26 fps average for all of them

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! fpsdisplaysink video-sink=fakesink -v

[ShaneCCC]

Could you boost the emc clock and below clocks to check.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate

[kuskov.stanislav]

root@jetson:/home/sda# echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
root@jetson:/home/sda# echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
root@jetson:/home/sda# echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
root@jetson:/home/sda# cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
998400000
root@jetson:/home/sda# cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
1190400000
root@jetson:/home/sda# cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
400000000
root@jetson:/home/sda# 

When 4 streams are started

[ShaneCCC]

I don’t see the result?
Could you also check the emc clock is set as max too.

[kuskov.stanislav]

my results:
998400000
1190400000
400000000

[kuskov.stanislav]

@ShaneCCC

emc clock is set as max

sudo jetson_clocks --show
SOC family:tegra194 Machine:Jetson-AGX
Online CPUs: 0-7
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0
cpu1: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0
cpu2: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0
cpu3: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0
cpu4: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0
cpu5: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0
cpu6: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0
cpu7: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0
GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000
EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1
Fan: speed=255
NV Power Mode: MAXN

[ShaneCCC]

What’s the frame rate?

[kuskov.stanislav]

26 fps.
When only 3 camera work fps is 30 for each.
When I connect fourth fps decrease to 26

[Honey_Patouceul]

@kuskov.stanislav

8 cameras leopard imx390.

This might not be related, but you may check this.

[KevinGong]

Hi. Everyone.

Thank you all for helping debugging on this issue. I just saw this issue. Let me check if I can help.

1. I will test this issue on my side to see if this issue can re-produced stable.

2. I don’t think this issue is same with post " Decreased Framerate - Jetson Xavier Board - 4K video capture with (6) cameras". Since in that case. Customer using a special resolution which not defined in firmware to recording video. That may require Xavier to cost extra resource for resize. We believe this is the reason for frame drop at that case. When we test with the default resolution (even the full resolution). The frame rate can keep up than 28fps.

3. This driver using 4 MIPI lane setting in MAX9296 chip. I need to confirm with our team how we config the 4 MIPI lane setting. Since we use 2 x 1.2G or 4 x 700M for MAX9296 setting before (on other platform). If this driver using 4 x 700M setting. Then the bandwidth for each camera (1.4G/camera) is actually less than 2 x 1.2G (setting which in the other post Honey_Patouceul mentioned).

4. I will actual test here and check with our team to make sure the bandwidth is enough. We will try to address the issue and keep update here.

Kevin Gong (leopard Imaging Inc).

[kuskov.stanislav]

Let me give You more information about the environment.
I’m using Leopard Imaging IMX390 cameras, Jetson AXG Cavier Dev kit with GMSL camera board JCB004 for 8 cameras.
Jetpack 4.3 with drivers for GMSL camera board.
I will expect Your test results. Thank You!

[KevinGong]

Hi. Stanislav.

Thank you for your detail information.

I did some quick test this morning. But I don’t think I can reproduce this issue. I use the latest driver (Dropbox - IMX390-GMSL2-DUAL_R32.4.3_Xavier_NV-Quad_20200921_Driver_Guide.pdf - Simplify your life). I use argus to run 6 cameras at same time. I can get stable 30fps output. I use KPI in Argus to check the frame rate. Please check my result below.

Please make sure to switch the power mode to max (using command “sudo nvpmodel -m 0”).

Kevin Gong (Leopard Imaging Inc)

[kuskov.stanislav]

Hi Kevin! Thank for Your answer and manual!
Would you know about [this] (L4T R32.4.3 Breaks argus_camera and causes segmentation fault (SIGSEGV) - #9 by JerryChang) problem with libargus.
I’m tried to use jetpack 4.4 earlier and has same problem.
Did You use another version of multimedia api?

PS. I would try some experiments with Your manual later

[ShaneCCC]

Have a check below link for all patch for camera.

https://elinux.org/Jetson/L4T/r32.4.x_patches

[KevinGong]

What error message do you get? The version which I use to test is: Dropbox - jetson_multimedia_api_R32.4.3.tar.gz - Simplify your life.

You can also check Shane’s link for all released patches.

Thank you.

[kuskov.stanislav]

Just jetson reboots when I’m would use libargus. I’m write here about it. This problem described in Shane’s link

[kuskov.stanislav]

Hi Kevin! I’ve tried this manual but the cameras did not start.
I got to point 7 of “Setup Procedure” but “nvgstcapture-1.0” cmd write me “No cameras available”. I can not find cameras into /dev directory too.

[KevinGong]

Hi. Stanislav.

If you can’t find any video device. Then you should get some wrong operation or error when you flash the device tree (dtb file) which is step 2 & 3 in the instruction. Please make sure you can flash the device tree correctly.
BTW. Do you use our adapter board? You need to use our adapter board & camera to make this driver work.

Thank you.