With the use of NVIDIA ISP, Jetson Thor cannot operate 16 cameras properly

Robotics & Edge Computing Jetson Systems Jetson Thor
1

HI,NVIDIA

DRIVE OS Version: Jetpack 7.0

I used Thor to connect 16 GMSL cameras. The cameras have a resolution of 1920*1080 and a frame rate of 60 with raw10 format.

Using the following command to print the frame rate will enable you to successfully display the frame rates of 16 channels.

v4l2-ctl -V --set-fmt-video=width=1920,height=1080 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000000 -d /dev/video0

However, when I use the “argus_camera -d 0” command to turn on 16 channels, there will be random instances where the cameras get stuck. In such cases, I need to use the command again to turn them on.

I attempted to activate using the following instructions, but when I tried to turn on more than ten cameras, the following error occurred.

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),framerate=60/1,format=NV12' ! nvvidconv ! xvimagesink

[  466.824931] tegra-vic 8188050000.vic: consumer_thread: all memory contexts are busy, waiting
[  466.835133] tegra-vic 8188050000.vic: consumer_thread: all memory contexts are busy, waiting
[  466.846602] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.849679] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.865418] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.866054] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.874566] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.886876] tegra-vic 8188050000.vic: consumer_thread: all memory contexts are busy, waiting
[  466.890937] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.899192] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.908377] tegra-vic 8188050000.vic: consumer_thread: all memory contexts are busy, waiting
[  466.918711] tegra-vic 8188050000.vic: consumer_thread: all memory contexts are busy, waiting
[  466.924251] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.932778] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.942964] tegra-vic 8188050000.vic: consumer_thread: all memory contexts are busy, waiting
[  466.951078] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.966470] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.966567] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.973852] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.981817] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  466.991411] tegra-vic 8188050000.vic: consumer_thread: all memory contexts are busy, waiting
[  466.998626] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  467.006569] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  467.015055] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  467.025144] tegra-vic 8188050000.vic: consumer_thread: all memory contexts are busy, waiting
[  467.046465] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting
[  467.047366] tegra-vic 8188050000.vic: argus_thread: all memory contexts are busy, waiting

Is this the bottleneck for Thor, or are there any areas for optimization?Could you give me some solutions?

Thanks

5

Hi,
Please apply this solution and see if it works:

Making sure you're not a bot!
[Multimedia]Performance of VIC drops significantly when running 8+ processes

If the issue persists, it may exceed constraint of hardware VIC engine. You can run $ sudo tegrastats to profile loading of VIN engine. See if it keeps in 99%, 100%.

And would suggest upgrade to Jetpack 7.1 r38.4.

6

HI,

Thank you very much for your reply.

I referred to the link you provided: Nvvidconv slow in multi-process: why does performance collapse when using multiple GStreamer processes? - #12 by louity

and changed “iommu-map” to 32 instances.

iommu-map = <0x00 0x05 0x3001 0x01 0x01 0x05 0x3002 0x01 0x02 0x05 0x3003 0x01 0x03 0x05 0x3004 0x01 0x04 0x05 0x3005 0x01 0x05 0x05 0x3006 0x01 0x06 0x05 0x3007 0x01 0x07 0x05 0x3008 0x01 0x08 0x05 0x3009 0x01 0x09 0x05 0x300a 0x01 0x0a 0x05 0x300b 0x01 0x0b 0x05 0x300c 0x01 0x0c 0x05 0x300d 0x01 0x0d 0x05 0x300e 0x01 0x0e 0x05 0x300f 0x01 0x0f 0x05 0x3010 0x01 0x10 0x05 0x3011 0x01 0x11 0x05 0x3012 0x01 0x12 0x05 0x3013 0x01 0x13 0x05 0x3014 0x01 0x14 0x05 0x3015 0x01 0x15 0x05 0x3016 0x01 0x16 0x05 0x3017 0x01 0x17 0x05 0x3018 0x01 0x18 0x05 0x3019 0x01 0x19 0x05 0x301a 0x01 0x1a 0x05 0x301b 0x01 0x1b 0x05 0x301c 0x01 0x1c 0x05 0x301d 0x01 0x1d 0x05 0x301e 0x01 0x1e 0x05 0x301f 0x01 0x1f 0x05 0x3020 0x01>;

The situation has improved somewhat. However, when I attempted to use my hand to block the camera’s AE function, it got stuck. Could this be because the hardware has reached its limit?

7

Hi,

Please check if this occurs when launching all 16 cameras. Or it is also observed when you launch 4 or 8 cameras.

8

It only occurs when all 16 cameras are turned on simultaneously.

9

Boost the clocks to try.

sudo jetson_clocks
sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
10

I have tried to speed up the clock, but it didn’t work.

11

Could you check the temperture by tegrastats if any unnormally.

12

Do you mean that the temperature of the SOC (System On Chip) or the camera is overheating?Actually, when I checked the system resource usage, it wasn’t very high, and the camera temperature wasn’t high either. We verified its durability. However, when using Argus to turn on and adjusting the automatic exposure, several camera images would get stuck. To fix this, we had to close the Argus display interface and re-execute the command. Clearly, this indicates that there is a problem with the ISP of Argus. If it were the camera or any other issue, theoretically, it would not be possible to restart the device.

13

Hi,
It may hit constraint of VIC engine. Please check $ sudo tegrastats. Or may try smaller resolutions.

14

Please also record the daemon log by below command while run the cameras

journalctl -u nvargus-daemon -f 2>&1 | tee log.txt