Jetson NX high CPU syscall usage on nvargus kernel 4.9 vs 5.10

[ademsah13]

We noticed this problem after upgrading our NX to jetpack 5 and we took the time to investigate this behaviour.

We did some comparison tests between two Xavier NX SoCs: first is running with old,latest jetpack 4.6 (with kernel 4.9.253-tegra) and second is running jetpack 5.1.2 (with kernel 5.10.120-tegra), both on the exact same carrier board and cameras.

In our setup, we use two Framos IMX464 sensor cameras running both on 4 lanes because we need the high FPS.

In our side by side tests, for power mode 5 (10W_4CORE) the old driver and kernel use around 47% CPU, while the new one uses around 80% CPU, mostly in syscalls (red part of the bars).

The same behaviour can be seen with power mode 8 (20W_6CORE), the CPU usage is 37% vs 74%, please see attached screenshots.

The command to start the cameras that we used are those:

$ gst-launch-1.0 --eos-on-shutdown nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM), width=2712, height=1538, framerate=90/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)I420' ! queue leaky=downstream max-size-buffers=30 ! nvv4l2h264enc maxperf-enable=true bitrate=44000000 profile=4 insert-aud=1 insert-sps-pps=1 insert-vui=true ! 'video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high, level=(string)3.1' ! h264parse config-interval=-1 ! queue leaky=downstream max-size-buffers=200 ! mp4mux name=mux ! filesink location=test.mp4

Please note that we even tried to upgrade the old jetpack manually to Ubuntu 20, keeping the 4.9 kernel, just to have the same gstreamer version 1.16.3, but the problem is still reproducible.

I know the ISP pipeline is adding load, but it can’t add double the load and also it is not reasonable to say that the same hardware is half as powerful as before, just by upgrading the kernel and camera driver, right?

Comparison CPU time in syscalls between the two kernels, BEFORE and AFTER, running on the same hardware with the same workload:

I can upload the clickable SVG flame graph if it helps.

Thank you in advance!

[ademsah13]

here is before, with the old kernel:

Also CPU load comparison:

[ShaneCCC]

Does others source element like videotestsrc have the same result?

[ademsah13]

I tried with videotestsrc instead of the nvarguscamera and I can see the same difference in CPU syscals, just not so dramatic, since videotestsrc uses a lot of CPU as well:

Can you please trace this internally too?

[ShaneCCC]

Could you please also capture the nvargus CPU load information when running the workload. You can use systemd-cgtop. Provide an example here.

$ systemd-cgtop
Control Group                                              Tasks   %CPU   Memory  Input/s [0/1813]
/                                                            482  291.4     3.1G       0B    47.0K
system.slice                                                 188  233.7   594.1M        -        -
system.slice/nvargus-daemon.service                           99  233.7   332.0M        -        -
user.slice                                                    59   19.1   171.7M        -        -
user.slice/user-1000.slice           

[ShaneCCC]

The r35.6 release have improved CPU usage for argus.

Camera improvements
Updated the Argus library, resulting in up to a 40% reduction in CPU load for CSI camera capture with Argus cameras.