Memory warning logs when using 9 gstreamer commands with nvdec

Robotics & Edge Computing Jetson Systems Jetson Orin NX
, ,
1

I am using an Orin Nx on a custom carrier board.

When opening 9 or more gstreamer processes with the nvdec element I get the following warning logs.

I have applied the patches in this form post to L4T r36.4.4

Warning logs

[ 2750.808733] tegra-vic 15340000.vic: nvv4l2decoder0:: all memory contexts are busy, waiting
[ 2750.810253] tegra-nvdec 15480000.nvdec: queue0:src: all memory contexts are busy, waiting
[ 2751.234759] tegra-nvdec 15480000.nvdec: queue0:src: all memory contexts are busy, waiting
[ 2751.237843] tegra-nvdec 15480000.nvdec: queue0:src: all memory contexts are busy, waiting

This is the command that I ran 9+ times

nohup env DISPLAY=:0 gst-launch-1.0 -e filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink &

I also tried the nvenc and the same thing happened when I run more that 8 gstreamer processes that encode video.

I want to know how to get rid of this warning message.

  1. Can I increase the memory to the nvdec to stop this message from appearing?
  2. Can I disable this message?

I am also in the process of installing the patch on the devkit.

Thank You

When using gst-launch-1.0 nvarguscamerasrc, an error occurs when trying to start streaming from the 8th camera
2

*** Please note that this reply is generated by LLM automatically ***

The warning message you’re encountering is related to the NVIDIA Video Decoder (NVDEC) limitation on the Jetson Orin NX. By default, NVDEC is limited to decoding 8 streams simultaneously. When you try to open more than 8 gstreamer processes with the nvdec element, you receive a warning message indicating that the maximum number of streams has been reached.

To resolve this issue, you have a few options:

  1. Apply the patches: As you mentioned, you’ve already applied the patches to L4T r34.44. However, you need to rebuild the kernel after applying the patches. Make sure to follow the instructions carefully and rebuild the kernel to ensure the patches are properly applied.
  2. Increase the memory allocation for NVDEC: You can try increasing the memory allocation for NVDEC by modifying the kernel parameters. However, this might not completely eliminate the warning message, as the limitation is not solely due to memory constraints.
  3. Disable the warning message: You can disable the warning message by modifying the kernel code. However, this is not recommended, as the warning message is intended to inform you of the potential performance impact of exceeding the recommended number of streams.
  4. Use a different decoder: If possible, you can try using a different decoder, such as the NVIDIA Video Encoder (NVENC), which might not have the same limitations as NVDEC. However, you’ve already mentioned that you’ve tried NVENC and encountered the same issue.
  5. Optimize your gstreamer pipeline: You can try optimizing your gstreamer pipeline to reduce the number of streams being decoded simultaneously. This might involve using a more efficient decoding algorithm, reducing the resolution or frame rate of the video, or using a different container format.

To increase the memory allocation for NVDEC, you can try modifying the kernel parameters. Specifically, you can try increasing the value of the nvdec_mem_size parameter. However, this might require rebuilding the kernel, and the exact steps may vary depending on your specific setup.

To disable the warning message, you can try modifying the kernel code. However, this is not recommended, as the warning message is intended to inform you of the potential performance impact of exceeding the recommended number of streams.

In summary, the best course of action would be to apply the patches and rebuild the kernel. If you’re still encountering issues, you can try optimizing your gstreamer pipeline or exploring alternative decoders. Disabling the warning message is not recommended, as it may mask potential performance issues.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

5

The nvdec_mem_size parameter does not exist in the l4t kernel. Can you help me find the equivalent variable in code to change?

6

Hi,
It looks the patches are not well applied. Please ensure you follow the steps in
Kernel Customization — NVIDIA Jetson Linux Developer Guide

7

I am trying to install the patches on a devkit and I read that guide and did the following in order.

  1. Extract the kernel and the NVIDIA out-of-tree modules source files to the l4t directory
  2. Build the kernel using this command
$ export CROSS_COMPILE=/home/test/projects/nvidia/toolchain_l4t/aarch64--glibc--stable-2022.08-1/bin/aarch64-buildroot-linux-gnu-/bin/aarch64-buildroot-linux-gnu-
$ make -C kernel
  1. Install the kernel and copy the Image file
  2. cd to the nvidia_oot/ directory
  3. Apply the patches using these commands
    patch -p1 < ../0001-gpu-host1x-Memory-context-stealing.patch
    patch -p1 < ../0002-gpu-host1x-When-out-of-memory-contexts-wait-for-free.patch
  4. Set environment variables for building the nvidia_oot modules.
  5. Build the nvidia_oot modules using this command make modules
  6. Install the nvidia_oot modules
  7. run sudo ./tools/l4t_update_initrd.sh
  8. Build the dtb
  9. Install the dtb
  10. Create default user
  11. Flash the dev kit.

However on boot i get the following error logs in the boot log.

[   32.267370] nvidia_modeset: Unknown symbol host1x_syncpt_read (err -22)
[   32.267486] nvidia_modeset: disagrees about version of symbol host1x_syncpt_id
[   32.267488] nvidia_modeset: Unknown symbol host1x_syncpt_id (err -22)
[   32.341730] nvidia_modeset: disagrees about version of symbol host1x_syncpt_alloc
[   32.341739] nvidia_modeset: Unknown symbol host1x_syncpt_alloc (err -22)
[   32.341892] nvidia_modeset: disagrees about version of symbol host1x_syncpt_get_by_id_noref
[   32.341894] nvidia_modeset: Unknown symbol host1x_syncpt_get_by_id_noref (err -22)
[   32.341907] nvidia_modeset: disagrees about version of symbol host1x_syncpt_put
[   32.341908] nvidia_modeset: Unknown symbol host1x_syncpt_put (err -22)
[   32.341921] nvidia_modeset: disagrees about version of symbol host1x_syncpt_read
[   32.341922] nvidia_modeset: Unknown symbol host1x_syncpt_read (err -22)
[   32.342049] nvidia_modeset: disagrees about version of symbol host1x_syncpt_id
[   32.342050] nvidia_modeset: Unknown symbol host1x_syncpt_id (err -22)

Can you provide advice on how to install the patches correctly?

8

I still need your help with installing the patches correctly on the devkit, however I tested the nvenc gstreamer element on our custom carrier board to see what happens.

On our custom carrier board with the patch installed I also get a flood of constant log messages(as seen below) after running the 9th process of this gstreamer command that uses nvenc element.

Gstreamer Command

nohup gst-launch-1.0 videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink &

Log message

[ 1075.385222] tegra-nvenc 154c0000.nvenc: V4L2_EncThread: all memory contexts are busy, waiting
[ 1075.398240] tegra-nvenc 154c0000.nvenc: V4L2_EncThread: all memory contexts are busy, waiting
[ 1075.402366] tegra-nvenc 154c0000.nvenc: V4L2_EncThread: all memory contexts are busy, waiting
[ 1075.419742] tegra-vic 15340000.vic: V4L2_EncThread: all memory contexts are busy, waiting
[ 1075.420387] tegra-nvenc 154c0000.nvenc: V4L2_EncThread: all memory contexts are busy, waiting
[ 1075.422938] tegra-vic 15340000.vic: videotestsrc0:s: all memory contexts are busy, waiting
[ 1075.444818] tegra-vic 15340000.vic: V4L2_EncThread: all memory contexts are busy, waiting
[ 1075.444922] tegra-vic 15340000.vic: videotestsrc0:s: all memory contexts are busy, waiting
9

Hi,
The print is in nvidia-oot:

nvidia-oot/drivers/gpu/host1x/context.c

You may change it to verbose prints and rebuilt kernel.

10

Sorry are you talking about the log messages i get on the devkit when trying to install the patches?

or are you talking about testing the nvenc on the custom carrier board?

11

Hi,
The warning print is in kernel. If you would like to remove it, can revise the file to remove the line, and rebuild kernel.

Hardware engine has limited instances and if all are occupied, it gets printed. You have 9 instances which are more than hardware instances, so the print is triggered.

12

Sorry, I am a bit confused by your response.

When I run the commands below that start 9 video streams using one process I do not get the log messages in my previous comments.

  1. How can the hardware engine run a single multi-threaded process with more than 8 video streams but can’t run more than 8 processes each with 1 video stream?

  2. In that form post with the patch it seems like the problem was software related. Is the number of video streaming processes a hardware limitation or a software limitation?

  3. Is the hardware limitation not with the decoder and encoder core but with other hardware?

Any extra information and explanation would be greatly appreciated.
Thank you

Command for decoding 9 video streams in one process

DISPLAY=:0 gst-launch-1.0 -e filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink \
filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink \
filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink \
filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink \
filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink \
filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink \
filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink \
filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink \
filesrc location=/home/test/test_30M_h265_BL_FullHD_30fps_4M.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=720, height=480 ! glimagesink

Command for encoding 9 video streams in one process

gst-launch-1.0 videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink \
videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink \
videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink \
videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink \
videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink \
videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink \
videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink \
videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink \
videotestsrc ! video/x-raw,width=1280,height=720,framerate=30/1 ! nvvidconv ! nvv4l2h265enc profile=0 bitrate=4096000 qp-range=32,51:32,51:32,51 idrinterval=15 iframeinterval=15 insert-sps-pps=true control-rate=0 ! h265parse ! queue ! fakesink
13

Hi,
You may take a look at the blurring part in Jetson AGX Orin FAQ

After applying the patches, when all hardware resource is in use and some requests are waiting, the warning is printed. So the print is expected since you have more processes than hardware resource. You may modify kernel code to eliminate the print.

14

So even with the patch installed there is no way to increase the memory context for the nvdec and nvenc because it is a hardware limitiation?

15

Hi

You are correct. It is a hardware constraint.

16

Ok,
Thank you for the help and information.

18

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.