DeepStream Inference Terminated Due to Buffer Pool Size Error

• DeepStream Version = 6.4

I am seeking assistance with using NVIDIA DeepStream to perform inference on 8 sources, each a minimum of 10 hours long. I am utilizing the Kafka protocol to read the output messages. However, the inference process is terminated after an hour with the following error:

gstnvtracker: Unable to acquire a user meta buffer. Try increasing user-meta-pool-size

The documentation suggests that this issue occurs when downstream plugins take too long to release buffers, causing the buffer pool to become empty. It recommends increasing the pool size from the default 32 to a larger value like 64.

Will I encounter the same error after setting the pool size to 64? Additionally, where should the parameter user-meta-pool-size be added?

Thank you in advance for your help.

inception

You can try that. Just set the user-meta-pool-size in your config file or set that in the source code as the following.

  g_object_set (G_OBJECT (tracker), "user-meta-pool-size",
      64, NULL);

Please provide complete information as applicable to your setup. Thanks
Hardware Platform (Jetson / GPU)
DeepStream Version
JetPack Version (valid for Jetson only)
TensorRT Version
NVIDIA GPU Driver Version (valid for GPU only)

Hardware Platform (Jetson / GPU) - Nvidia T1000 8GB
TensorRT Version - 8.6.1.6
NVIDIA GPU Driver Version (valid for GPU only) - 535.183.01

Thank you for your response. I increased the user-meta-pool-size to 64 as suggested, but unfortunately, the inference process still terminated after about an hour of inference data.

Considering that 8 sources maybe the issue, I tested for 1 source which was 12 hours long, the same problem occurred. It did not give error, it just killed the inferencing.

Are there any other potential solutions or adjustments that I can make to address this issue?

Thanks

Could you attach your pipeline and the whole log with GST_DEBUG=3?
You can run the following command to dump the log into a file.

GST_DEBUG=3   <your_command>   2>&1|tee log.txt

Also you can also try that on our latest version DS7.0 to check whether there is still the same issue.

I am using the code from the following GitHub repository to run the inferencing:

I have run the command with GST_DEBUG=3 and attached the log file as requested. Please find the log file attached.

I have not yet tried the latest version DS7.0. If the issue persists, I will consider upgrading to check if that resolves the problem.
log.txt (11.7 MB)

After some testing, it was found out that, the RAM was increasing continuously until it reached max utilization - 32 GB and killed the inferencing.

How can we restrict the utilization or is it possible to handle this issue in another manner?


gpu

OK. This is most likely a memory leak problem. Have you made any changes to the demo code?

No, I haven’t made any changes.

I have tried our demo on my board. The memory has not grown significantly. Are you sure you didn’t change anything in our project?

Could you try not to run your code, just run our demo without any change?

Hey, thanks for the reply.

I ran the initial code from the GitHub repository you provided, and I observed that the memory usage on the CPU increases slowly over time until it finally consumes all available memory. Could this be related to the error I’m encountering? Do you have any insights into what might be causing this issue?

Memory Leak in the Pipeline
Improper Buffer Handling

Where did these 2 lines print? There is no such lines in the log file you provided.
Could you update your DeepStream to our latest version 7.0 and try again?

No they are not part of the logs, I only suggested that, those 2 cases could be the issues I’m facing.

Sure, I’ll run it in DS7.0 and let you know.
Thanks.

I have tested it on DS7.0 and the case remains the same. Here are the config files that I made changes to and the snap of memory reaching it’s limit. Please let me know if any other document is required.
image(6)
dstest.txt (6.7 KB)
nvdanalytics.txt (8.9 KB)

OK. May I know what tools are you using to check the memory? Could you just use our demo video instead of 1.mp4 in your config file to reproduce this problem and attach the specific statistics, like the memory growth per hour?