Tracking Memory Leaks

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

Hi, I have an Jetson Nano application which runs fine for 20-30 minutes and then on the following function call:

ret = NvBufferTransform(src_fd, dest_fd, &transform_params);

All arguments (src_fd, dest_fd, transform_params) are identical when the application runs, and when it fails. So, assumptions:

  • I assume that NvBufferTransform is thread safe (used in 4 threads, one per camera).

  • I assume it’s an application memory leak which writes over memory which is used by NvBufferTransform. So, I’m thinking that if I find the memory associated with fd, obtain an application memory map, I can start to detect corruption.

So, what does fd map to? a DMA buffer, or is there any structures created in memory which are allocated in NvBufferCreateEx()? How can I inspect this memory, or create a clone to detect for corruption?

In general, is there a good manual, reference material which details the operation?

3

Hi,
The memory is allocated in DRAM as the layout that hardware engines can access it. You may run sudo tegrastats to check if used memory is increasing. And run top to check used memory of the process.

NvBufferTransform() is thread safe and for having optimal throughput, you can create individual NvBufferSession in each thread.

4

Hi DaneLLL,

Thanks for the reply. I’ve tried the above and still no joy. I’ve run TOP on the process and the memory usage is constant.

So I’ve tried to isolate the issue surrounding NvBufferTransform using the following code (this code is extracted from the main application which runs alongside it). Essentially I create two NvBuffer which are completely independent from the rest of the application: Their purpose is only to test the operation of the NvBufferTransform.

// Initialise destination buffer
NvBufferCreateParams create_params;
memset(&create_params, 0, sizeof(create_params));
create_params.payloadType = NvBufferPayload_SurfArray;
create_params.width = tensor_engine->GetConfig().inf_width;
create_params.height = tensor_engine->GetConfig().inf_height;
create_params.layout = NvBufferLayout_Pitch;
create_params.colorFormat = NvBufferColorFormat_ABGR32;
create_params.nvbuf_tag = NvBufferTag_VIDEO_CONVERT;

if (NvBufferCreateEx(&fd, &create_params))
{
std::cout << “NvBufferCreateEx() FAIL” << std::endl;
std::exit(1);
}

if (NvBufferCreateEx(&fd1, &create_params))
{
std::cout << “NvBufferCreateEx() FAIL” << std::endl;
std::exit(1);
}

while (running)
{
int result = NvBufferTransform(fd1, fd, &transform_params);
if (result)
{
std::cout << “NvBufferTransform() FAIL” << std::endl;
std::exit(1);
}
}

This still fails after approximately 20 minutes. Hopefully(!) there’s no issue within the NvBufferTransform function, so there must be an adjacent memory/cpu issue which is causing it to fail.

So, where does the memory pointed to by fd reside, and how can I determine memory structure which are adjacent to it. I asusme we must have a small leak somewhere which starts overwriting the FD table.

Any help much appreciated. Richard.