Kernel panic due to incompatibility between freeCudaHost and free_pages

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
1

Hi there! This topic is a continuation of this topic.

So, I tested the code for 16 256K buffers, and it’s true: NO ISSUES. But, when I tried to run with 16 256M buffers, the error returned. The same kernel panic, but, if I comment the cudaFreeHost, the program runs as expected. Than:

  1. 16x 256K + cudaFreeHost = OK
  2. 16x 256M - cudaFreeHost = OK
  3. 16x 256M + cudaFreeHost = FAIL

I will share my code here. There is a run script in the root directory.
To change the number and size of the buffers: change the L43 - main_cuda_buffer.cu. To comment the cudaFreeHost: change the L101 - main_cuda_buffer.cu.
driver.zip (6.2 KB)

Thanks a lot!

3

Hi,

Thanks for the update and source.
We will test it and provide more info to you later.

Thanks.

5

Thanks AastaLLL! Looking forward to hearing from you.

6

Hi there! Any news?

Thanks.

7

Hi,

We are still discussing this issue internally.
In our testing, the system reboots when running the app (the cause of kernel panic).

However, we are not able to collect related crash logs on the kernel or uart log.
Do you have the error log related to the panic so we can share it with our internal team?

Thanks.

8

Yes, I’m facing the same behavior. The log that I achieved was the attached in the previous ticket. However, it’s here the debug UART output: debug.txt (6.6 KB)
.

9

Hi,

Thanks for the info.
We have shared the logs with our internal team and will let you know once we have more info about this issue.

10

Hi,

Thanks for your patience. This is what we got from our internal team:

I see at least an issue in their kernel module. They are leaking dma descriptors. They miss a call to nvidia_p2p_free_dma_mapping() in p2p_free_callback(), before nvidia_p2p_free_page_table():

static void p2p_free_callback(void *data)
{
struct cuda_buffer *cubuffer = data;
nvidia_p2p_free_page_table(cubuffer->dma.page_table);
kfree(cubuffer->dma.page_table);
}

[ 935.336856] WARNING: CPU: 2 PID: 1964 at kernel/fork.c:709 __mmdrop+0x148/0x190
[ 935.337361] —[ end trace fa7822d9e3a12676 ]—
[ 936.342353] WARNING: CPU: 2 PID: 1964 at kernel/fork.c:710 __mmdrop+0x150/0x190
[ 936.342707] —[ end trace fa7822d9e3a12677 ]—
[ 936.443254] BUG: Bad rss-counter state mm:0000000045a8f79e type:MM_FILEPAGES val:253732
[ 936.443259] BUG: Bad rss-counter state mm:0000000045a8f79e type:MM_ANONPAGES val:2052
[ 936.443260] BUG: non-zero pgtables_bytes on freeing mm: 2228224
[ 936.443311] Unable to handle kernel paging request at virtual address dead000000000100

For this issue, more reviewing of their kernel module should be done.
For example, they are not cleaning up the page_table object when releasing the inode.

int buffer_release(struct inode* inode, struct file* filp)
{
struct cubuffer *cubuffer = filp->private_data;
kfree(cubuffer);
return 0;
}

Thanks.