[Solved]GPUDirect RDMA free_callback() questions

From Nvidia’s GPUDirect RDMA pagehttp://docs.nvidia.com/cuda/gpudirect-rdma/index.html nvidia_p2p_get_pages() requires a call to another function to “be invoked if the pages underlying the virtual address range are freed implicitly. Cannot be NULL.” I don’t really know what this means and am hoping someone can provide a better explanation. Also the example for this free_callback() is:

void free_callback(void *data)
    my_state *state = data;

I am new to kernel programming and am wondering what actual function could be used in place of “wait_for_pending_transfers(state)”? I believe this call is crashing my kernel when my device driver grabs GPU memory as can be seen in this segment of a vmcore-dmesg.txt crash report:

nvidia 0000:82:00.0: irq 135 for MSI/MSI-X
ioctl_dev: mapping GPU page…
ioctl_dev: BOOM! pinned 65536 bytes of GPU memory…
releasing GPU page…
BUG: unable to handle kernel NULL pointer dereference at 0000000000000034
IP: [] nvidia_p2p_free_page_table+0x39/0x70 [nvidia]
Oops: 0000 [#1] SMP

Thanks for your help. If you think a glimpse of my kernel code would be useful, I can post that too.


seems we have stumbled upon the cuda joker

“am wondering what actual function could be used in place of “wait_for_pending_transfers(state)””

have you written a function already, such that you think it is crashing your kernel?

seeing that state is set by data, i am actually more interested in data; are you certain that ‘data’ is not crashing your kernel?
i presume data is an input; so, what sets ‘data’?

thanks, you were right, I was not passing the pointer to data correctly in nvidia_p2p_get_pages() and this was causing nvidia_p2p_free_page_table() to access an invalid pointer. Still not clear on what wait_for_pending_transfers(state) means.

This is outside of my area of expertise, but the way I read it, wait_for_pending_transfers(state) ensures that a mapping is not removed while there are still DMA transfers in flight that may be using that very mapping. So presumably this would involve checking your device (e.g. camera, high-speed storage) to see whether there are still active or queued DMA reguests on its end, and waiting until those have finished.