Invalid Managed Memory Access

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

hi,

I’m having a “Invalid Managed Memory Access” issue. The application uses gstreamer, with appsrc as head of the pipeline. Something like:

appsrc ! tee ! queue ! omxh264enc ! …

We push cudaMallocManaged video frame buffer into appsrc, along with the free-buffer callback.

static void free_cuda_buffer_cb(gpointer buf)
{
    checkCudaErrors(cudaFree(buf));
}

void push_buffer(unsigned char * data)
{
    GstBuffer *buf;
    buf = gst_buffer_new_wrapped_full(
        GST_MEMORY_FLAG_READONLY,
        (gpointer)data,
        width * height * 3 / 2,
        0,
        width * height * 3 / 2,
        (gpointer)data,
        free_cuda_buffer_cb
        );

    if (GST_FLOW_OK != gst_app_src_push_buffer (GST_APP_SRC (src), buf))
    {
        g_printerr("appsrc push buffer error\n");
    }
}

Then, when running the application, it crashed into a memcpy inside the gstreamer’s queue thread (a Segmentation fault or Bus Error). By cuda-gdb, I got the

cuda frame buf pushed: 0x702e000
[Switching to Thread 0x7fa97ff1e0 (LWP 2139)]

Breakpoint 1, gst_buffer_extract (buffer=0x7f6c003080, offset=0, dest=<optimized out>, size=11059200) at gstbuffer.c:1891
1891          left -= tocopy;
(cuda-gdb) p info.data
$2 = (@managed guint8 *) 0x10702e000 ""
(cuda-gdb) p ptr
$3 = (guint8 *) 0x7f8ed73010 ""
(cuda-gdb) p tocopy
$4 = 11059200
(cuda-gdb) n
1892          ptr += tocopy;
(cuda-gdb) n
1881      for (i = 0; i < len && left > 0; i++) {
(cuda-gdb) n
1893          offset = 0;
(cuda-gdb) n
1890          memcpy (ptr, (guint8 *) info.data + offset, tocopy);
(cuda-gdb) p info.data
$6 = (@managed guint8 *) 0x10702e000 ""
(cuda-gdb) p ptr
$7 = (guint8 *) 0x7f8ed73010 ""
(cuda-gdb) n

Program received signal CUDA_EXCEPTION_15, Invalid Managed Memory Access.
0x0000007fb78d1b58 in memcpy () from /lib/aarch64-linux-gnu/libc.so.6

The crash always happen if it’s the first time run after a fresh system reboot. After the first time, it has a chance to run normally.

Any suggestions on how to debug further?

Attach gstbuffer.c

Thanks.

gstbuffer.c (69.9 KB)

2

Hi,

A managed memory is set to global if no CUDA stream attached.
Launching a kernel makes all the global managed memory inaccessible to CPU, which leads to CPU reading fail.

Please find this comment for more information:
[url]https://devtalk.nvidia.com/default/topic/1030988/jetson-tx2/unified-memory-and-concurrent-c-objects/post/5251476/#5251476[/url]

Thanks.