VPI "Fatal assertion error" when trying to use CUDA-wrapped unified memory in remap on VIC

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hello.

When we are trying to use VPI CUDA-wrapped memory buffer allocated by CUDA unified memory subsystem in VPI remap call, it fails with unspecified “Fatal assertion error” without any further details.

Reproduction steps:

  1. Allocate memory with cudaMallocManaged
  2. Associate that memory with current CUDA stream
  3. Wrap memory to VPIImage via vpiImageCreateCUDAMemWrapper
  4. Allocate second VPI image via ordinary vpiImageCreate call.
  5. Do remapping operation from image (4) to image (3)

Does not reproduce when you wrap managed memory as a host memory, but introduces significant performance degradation on copying data back and forth.
Does not reproduce when you wrap managed memory as a CUDA memory and execute remap on CUDA itself instead of VIC.

Version: VPI 1.1.15 from JetPack 4.6, SHA256 of libnvvpi.so file is bc5c57d80e…

Program logs:

[2022-01-09 21:44:17.060] [app] [info] Initialization completed
Fatal assertion error

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

GDB backtrace:

[New Thread 0x7f4c37ada0 (LWP 452)]
[Switching to Thread 0x7f6f00cda0 (LWP 396)]

Thread 48 "Work queue" hit Breakpoint 2, __GI_abort () at abort.c:49
49	abort.c: No such file or directory.
(gdb) bt
#0  __GI_abort () at abort.c:49
#1  0x0000007fb52497a8 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#2  0x0000007fb4c4ce3c in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#3  0x0000007fb4c61e54 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#4  0x0000007fb4c6215c in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#5  0x0000007fb4c52ef0 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#6  0x0000007fb4c53154 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#7  0x0000007fb4d36c24 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#8  0x0000007fb4d36eac in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#9  0x0000007fb4d2e2b8 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#10 0x0000007fb4d4bab4 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#11 0x0000007fb4d492e8 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#12 0x0000007fb5348234 in ?? () from /opt/nvidia/vpi1/lib64/libnvvpi.so.1
#13 0x0000007fb41c6088 in start_thread (arg=0x5557beeeef) at pthread_create.c:463

(gdb) info sharedlib
From                To                  Syms Read   Shared Object Library
0x0000007fb7fd2040  0x0000007fb7fe9188  Yes         /lib/ld-linux-aarch64.so.1
0x0000007fb7faaed0  0x0000007fb7facc2c  Yes         /lib/aarch64-linux-gnu/libdl.so.2
0x0000007fb7f94e90  0x0000007fb7f97f64  Yes         /lib/aarch64-linux-gnu/librt.so.1
0x0000007fb7f258f0  0x0000007fb7f4556c  Yes         /usr/lib/aarch64-linux-gnu/libtcmalloc.so.4
0x0000007fb7d55c10  0x0000007fb7e8f39c  Yes         /usr/lib/aarch64-linux-gnu/libprotobuf.so.10
0x0000007fb7c894f0  0x0000007fb7c927e8  Yes         /usr/lib/libboost_fiber.so.1.77.0
0x0000007fb4bde130  0x0000007fb5391950  Yes         /opt/nvidia/vpi1/lib64/libnvvpi.so.1
0x0000007fb4910b60  0x0000007fb4915ab0  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0
0x0000007fb47fbe00  0x0000007fb4801be0  Yes         /usr/lib/aarch64-linux-gnu/tegra/libv4l2.so.0
0x0000007fb421bae0  0x0000007fb421be58  Yes         /usr/lib/libboost_context.so.1.77.0
0x0000007fb41f3110  0x0000007fb4204ab8  Yes         /usr/lib/libboost_filesystem.so.1.77.0
0x0000007fb41c4690  0x0000007fb41d2aa4  Yes         /lib/aarch64-linux-gnu/libpthread.so.0
0x0000007fb2460610  0x0000007fb27e4a38  Yes         /usr/lib/aarch64-linux-gnu/libopencv_cudaarithm.so.4.5
0x0000007fb1efa940  0x0000007fb21ff5d0  Yes         /usr/lib/aarch64-linux-gnu/libopencv_core.so.4.5
0x0000007fb1d8ff10  0x0000007fb1e3cc48  Yes         /usr/lib/aarch64-linux-gnu/libstdc++.so.6
0x0000007fb1ce1910  0x0000007fb1cef4d0  Yes         /lib/aarch64-linux-gnu/libgcc_s.so.1
0x0000007fb1ba6380  0x0000007fb1c966f8  Yes         /lib/aarch64-linux-gnu/libc.so.6
0x0000007fb1ad7f40  0x0000007fb1b2cd68  Yes         /lib/aarch64-linux-gnu/libm.so.6
0x0000007fb1aa2020  0x0000007fb1ab6e94  Yes         /lib/aarch64-linux-gnu/libz.so.1
0x0000007fb1a3ef60  0x0000007fb1a6fba0  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so
0x0000007fb19f7940  0x0000007fb1a167f4  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so
0x0000007fb19d51e0  0x0000007fb19dcdec  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so
0x0000007fb1945bb0  0x0000007fb19a9784  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so
0x0000007fb1927b40  0x0000007fb192d080  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
0x0000007fb1906170  0x0000007fb1911510  Yes         /usr/lib/aarch64-linux-gnu/libEGL.so.1
0x0000007fb180a820  0x0000007fb18836bc  Yes         /usr/lib/aarch64-linux-gnu/tegra-egl/libEGL_nvidia.so.0
0x0000007fb175b8a0  0x0000007fb17a5370  Yes         /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudart.so.10.2
0x0000007fb08d9680  0x0000007fb0b4d5b4  Yes         /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
0x0000007fa478f740  0x0000007fa49acf74  Yes         /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcufft.so.10
0x0000007fa4747e30  0x0000007fa474edc4  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so
0x0000007fa4733ae0  0x0000007fa47346bc  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0
0x0000007fa46b0b10  0x0000007fa46c5a38  Yes         /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0
0x0000007fa45072a0  0x0000007fa467e224  Yes         /usr/lib/aarch64-linux-gnu/libvpx.so.5
0x0000007fa4339000  0x0000007fa447ab38  Yes         /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
0x0000007fa423dc00  0x0000007fa428c954  Yes         /usr/local/cuda/lib64/libnppc.so.10
0x0000007fa374d3e0  0x0000007fa3966bcc  Yes         /usr/local/cuda/lib64/libnppial.so.10
0x0000007fa2f4cb90  0x0000007fa31078d4  Yes         /usr/local/cuda/lib64/libnppidei.so.10
0x0000007fa13c0820  0x0000007fa1575cec  Yes         /usr/local/cuda/lib64/libnppig.so.10
0x0000007f9ffea330  0x0000007fa01ce7b4  Yes         /usr/local/cuda/lib64/libnppist.so.10
0x0000007f9fcb4ff0  0x0000007f9fd673ac  Yes         /usr/local/cuda/lib64/libnppitc.so.10
0x0000007f9af66eb0  0x0000007f9ba48df4  Yes         /usr/local/cuda/lib64/libcublas.so.10
0x0000007f9af204b0  0x0000007f9af29314  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so
0x0000007f9aee19d0  0x0000007f9af03f50  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so
0x0000007f9ae489c0  0x0000007f9ae9d978  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvdla_runtime.so
0x0000007f9ad54c40  0x0000007f9ad57d24  Yes         /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0
0x0000007f9ac77e00  0x0000007f9acd91b0  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvidia-glsi.so.32.6.1
0x0000007f9ac2fa70  0x0000007f9ac49374  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so
0x0000007f9abb9cd0  0x0000007f9abf52e0  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.440.18
0x0000007f98bdf7e0  0x0000007f98d6efec  Yes         /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublasLt.so.10
0x0000007f98b89620  0x0000007f98b8bd38  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so
0x0000007f98b510a0  0x0000007f98b69f5c  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvidia-rmapi-tegra.so.32.6.1
0x0000007f98af4300  0x0000007f98b081b0  Yes         /usr/lib/aarch64-linux-gnu/libdrm.so.2
0x0000007f98ad30c0  0x0000007f98ad7310  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvll.so
0x0000007f98ab4fc0  0x0000007f98abe6e4  Yes         /usr/lib/aarch64-linux-gnu/libdrm.so.2.4.0
0x0000007f98a987e0  0x0000007f98a9cda4  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvidia-egl-wayland.so.1
0x0000007f98a7e090  0x0000007f98a82fc8  Yes         /usr/lib/aarch64-linux-gnu/libwayland-client.so.0
0x0000007f98a5e310  0x0000007f98a64ce0  Yes         /usr/lib/aarch64-linux-gnu/libwayland-server.so.0
0x0000007f98a417c0  0x0000007f98a45e2c  Yes         /usr/lib/aarch64-linux-gnu/libffi.so.6
0x0000007f98a01c70  0x0000007f98a26a50  Yes         /usr/lib/aarch64-linux-gnu/libEGL_mesa.so.0
0x0000007f989dcc00  0x0000007f989e32e0  Yes         /usr/lib/aarch64-linux-gnu/libgbm.so.1
0x0000007f9899f400  0x0000007f989a93cc  Yes         /usr/lib/aarch64-linux-gnu/libglapi.so.0
0x0000007f9894ea80  0x0000007f9896d534  Yes         /lib/aarch64-linux-gnu/libexpat.so.1
0x0000007f989395c0  0x0000007f989396ac  Yes         /usr/lib/aarch64-linux-gnu/libX11-xcb.so.1
0x0000007f98914280  0x0000007f98925564  Yes         /usr/lib/aarch64-linux-gnu/libxcb.so.1
0x0000007f988f6970  0x0000007f988f796c  Yes         /usr/lib/aarch64-linux-gnu/libxcb-dri2.so.0
0x0000007f988e0cc0  0x0000007f988e3044  Yes         /usr/lib/aarch64-linux-gnu/libxcb-xfixes.so.0
0x0000007f988cb640  0x0000007f988cc318  Yes         /usr/lib/aarch64-linux-gnu/libxcb-dri3.so.0
0x0000007f988b7e70  0x0000007f988b85ac  Yes         /usr/lib/aarch64-linux-gnu/libxcb-present.so.0
0x0000007f988a32c0  0x0000007f988a527c  Yes         /usr/lib/aarch64-linux-gnu/libxcb-sync.so.1
0x0000007f9888eaa0  0x0000007f9888edac  Yes         /usr/lib/aarch64-linux-gnu/libxshmfence.so.1
0x0000007f9887bf00  0x0000007f9887cce8  Yes         /usr/lib/aarch64-linux-gnu/libXau.so.6
0x0000007f98867280  0x0000007f98868c94  Yes         /usr/lib/aarch64-linux-gnu/libXdmcp.so.6
0x0000007f988474f0  0x0000007f98853920  Yes         /lib/aarch64-linux-gnu/libbsd.so.0
0x0000007f94561a20  0x0000007f95128044  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvidia-eglcore.so.32.6.1
0x0000007f942f7550  0x0000007f942ff9f0  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvpva.so
0x0000007f53c05510  0x0000007f53c85070  Yes         /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
0x0000007f53b99290  0x0000007f53bccd24  Yes         /usr/lib/aarch64-linux-gnu/libgobject-2.0.so.0
0x0000007f54121da0  0x0000007f54122514  Yes         /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so
0x0000007f540d13e0  0x0000007f540ee98c  Yes         /usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so
0x0000007f54091630  0x0000007f540b1474  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so
0x0000007f54016eb0  0x0000007f5406d4bc  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so.1.0.0
0x0000007f53ff18f0  0x0000007f53ff9b84  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvmm_utils.so
0x0000007f53fdb810  0x0000007f53fdbfb4  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so
0x0000007f53fc8360  0x0000007f53fc9578  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_utils.so
0x0000007f53fb4a10  0x0000007f53fb540c  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvavp.so
0x0000007f53f90b10  0x0000007f53f9e99c  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so
0x0000007f53f7d410  0x0000007f53f7d4dc  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvtnr.so
0x0000007f53f638e0  0x0000007f53f6a014  Yes         /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so

(gdb) cont
[Switching to thread 104 (Thread 0x7f4c37ada0 (LWP 452))](running)

Thread 48 "Work queue" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
3

Hi,

Which image format do you use?
Please note that VIC only supports the following data format:

https://docs.nvidia.com/vpi/algo_remap.html#algo_remap_limitations

If you do use the above’s format, would you mind sharing a sample to reproduce this issue with us?

Thanks.

4

The color format is NV12_ER. The issue is in type of wrapped memory, just wrapping same memory as host (instead of CUDA) makes it work again, but with severely degraded performance.

As for the sample - this is more difficult, I will try to minimize and extract it.

5

Sample that reproduces the issue (turns out that any algoirthm running on VIC causes it):

#include <cuda_runtime_api.h>
#include <vpi/algo/ConvertImageFormat.h>
#include <vpi/CUDAInterop.h>

int main(int argc, char **argv) {
    uint32_t width = 128;
    uint32_t height = 128;

    cudaStream_t stream;
    assert(cudaStreamCreate(&stream) == cudaSuccess);

    void *memory;
    assert(cudaMallocManaged(&memory, width * height * 4) == cudaSuccess);
    assert(cudaStreamAttachMemAsync(stream, memory, 0, cudaMemAttachSingle) == cudaSuccess);

    VPIStream vpiStream;
    assert(vpiStreamCreateCUDAStreamWrapper(stream, VPI_BACKEND_ALL, &vpiStream) == VPI_SUCCESS);

    VPIImage srcImage, dstImage;

    VPIImageData srcData {
        .format = VPI_IMAGE_FORMAT_RGBA8,
        .numPlanes = 1,
        .planes = {
            {
                .pixelType = VPI_PIXEL_TYPE_4U8,
                .width = (int32_t) width,
                .height = (int32_t) height,
                .pitchBytes = (int32_t) (4 * width),
                .data = memory
            }
        }
    };

    assert(vpiImageCreateCUDAMemWrapper(&srcData, 0, &srcImage) == VPI_SUCCESS);
    assert(vpiImageCreate(width, height, VPI_IMAGE_FORMAT_NV12_ER, 0, &dstImage) == VPI_SUCCESS);

    VPIConvertImageFormatParams params {
        .policy = VPI_CONVERSION_CLAMP,
        .scale = 1,
        .offset = 0,
        .flags = 0,
        .chromaUpFilter = VPI_INTERP_NEAREST,
        .chromaDownFilter = VPI_INTERP_NEAREST
    };

    assert(vpiSubmitConvertImageFormat(vpiStream, VPI_BACKEND_VIC, srcImage, dstImage, &params) == VPI_SUCCESS);
}

Output:

NVMEDIA_ARRAY:   53,  Version 2.1
NVMEDIA_VPI :  172,  Version 2.4
Fatal assertion error
#0 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x91ed38) [0x7fa47b9d38]
#1 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x91f794) [0x7fa47ba794]
#2 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x343a24) [0x7fa41dea24]
#3 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x32dafc) [0x7fa41c8afc]
#4 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x33fe1c) [0x7fa41dae1c]
#5 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x40ca64) [0x7fa42a7a64]
#6 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x40ceac) [0x7fa42a7eac]
#7 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x4042b8) [0x7fa429f2b8]
#8 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x421ab4) [0x7fa42bcab4]
#9 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0x41f2e8) [0x7fa42ba2e8]
#10 /opt/nvidia/vpi1/lib64/libnvvpi.so.1(+0xa1e234) [0x7fa48b9234]
#11 /lib/aarch64-linux-gnu/libpthread.so.0(+0x7088) [0x7fa3737088]
6

Hi,

Thanks for sharing the sample with us.

Confirmed that we can reproduce the assertion error in our environment as well.
We are checking this issue and will share more information with you later.

Thanks.

8

Hi,

Thanks for reporting this issue.

Confirm this is an issue from VPI.
We will fix this in our future releases.
Thanks.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.