Deepstream_parallel_inference_app gst_nvvideoconvert_transform: buffer transform failed

chanduhna · July 24, 2024, 12:08pm

• Hardware Platform ( GPU: Nvidia GeForce RTX 3090)
• DeepStream Version: 6.3 docker container
• NVIDIA GPU Driver Version: 535.183.06
• Issue Type( questions)

I tried this repo GitHub - NVIDIA-AI-IOT/deepstream_parallel_inference_app: A project demonstrating how to use nvmetamux to run multiple models in parallel. for a parallel inference app using 2 models. Sometimes it runs but fine but sometimes it ends up with this error. How do i fix it? I tried changing nvbuf-memory-type from 0 to 1,2 and 3 but it still didn’t help

0:01:42.824551192 57145 0x55e28cb3e470 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 2]: Load new model:/opt/nvidia/deepstream/deepstream-6.3/deepstream_parallel_inference_app/tritonclient/sample/configs/bodypose2d/config_body2_infer.txt sucessfully
Running...
Processing frame number = 0
0:01:43.473422382 57145 0x55e28bcc8120 ERROR         nvvideoconvert gstnvvideoconvert.c:4095:gst_nvvideoconvert_transform: buffer transform failed
Processing frame number = 4
Processing frame number = 7
Processing frame number = 9
Processing frame number = 11
0:01:44.964598101 57145 0x55e28bdc5120 ERROR            nvdsmetamux gstnvdsmetamux.cpp:1005:gst_nvdsmetamux_aggregate:<infer_bin_muxer> push error
ERROR from element osd_queue: Internal data stream error.
Error details: gstqueue.c(988): gst_queue_handle_sink_event (): /GstPipeline:deepstream-tensorrt-openpose-pipeline/GstBin:osd_bin/GstQueue:osd_queue:
streaming stopped, reason error (-5)
Quitting
Returned, stopping playback
0:01:44.964844951 57145 0x55e28cb3e000 WARN                 nvinfer gstnvinfer.cpp:2397:gst_nvinfer_output_loop:<primary_gie> error: Internal data stream error.
0:01:44.964859108 57145 0x55e28cb3e000 WARN                 nvinfer gstnvinfer.cpp:2397:gst_nvinfer_output_loop:<primary_gie> error: streaming stopped, reason error (-5)
0:01:44.976088933 57145 0x55e28bdc5240 WARN                 nvinfer gstnvinfer.cpp:2397:gst_nvinfer_output_loop:<primary_gie> error: Internal data stream error.
0:01:44.976107729 57145 0x55e28bdc5240 WARN                 nvinfer gstnvinfer.cpp:2397:gst_nvinfer_output_loop:<primary_gie> error: streaming stopped, reason error (-5)
Deleting pipeline
App run successful

junshengy · July 25, 2024, 3:59am

How did you start Docker? And how did you reproduce the problem? Can you share the command line?

Also try to execute export DBG_NVBUFSURFTRANSFORM=1 before run this sample, I want to know the reason why transform failed

chanduhna · July 25, 2024, 6:54am

I start my docker container by

export DISPLAY=:0
xhost +
docker run -it --net=host --gpus all -d -e DISPLAY=$DISPLAY --device /dev/snd -v /tmp/.X11-unix/:/tmp/.X11-unix --restart unless-stopped ds_parallel_app:latest

When i launch a new container and pull the parallel repo from github and build everything from scratch it works fine. I then docker commit the same container. The system reboots every night so when i launch the committed docker container the next day and try to run the command, it ends up with this error.

EDIT: I had changed the tiled display to turn off for because I was using one source, when I turned it on, the error went away

Here is the output with export DBG_NVBUFSURFTRANSFORM=1

0:00:02.210523660  4465 0x559d662d8780 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 2]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.3/deepstream_parallel_inference_app/tritonserver/models/bodypose2d/1/model.onnx_b4_gpu0_fp16.engine
nvbufsurftransform.cpp:1867: NvBufSurfTransformSessionCreate=> Session created 0x559d88d94280 (nil)
nvbufsurftransform.cpp:2065: NvBufSurfTransformSetSessionParams=> No session associated created 0x559d88d94280 for thread=139885129555968
0:00:02.219865407  4465 0x559d662d8780 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 2]: Load new model:/opt/nvidia/deepstream/deepstream-6.3/deepstream_parallel_inference_app/tritonclient/sample/configs/bodypose2d/config_body2_infer.txt sucessfully
Running...
nvbufsurftransform.cpp:1867: NvBufSurfTransformSessionCreate=> Session created 0x7f37d8007e30 (nil)
nvbufsurftransform.cpp:2065: NvBufSurfTransformSetSessionParams=> No session associated created 0x7f37d8007e30 for thread=139880793362432
nvbufsurftransform.cpp:1867: NvBufSurfTransformSessionCreate=> Session created 0x7f3788000d70 (nil)
nvbufsurftransform.cpp:2065: NvBufSurfTransformSetSessionParams=> No session associated created 0x7f3788000d70 for thread=139880955736064
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d8510d0e0 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=219690us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f37f4004550, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d21bf0 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=219660us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d87ea9560 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=894us
GPU 0 used srcbuf=0x7f37f4002150, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d27050 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=635us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d880cd2d0 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=15us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f37f4009c10, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d21bf0 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=4us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d8510d0e0 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=11us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f37f400a850, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d27050 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=58us
Processing frame number = 0
nvbufsurftransform.cpp:1867: NvBufSurfTransformSessionCreate=> Session created 0x7f38f8093bb0 (nil)
nvbufsurftransform.cpp:2065: NvBufSurfTransformSetSessionParams=> No session associated created 0x7f38f8093bb0 for thread=139883268190208
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f38fc004800, filled=4, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x7f38f8005140 filled=4, 1920x1080, fmt=NVBUF_COLOR_FORMAT_RGBA mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f38f8093bb0 totaltime=294us
nvbufsurftransform.cpp:1867: NvBufSurfTransformSessionCreate=> Session created 0x7f39087ae410 (nil)
nvbufsurftransform.cpp:2065: NvBufSurfTransformSetSessionParams=> No session associated created 0x7f39087ae410 for thread=139883293368320
nvbufsurftransform.cpp:2543: NvBufSurfTransformAsync=> Not enough buffers in dst
0:00:02.623208687  4465 0x559d65418520 ERROR         nvvideoconvert gstnvvideoconvert.c:4095:gst_nvvideoconvert_transform: buffer transform failed
Processing frame number = 4
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f37f400b440, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d21bf0 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=38us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d87ea9560 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=79us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f37f4009490, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d27050 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=9us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d880cd2d0 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=127us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f38fc006370, filled=2, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x7f38f800e3d0 filled=2, 1920x1080, fmt=NVBUF_COLOR_FORMAT_RGBA mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f38f8093bb0 totaltime=364us
Processing frame number = 6
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d8510d0e0 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=187us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f37f400ba00, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d21bf0 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=193us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f379c008e80, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x7f38f8012090 filled=1, 1920x1080, fmt=NVBUF_COLOR_FORMAT_RGBA mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f38f8093bb0 totaltime=279us
Processing frame number = 7
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f37f400adb0, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d27050 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=186us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d87ea9560 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=184us
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f38fc004190, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x7f38f8015da0 filled=1, 1920x1080, fmt=NVBUF_COLOR_FORMAT_RGBA mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f38f8093bb0 totaltime=270us
Processing frame number = 8
0:00:04.187566237  4465 0x559d65515120 ERROR            nvdsmetamux gstnvdsmetamux.cpp:1005:gst_nvdsmetamux_aggregate:<infer_bin_muxer> push error
ERROR from element osd_queue: Internal data stream error.
Error details: gstqueue.c(988): gst_queue_handle_sink_event (): /GstPipeline:deepstream-tensorrt-openpose-pipeline/GstBin:osd_bin/GstQueue:osd_queue:
streaming stopped, reason error (-5)
Quitting
Returned, stopping playback
0:00:04.187758395  4465 0x559d65515300 WARN                 nvinfer gstnvinfer.cpp:2397:gst_nvinfer_output_loop:<primary_gie> error: Internal data stream error.
0:00:04.187730616  4465 0x7f37b01af190 WARN           nvinferserver gstnvinferserver.cpp:567:gst_nvinfer_server_push_buffer:<primary_gie> error: Internal data stream error.
0:00:04.187783056  4465 0x559d65515300 WARN                 nvinfer gstnvinfer.cpp:2397:gst_nvinfer_output_loop:<primary_gie> error: streaming stopped, reason error (-5)
0:00:04.187791146  4465 0x7f37b01af190 WARN           nvinferserver gstnvinferserver.cpp:567:gst_nvinfer_server_push_buffer:<primary_gie> error: streaming stopped, reason error (-5)
nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> nvbufsurftransform.cpp:2652: NvBufSurfTransformAsync=> GPU 0 used srcbuf=0x7f37f400bd60, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d66d21bf0 filled=1, 416x416, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Bilinear session=0x7f3788000d70 totaltime=47usGPU 0 used srcbuf=0x559d658fd478, filled=1, 1920x1080 fmt=NVBUF_COLOR_FORMAT_NV12 mem=NVBUF_MEM_CUDA_DEVICE dstbuf=0x559d880cd2d0 filled=1, 224x224, fmt=NVBUF_COLOR_FORMAT_RGB mem=NVBUF_MEM_CUDA_DEVICE method=NvBufSurfTransformInter_Default session=0x7f37d8007e30 totaltime=30us

nvbufsurftransform.cpp:1890: NvBufSurfTransformSessionDestroy=> Session destroyed 0x7f3788000d70
nvbufsurftransform.cpp:1890: NvBufSurfTransformSessionDestroy=> Session destroyed 0x7f39087ae410
nvbufsurftransform.cpp:1890: NvBufSurfTransformSessionDestroy=> Session destroyed 0x7f38f8093bb0
Deleting pipeline
App run successful
nvbufsurftransform.cpp:1890: NvBufSurfTransformSessionDestroy=> Session destroyed 0x7f37d8007e30
nvbufsurftransform.cpp:985: taskTrim=> Exiting Clean up thread
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:518: deleteCuResData=> Destroy tex array
nvbufsurftransform.cpp:1899: libnvbufsurf_transform_deinit=> Deinit lib
nvbufsurftransform.cpp:1890: NvBufSurfTransformSessionDestroy=> Session destroyed 0x559d654a2ea0

junshengy · July 25, 2024, 7:31am

Try add the --privilaged parameter. From your description, this seems more like an environment issue than a code issue

chanduhna · July 25, 2024, 7:32am

Got it, thanks

system · August 8, 2024, 7:33am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Deepstream 6.1: deepstream-app not working after install DeepStream SDK deepstream61	9	2041	September 5, 2022
cuGraphicsGLRegisterBuffer failed with error(219) gst_eglglessink_cuda_init texture = 1 DeepStream SDK ubuntu	6	2453	February 1, 2022
Problem facing with DeepStream App DeepStream SDK	13	2400	October 12, 2021
The DeepStream image nvcr.io/nvidia/deepstream-l4t:5.1-21.02-samples pulled from NGC to my NVIDIA NX failed to start any application DeepStream SDK	34	3098	October 14, 2021
Internal data stream error while running deepstream-test1 DeepStream SDK	6	557	February 17, 2023
Can't start deepstream-app DeepStream 6.1 DeepStream SDK deepstream61	10	1115	May 31, 2022
Error while running deepstream-test1(ERROR from element primary-nvinference-engine: Internal data stream error.) DeepStream SDK	8	789	February 20, 2023
New installation Multiple Failues DeepStream SDK	18	1130	June 28, 2022
Same error with nvcr.io/nvidia/deepstream:6.2-devel : "cuGraphicsGLRegisterBuffer failed with error(304) gst_eglglessink_cuda_init texture = 1" DeepStream SDK gstreamer	5	348	June 30, 2023
DeepStream not working on Azure V100 vm DeepStream SDK	6	655	October 12, 2021

Deepstream_parallel_inference_app gst_nvvideoconvert_transform: buffer transform failed

Related topics