Please provide complete information as applicable to your setup.
• Hardware Platform (GPU)
• DeepStream Version : 5.0 release
• TensorRT Version 7.0
• NVIDIA GPU Driver Version (440.100)
Hello, I faced with below error when run deeptream-app after a few minutes.
Please help!
ERROR: Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
1:39:17.039528673 11930 0x564e619d7b20 WARN nvinfer gstnvinfer.cpp:2046:gst_nvinfer_output_loop:<secondary_gie_0> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
1:39:17.039563500 11930 0x564e618a0a30 WARN nvinfer gstnvinfer.cpp:2046:gst_nvinfer_output_loop:<secondary_gie_1> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
1:39:17.039654445 11930 0x564e619d7b20 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1598> [UID = 2]: Tried to release an unknown outputBatchID
ERROR from secondary_gie_0: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: gstnvinfer.cpp(2046): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:secondary_gie_bin/GstNvInfer:secondary_gie_0
1:39:17.039693270 11930 0x564e618a0a30 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 3]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1606> [UID = 3]: Tried to release an outputBatchID which is already with the context
ERROR from secondary_gie_1: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: gstnvinfer.cpp(2046): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:secondary_gie_bin/GstNvInfer:secondary_gie_1
ERROR: Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
1:39:17.039871967 11930 0x564e619d7b20 WARN nvinfer gstnvinfer.cpp:2046:gst_nvinfer_output_loop:<secondary_gie_0> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
1:39:17.039904093 11930 0x564e619d7b20 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1598> [UID = 2]: Tried to release an unknown outputBatchID
ERROR from secondary_gie_0: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: gstnvinfer.cpp(2046): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:secondary_gie_bin/GstNvInfer:secondary_gie_0
Cuda failure: status=700
nvbufsurface: NvBufSurfaceMemSet: failed to set memory
Cuda failure: status=700 in CreateTextureObj at line 2555
nvbufsurftransform.cpp:2624: => Cuda failure: status=700
nvbufsurface: NvBufSurfaceMemSet: failed to set memory
Cuda failure: status=700 in CreateTextureObj at line 2555
Cuda failure: status=700
nvbufsurface: NvBufSurfaceMemSet: failed to set memory
Cuda failure: status=700 in CreateTextureObj at line 2555
Cuda failure: status=700
nvbufsurface: NvBufSurfaceMemSet: failed to set memory
Cuda failure: status=700 in CreateTextureObj at line 2555
Cuda failure: status=700 in CreateTextureObj at line 2555
nvbufsurftransform.cpp:2624: => Transformation Failed -2
Segmentation fault (core dumped)
With which configuration and models?
My customize SSD model and configuration specifics. But I traced gdb error that showing bellow.
#0 0x00007f9a2015ac2b in () at /usr/lib/x86_64-linux-gnu/libcuda.so.1
#1 0x00007f9a20075c4b in () at /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2 0x00007f9a20222d09 in cuEventDestroy_v2 () at /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3 0x00007f9a787de040 in () at /usr/local/cuda-10.2/lib64/libcudart.so.10.2
#4 0x00007f9a78818a54 in cudaEventDestroy () at /usr/local/cuda-10.2/lib64/libcudart.so.10.2
#5 0x00007f9a215b0d04 in NvBufSurfTransform_GPU_CuTex () at ///opt/nvidia/deepstream/deepstream-5.0/lib/libnvbufsurftransform.so
#6 0x00007f9a215b1606 in NvBufSurfTransform_GPU () at ///opt/nvidia/deepstream/deepstream-5.0/lib/libnvbufsurftransform.so
#7 0x00007f9a215adac9 in NvBufSurfTransform () at ///opt/nvidia/deepstream/deepstream-5.0/lib/libnvbufsurftransform.so
#8 0x00007f99e8284e15 in ConvBufManager::convertBatch(NvBufSurface*) ()
at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#9 0x00007f99e82886d8 in NvTrackerProc::submitInput(InputParams const&) ()
at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#10 0x00007f99e8283c9e in gst_nv_tracker_submit_input_buffer(_GstBaseTransform*, int, _GstBuffer*) ()
at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#11 0x00007f9a668bbb61 in () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#12 0x00007f9a79d1a88b in () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#13 0x00007f9a79d22bb3 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#14 0x00007f9a79d08aab in gst_proxy_pad_chain_default () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#15 0x00007f9a79d1a88b in () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#16 0x00007f9a79d22bb3 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#17 0x00007f9a79d08aab in gst_proxy_pad_chain_default () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#18 0x00007f9a79d1a88b in () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#19 0x00007f9a79d22bb3 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#20 0x00007f99e9eec47c in gst_nvinfer_output_loop(void*) ()
at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
#21 0x00007f9a79586175 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007f9a672ae6db in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#23 0x00007f9a78fc588f in clone () at /lib/x86_64-linux-gnu/libc.so.6
Hello, with the same log error but this time trace log error from transform in OSD gst-plugin:
Cuda failure: status=700 in CreateTextureObj at line 2555
nvbufsurftransform.cpp:2624: => Transformation Failed -2
nvbufsurftransform.cpp(2793) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
free(): corrupted unsorted chunks
ERROR: Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
0:15:32.623962061 10669 0x55d904f0ad40 WARN nvinfer gstnvinfer.cpp:2046:gst_nvinfer_output_loop:<secondary_gie_0> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
0:15:32.624015193 10669 0x55d904f0ad40 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1606> [UID = 2]: Tried to release an outputBatchID which is already with the context
Thread 24 "osd_conv_queue:" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f95fe7fc700 (LWP 10744)]
0x00007f96e57b7ff6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)
(gdb) bt
#0 0x00007f96e57b7ff6 in () at /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f96e57b813a in () at /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f968012a9ba in CHECK_(int, int, char const*) ()
at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_osd.so
#3 0x00007f968012a192 in nvll_osd_draw_rectangles_gpu(void*, _NvOSD_FrameRectParams*) ()
at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_osd.so
#4 0x00007f968012b78b in nvll_osd_draw_rectangles ()
at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_osd.so
#5 0x00007f968034d642 in gst_nvds_osd_transform_ip ()
at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_osd.so
#6 0x00007f96d318d2dc in () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#7 0x00007f96d318cb84 in () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#8 0x00007f96e65eb88b in () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#9 0x00007f96e65f3bb3 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#10 0x00007f9683f16ba9 in () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstcoreelements.so
#11 0x00007f96e6620269 in () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#12 0x00007f96e5e57b40 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x00007f96e5e57175 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007f96d3b7f6db in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007f96e589688f in clone () at /lib/x86_64-linux-gnu/libc.so.6
There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks
It is hard to get useful information from the log. Can you provide the method of how to reproduce the problem in our platform? If we can not reproduce the problem, it is hard to identify the reason.
Are you using Jetson or dGPU?
1 Like
what’s the solution to this problem
I got a similar error after the project has been running for 13k frames of the stream
Cuda failure: status=700 in CreateTextureObj at line 2555
nvbufsurftransform.cpp:2624: => Transformation Failed -2
Segmentation fault (core dumped)
my project has 2 models (a detector and a classifier)
it was running on two streams , the shorter one ended way before this error showed
Hi @mai.algendy. Based on my experiment, this problem cause by the box position is out of range 0-1.0 or child box out of the parent box, the transformation do a crop with bad rect parameter. Please check your output box coordinate.
1 Like
Thank you for responding
I changed a config file parameter based on your advice that scales bounding boxes between models, and it worked
1 Like