Error while Running SCRFD in deepstream

[neo21995]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version 6.1.1
• JetPack Version (valid for Jetson only) 5.0.2
• TensorRT Version 8.4
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

following this page scrfd detection

I followed the guide to compile custom parser and nms plugins. and ran only face detector without person or face embeddings models.

got the below error

ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) ERROR: Failed to enqueue trt inference batch ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR 0:00:12.100863066 338608 0xd770f60 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop:<primary-inference> error: Failed to queue input batch for inferencing Error: gst-stream-error-quark: Failed to queue input batch for inferencing (1): gstnvinfer.cpp(1357): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline0/GstNvInfer:primary-inference Exiting app

ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) ERROR: Failed to enqueue trt inference batch ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR 0:00:12.195473559 338608 0xd770f60 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop:<primary-inference> error: Failed to queue input batch for inferencing ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) ERROR: Failed to enqueue trt inference batch ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR 0:00:12.216500136 338608 0xd770f60 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop:<primary-inference> error: Failed to queue input batch for inferencing ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) ERROR: Failed to enqueue trt inference batch ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR 0:00:12.236711767 338608 0xd770f60 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop:<primary-inference> error: Failed to queue input batch for inferencing free(): invalid pointer Aborted (core dumped)

[fanzh]

1 could share the whole log？was the trt engine created?
2 please refer to this link to check if the libs version is right.
3 can the simple sample deepstream-test1 run successfully?

[neo21995]

@fanzh

1. That is the whole error log
2. I have ran other applications and its working
3. Yes

[fanzh]

1. there is no engine generating log, how did you generate the trt engine? did you use the trtexe?
2. I sugget using NVIDIA official embedding extactomg sample.

[neo21995]

@fanzh
I have ran this multiple times to check, on first run engine file created
Let me recreate the engine file and share the whole log here.

and the sample you have shared works with deepstream-6.2 but i am using deepstream-6.1.1. i will try with ds6.2.

[fanzh]

Sorry for the late reply. is there any update?

[neo21995]

@fanzh i am trying to upgrade jetpack and deepstream versions in LENEVO SE70 box, but got some issues.
for that i raised issuehere

[neo21995]

I just recreated the engine file and please find the error log in deepstream-6.1.1

`{‘stream0’: <common.FPS.GETFPS object at 0xffff880859a0>}
Creating Pipeline

Creating streamux

Creating source_bin 0 Preformatted text

Creating source bin
source-bin-00
Creating Pgie

Creating tracker

Creating tiler

Creating nvvidconv

Creating nvosd

Creating transform

Creating FileSink

Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
1 : file:///home/sensormatic/Videos/audience-001.mp4
Starting pipeline

Decodebin child added: source

Decodebin child added: decodebin0

Decodebin child added: qtdemux0

Decodebin child added: multiqueue0

Decodebin child added: h264parse0

Decodebin child added: capsfilter0

Decodebin child added: aacparse0

Decodebin child added: avdec_aac0

Decodebin child added: nvv4l2decoder0

In cb_newpad

gstname= video/x-raw
features= <Gst.CapsFeatures object at 0xffff77f60640 (GstCapsFeatures at 0xfffeb46f0260)>
In cb_newpad

gstname= audio/x-raw
Exiting app

WARNING: Deserialize engine failed because file path: /home/sensormatic/Neo/deepstream-face-recognition/src/weights/scrfd-nms-full.nms.onnx_b1_gpu0_fp32.engine open error
WARNING: [TRT]: onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: builtin_op_importers.cpp:4714: Attribute scoreBits not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 6
0 INPUT kFLOAT input.1 3x640x640
1 OUTPUT kINT32 num_detections 1
2 OUTPUT kFLOAT nmsed_boxes 200x4
3 OUTPUT kFLOAT nmsed_scores 200
4 OUTPUT kFLOAT nmsed_classes 200
5 OUTPUT kFLOAT nmsed_landmarks 200x10

Opening in BLOCKING MODE

n.so python3 main_ff.py file:///home/sensormatic/Videos/audience-001.mp4 >> log.txt
main_ff.py:302: PyGIDeprecationWarning: Since version 3.11, calling threads_init is no longer needed. See: Projects/PyGObject/Threading - GNOME Wiki!
GObject.threads_init()
main_ff.py:465: PyGIDeprecationWarning: GObject.MainLoop is deprecated; use GLib.MainLoop instead
loop = GObject.MainLoop()
0:00:07.926557499 305154 0xffff24002360 WARN nvinfer gstnvinfer.cpp:663:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1897> [UID = 1]: deserialize engine from file :/home/sensormatic/Neo/deepstream-face-recognition/src/weights/scrfd-nms-full.nms.onnx_b1_gpu0_fp32.engine failed
0:00:08.041163987 305154 0xffff24002360 WARN nvinfer gstnvinfer.cpp:663:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2002> [UID = 1]: deserialize backend context from engine from file :/home/sensormatic/Neo/deepstream-face-recognition/src/weights/scrfd-nms-full.nms.onnx_b1_gpu0_fp32.engine failed, try rebuild
0:00:08.043765446 305154 0xffff24002360 INFO nvinfer gstnvinfer.cpp:666:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1923> [UID = 1]: Trying to create engine from model files
0:02:29.562478392 305154 0xffff24002360 INFO nvinfer gstnvinfer.cpp:666:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1955> [UID = 1]: serialize cuda engine to file: /home/sensormatic/Neo/deepstream-face-recognition/src/weights/scrfd-nms-full.nms.onnx_b1_gpu0_fp32.engine successfully
0:02:29.888386979 305154 0xffff24002360 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus: [UID 1]: Load new model:configs/config_scrfd_fullframe.txt sucessfully
0:02:29.906126362 305154 0xffff24002360 WARN basesrc gstbasesrc.c:3600:gst_base_src_start_complete: pad not activated yet
0:02:29.920847581 305154 0xffff24002360 WARN basesrc gstbasesrc.c:3600:gst_base_src_start_complete: pad not activated yet
0:02:29.965459016 305154 0xebe6700 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type .TIM
0:02:29.965588329 305154 0xebe6700 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type .TSC
0:02:29.965663402 305154 0xebe6700 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type .TSZ
0:02:29.965979116 305154 0xebe6700 WARN qtdemux qtdemux.c:3250:qtdemux_parse_trex: failed to find fragment defaults for stream 1
0:02:29.966460943 305154 0xebe6700 WARN qtdemux qtdemux.c:3250:qtdemux_parse_trex: failed to find fragment defaults for stream 2
0:02:31.451398518 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:4481:gst_v4l2_object_probe_caps:nvv4l2decoder0:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:02:31.451541847 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.451630232 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.451709113 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
0:02:31.475036181 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005240 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.575394518 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:4481:gst_v4l2_object_probe_caps:nvv4l2decoder0:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:02:31.575508215 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.575591031 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.575657496 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.593515855 305154 0xfffec000c980 WARN v4l2videodec gstv4l2videodec.c:1847:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency
0:02:31.594902873 305154 0xfffec000c980 WARN v4l2bufferpool gstv4l2bufferpool.c:1082:gst_v4l2_buffer_pool_start:nvv4l2decoder0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:02:31.604186551 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005360 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.604450105 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.604646458 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5120 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.604839163 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5900 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.605028797 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb800e480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.605217950 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffec0042a20 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.605412351 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c8000 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.625917385 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5c60 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.627160785 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xffff2c00c5a0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.628086967 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xffff2c00c360 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.628923293 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005240 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.629484417 305154 0xfffeb4027aa0 WARN v4l2bufferpool gstv4l2bufferpool.c:1533:gst_v4l2_buffer_pool_dqbuf:nvv4l2decoder0:pool:src Driver should never set v4l2_buffer.field to ANY
0:02:31.630033668 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005360 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.630524840 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.630898506 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5120 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.633014936 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5b40 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.633708893 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffec0042a20 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.637575735 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5ea0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.665233840 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb800e6c0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.667076413 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xffff2c00c480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.668744488 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005c60 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.670442131 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c4ea0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.692932746 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5360 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.708486002 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.718253844 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c57e0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.729938082 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c87e0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
ERROR: Failed to enqueue trt inference batch
ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:02:31.891126619 305154 0xe8a0760 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
Error: gst-stream-error-quark: Failed to queue input batch for inferencing (1): gstnvinfer.cpp(1357): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline0/GstNvInfer:primary-inference
0:02:31.960708301 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xffff2c00cea0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.967988574 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c8000 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.975887283 305154 0xfffeb4027aa0 WARN videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame: decreasing timestamp (0:00:00.583333333 < 0:00:02.333333333)
ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
ERROR: Failed to enqueue trt inference batch
ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:02:31.996058362 305154 0xe8a0760 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
ERROR: Failed to enqueue trt inference batch
ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:02:32.108086633 305154 0xe8a0760 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
ERROR: Failed to enqueue trt inference batch
ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:02:32.132506636 305154 0xe8a0760 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
free(): invalid pointer
Aborted (core dumped)`

[fanzh]

@neo21995

i am trying to upgrade jetpack and deepstream versions in LENEVO SE70 box, but got some issues.

ok, please try again after fix the setup issues.

[neo21995]

@fanzh i am not able to find the jetpack 5.1.2 custom build from lenovo.

Error running Deepstream-6.3

Hi, The default package is for developer kit so it may not be eligible for custom board. Generally the vendor would need to prepare the package for custom board. Besides, 5.0.2 is first Jetpack 5 release and it’s possible this function is not ready yet. If you have Xavier NX developer kit in 5.1.1 and would like to upgrade to 5.1.2, this method should work.

[fanzh]

Sorry for the late reply, Is this still an DeepStream issue to support? Thanks

[neo21995]

yes, i am not able to update the jetpack and deepstream versions.

Need help to resolve the issue, the original scrfd repo is able to run it on deepstream-6.0

but i am not able to run in jetson xavier NX with jetpack 5.0.2 (custom board)

[fanzh]

Noticing there was setup JetaPack 5.1.2 issue, did you revert to 6.1.1+ 5.0.2 again now? please share the deepstream-test1’s running log and the scfd model’s running error log.

[neo21995]

@neo21995

{‘stream0’: <common.FPS.GETFPS object at 0xffff880859a0>}
Creating Pipeline

Creating streamux

Creating source_bin 0 Preformatted text

Creating source bin
source-bin-00
Creating Pgie

Creating tracker

Creating tiler

Creating nvvidconv

Creating nvosd

Creating transform

Creating FileSink

Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
1 : file:///home/sensormatic/Videos/audience-001.mp4
Starting pipeline

Decodebin child added: source

Decodebin child added: decodebin0

Decodebin child added: qtdemux0

Decodebin child added: multiqueue0

Decodebin child added: h264parse0

Decodebin child added: capsfilter0

Decodebin child added: aacparse0

Decodebin child added: avdec_aac0

Decodebin child added: nvv4l2decoder0

In cb_newpad

gstname= video/x-raw
features= <Gst.CapsFeatures object at 0xffff77f60640 (GstCapsFeatures at 0xfffeb46f0260)>
In cb_newpad

gstname= audio/x-raw
Exiting app

WARNING: Deserialize engine failed because file path: /home/sensormatic/Neo/deepstream-face-recognition/src/weights/scrfd-nms-full.nms.onnx_b1_gpu0_fp32.engine open error
WARNING: [TRT]: onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: builtin_op_importers.cpp:4714: Attribute scoreBits not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 6
0 INPUT kFLOAT input.1 3x640x640
1 OUTPUT kINT32 num_detections 1
2 OUTPUT kFLOAT nmsed_boxes 200x4
3 OUTPUT kFLOAT nmsed_scores 200
4 OUTPUT kFLOAT nmsed_classes 200
5 OUTPUT kFLOAT nmsed_landmarks 200x10

Opening in BLOCKING MODE

n.so python3 main_ff.py file:///home/sensormatic/Videos/audience-001.mp4 >> log.txt
main_ff.py:302: PyGIDeprecationWarning: Since version 3.11, calling threads_init is no longer needed. See: Projects/PyGObject/Threading - GNOME Wiki!
GObject.threads_init()
main_ff.py:465: PyGIDeprecationWarning: GObject.MainLoop is deprecated; use GLib.MainLoop instead
loop = GObject.MainLoop()
0:00:07.926557499 305154 0xffff24002360 WARN nvinfer gstnvinfer.cpp:663:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1897> [UID = 1]: deserialize engine from file :/home/sensormatic/Neo/deepstream-face-recognition/src/weights/scrfd-nms-full.nms.onnx_b1_gpu0_fp32.engine failed
0:00:08.041163987 305154 0xffff24002360 WARN nvinfer gstnvinfer.cpp:663:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2002> [UID = 1]: deserialize backend context from engine from file :/home/sensormatic/Neo/deepstream-face-recognition/src/weights/scrfd-nms-full.nms.onnx_b1_gpu0_fp32.engine failed, try rebuild
0:00:08.043765446 305154 0xffff24002360 INFO nvinfer gstnvinfer.cpp:666:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1923> [UID = 1]: Trying to create engine from model files
0:02:29.562478392 305154 0xffff24002360 INFO nvinfer gstnvinfer.cpp:666:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1955> [UID = 1]: serialize cuda engine to file: /home/sensormatic/Neo/deepstream-face-recognition/src/weights/scrfd-nms-full.nms.onnx_b1_gpu0_fp32.engine successfully
0:02:29.888386979 305154 0xffff24002360 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus: [UID 1]: Load new model:configs/config_scrfd_fullframe.txt sucessfully
0:02:29.906126362 305154 0xffff24002360 WARN basesrc gstbasesrc.c:3600:gst_base_src_start_complete: pad not activated yet
0:02:29.920847581 305154 0xffff24002360 WARN basesrc gstbasesrc.c:3600:gst_base_src_start_complete: pad not activated yet
0:02:29.965459016 305154 0xebe6700 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type .TIM
0:02:29.965588329 305154 0xebe6700 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type .TSC
0:02:29.965663402 305154 0xebe6700 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type .TSZ
0:02:29.965979116 305154 0xebe6700 WARN qtdemux qtdemux.c:3250:qtdemux_parse_trex: failed to find fragment defaults for stream 1
0:02:29.966460943 305154 0xebe6700 WARN qtdemux qtdemux.c:3250:qtdemux_parse_trex: failed to find fragment defaults for stream 2
0:02:31.451398518 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:4481:gst_v4l2_object_probe_caps:nvv4l2decoder0:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:02:31.451541847 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.451630232 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.451709113 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
0:02:31.475036181 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005240 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.575394518 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:4481:gst_v4l2_object_probe_caps:nvv4l2decoder0:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:02:31.575508215 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.575591031 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.575657496 305154 0xfffec000c980 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffeb401b390 Failed to determine interlace mode
0:02:31.593515855 305154 0xfffec000c980 WARN v4l2videodec gstv4l2videodec.c:1847:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency
0:02:31.594902873 305154 0xfffec000c980 WARN v4l2bufferpool gstv4l2bufferpool.c:1082:gst_v4l2_buffer_pool_start:nvv4l2decoder0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:02:31.604186551 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005360 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.604450105 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.604646458 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5120 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.604839163 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5900 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.605028797 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb800e480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.605217950 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffec0042a20 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.605412351 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c8000 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.625917385 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5c60 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.627160785 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xffff2c00c5a0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.628086967 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xffff2c00c360 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.628923293 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005240 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.629484417 305154 0xfffeb4027aa0 WARN v4l2bufferpool gstv4l2bufferpool.c:1533:gst_v4l2_buffer_pool_dqbuf:nvv4l2decoder0:pool:src Driver should never set v4l2_buffer.field to ANY
0:02:31.630033668 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005360 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.630524840 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.630898506 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5120 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.633014936 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5b40 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.633708893 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffec0042a20 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.637575735 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5ea0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.665233840 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb800e6c0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.667076413 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xffff2c00c480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.668744488 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005c60 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.670442131 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c4ea0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.692932746 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c5360 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.708486002 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb4005480 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.718253844 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c57e0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.729938082 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c87e0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
ERROR: Failed to enqueue trt inference batch
ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:02:31.891126619 305154 0xe8a0760 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
Error: gst-stream-error-quark: Failed to queue input batch for inferencing (1): gstnvinfer.cpp(1357): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline0/GstNvInfer:primary-inference
0:02:31.960708301 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xffff2c00cea0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.967988574 305154 0xfffec000c980 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeb40c8000 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:02:31.975887283 305154 0xfffeb4027aa0 WARN videodecoder gstvideodecoder.c:2761:gst_video_decoder_prepare_finish_frame: decreasing timestamp (0:00:00.583333333 < 0:00:02.333333333)
ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
ERROR: Failed to enqueue trt inference batch
ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:02:31.996058362 305154 0xe8a0760 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
ERROR: Failed to enqueue trt inference batch
ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:02:32.108086633 305154 0xe8a0760 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
ERROR: [TRT]: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )
ERROR: Failed to enqueue trt inference batch
ERROR: Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:02:32.132506636 305154 0xe8a0760 WARN nvinfer gstnvinfer.cpp:1357:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
free(): invalid pointer
Aborted (core dumped)`

scrfd error log

[fanzh]

could you share log of “deepstream-test1-app /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_config.yml”? you can redirect it to a file.

[neo21995]

@fanzh

deepstream-test1-app /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_config.yml

getting this error
$ deepstream-test1-app /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_config.yml terminate called after throwing an instance of 'YAML::BadFile' what(): bad file Aborted (core dumped)

[neo21995]

Please find the dstest3 app log
dstest3-log.txt (111.5 KB)

[fanzh]

are you using the same steps in the link, are you testing in the docker?

[neo21995]

i am following the same steps and i am not using docker

[fanzh]

1. can you use docker to test? testing in docker should be fine since the author recommends using docker.
2. it is custom code. you can ask the author if have met this error when not using docker.