0:00:04.100234080 2190201 0xfffefc053300 WARN

[user53799]

Hardware Platform (Jetson / GPU) NX
• DeepStream Version 6.11
• JetPack Version (valid for Jetson only) 5.02
• TensorRT Version 8.4

I am using preprocess plug-in, by adding my own function to calculate the rect-params of roi. It seemd working, but when testing with a video, I found the inferce box is located a little more right than should be. And when running --gst-debug=3, I got the following:

Using winsys: x11
0:00:03.729875264 2190201 0xaaab05eac010 INFO nvinfer gstnvinfer.cpp:646:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.1/samples/configs/myconfigs/…/…/models/Primary_Detector/gh_fp16.trt
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 5
0 INPUT kFLOAT images 3x640x640
1 OUTPUT kFLOAT onnx::Sigmoid_477 3x80x80x6
2 OUTPUT kFLOAT onnx::Sigmoid_529 3x40x40x6
3 OUTPUT kFLOAT onnx::Sigmoid_581 3x20x20x6
4 OUTPUT kFLOAT output 25200x6

0:00:03.783599968 2190201 0xaaab05eac010 INFO nvinfer gstnvinfer.cpp:646:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.1/samples/configs/myconfigs/…/…/models/Primary_Detector/gh_fp16.trt
0:00:03.790726336 2190201 0xaaab05eac010 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-6.1/samples/configs/myconfigs/…/myconfigs/gh_config_infer_primary.txt sucessfully
Warning: converting ROIs to RGBA for VIC mode…I add it

0:00:03.806897344 2190201 0xaaab05eac010 WARN basesrc gstbasesrc.c:3600:gst_base_src_start_complete: pad not activated yet
0:00:03.808235968 2190201 0xaaab05eac010 WARN basesrc gstbasesrc.c:3600:gst_base_src_start_complete: pad not activated yet

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:194>: Pipeline ready

0:00:03.823347232 2190201 0xaaab05f6c5e0 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type pasp
0:00:03.823496800 2190201 0xaaab05f6c5e0 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type sgpd
0:00:03.823541792 2190201 0xaaab05f6c5e0 WARN qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type sbgp
0:00:03.823676832 2190201 0xaaab05f6c5e0 WARN qtdemux qtdemux.c:3250:qtdemux_parse_trex: failed to find fragment defaults for stream 1
0:00:03.824049536 2190201 0xaaab05f6c5e0 WARN qtdemux qtdemux.c:3250:qtdemux_parse_trex: failed to find fragment defaults for stream 2
Opening in BLOCKING MODE
0:00:03.941957472 2190201 0xfffefc053300 WARN v4l2 gstv4l2object.c:4481:gst_v4l2_object_probe_caps:nvv4l2decoder0:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:03.942063200 2190201 0xfffefc053300 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffee4013d00 Failed to determine interlace mode
0:00:03.942158272 2190201 0xfffefc053300 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffee4013d00 Failed to determine interlace mode
0:00:03.942234208 2190201 0xfffefc053300 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffee4013d00 Failed to determine interlace mode
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
0:00:03.951510912 2190201 0xfffefc053300 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeec0b67e0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:00:04.052036064 2190201 0xfffefc053300 WARN v4l2 gstv4l2object.c:4481:gst_v4l2_object_probe_caps:nvv4l2decoder0:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:04.053013472 2190201 0xfffefc053300 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffee4013d00 Failed to determine interlace mode
0:00:04.053551904 2190201 0xfffefc053300 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffee4013d00 Failed to determine interlace mode
0:00:04.053621856 2190201 0xfffefc053300 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xfffee4013d00 Failed to determine interlace mode
** INFO: <bus_callback:180>: Pipeline running

0:00:04.074474336 2190201 0xfffefc053300 WARN v4l2videodec gstv4l2videodec.c:1847:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency
0:00:04.076066784 2190201 0xfffefc053300 WARN v4l2bufferpool gstv4l2bufferpool.c:1082:gst_v4l2_buffer_pool_start:nvv4l2decoder0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:04.085152256 2190201 0xfffefc053300 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffee40bb120 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:00:04.086453056 2190201 0xfffefc053300 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeec0b6d80 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:00:04.086639520 2190201 0xfffefc053300 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffee40bb5a0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:00:04.094346304 2190201 0xfffefc053300 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffee40bb240 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:00:04.100234080 2190201 0xfffefc053300 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffeec0c45a0 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
0:00:04.102245632 2190201 0xfffefc053300 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:nvv4l2decoder0:pool:sink Buffer 0xfffee40bbb40 without the memory tag has maxsize (0) that is smaller than the configured buffer pool size (4194304). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass

Please help me to fix it. Thank you.

[fanzh]

1. it is not deepstream error log, it is gstreamer 's warning log, which is about resuing buffer.
2. what is your media pipeline? which sample are your application based on? did you test your model? can your model give right coordinates?

[user53799]

My media is source0, file:///home/shisun/81fd5,mp4.
Sample app is deepstream-app, and customed preprocess plugin.
The model is torch.pt>>onnx>>trt, and has been tested for many times. It was giving correct result if there’s no preprocess plugin.
One thing strange, during the osd playing of .mp4, more than 2/3 time the result was correct, only the last 1/3 piece of time, the infer box located more righthand.

[fanzh]

about “It was giving correct result if there’s no preprocess plugin.”, please narrow down this issue, find which code in preprocess will cause this issue.

[user53799]

Ok, I am testing which part of the preprocess is the cause.
Thank you.

[user53799]

Hi,
I did several tests of the preprocess applying different size of customed ROI(my preprocessing roi was changing all the time), and found that when the ROI was smaller, such as less than 600480, the corresponding inference box was located correctly, on the other hand, when the ROI was larger, such as 800600, the infered box was located obviously righthand than where should be.
So, I wander if ROI should have a limited size. But I do need to preprocess a changable and large ROI, Please info me how to configure GSTBUFFER, or other BUFFER.
Thank you.

preprocess config_file:
[property]
enable=1
target-unique-ids=1
# 0=NCHW, 1=NHWC, 2=CUSTOM
network-input-order=0
# tensor shape based on network-input-order
#network-input-shape=7;3;368;640
network-input-shape=1;3;640;640
processing-width=640
processing-height=640
scaling-buf-pool-size=6
tensor-buf-pool-size=6
# 0=RGB, 1=BGR, 2=GRAY
network-color-format=0
# 0=FP32, 1=UINT8, 2=INT8, 3=UINT32, 4=INT32, 5=FP16
tensor-data-type=0
#tensor-name=Input_1
tensor-name=images
# 0=NVBUF_MEM_DEFAULT 1=NVBUF_MEM_CUDA_PINNED 2=NVBUF_MEM_CUDA_DEVICE 3=NVBUF_MEM_CUDA_UNIFIED 4=NVBUF_MEM_SURFACE_ARRAY
scaling-pool-memory-type=0
# 0=NvBufSurfTransformCompute_Default 1=NvBufSurfTransformCompute_GPU 2=NvBufSurfTransformCompute_VIC
scaling-pool-compute-hw=0
# Scaling Interpolation method
# 0=NvBufSurfTransformInter_Nearest 1=NvBufSurfTransformInter_Bilinear 2=NvBufSurfTransformInter_Algo1
# 3=NvBufSurfTransformInter_Algo2 4=NvBufSurfTransformInter_Algo3 5=NvBufSurfTransformInter_Algo4
# 6=NvBufSurfTransformInter_Default
scaling-filter=0

Streammux:
[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000

Set muxer output width and height

width=1280
height=720
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

If set to TRUE, system timestamp will be attached as ntp timestamp

If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached

attach-sys-ts-as-ntp=1

[fanzh]

1. about “ROI should have a limited size.” , there is no special limited size, please refer to official doc:
   Gst-nvdspreprocess (Alpha) — DeepStream 6.1.1 Release documentation
2. about “the infer box located more righthand.”, please provide a simplified code to reproduce this issue, you can use deepstream-preprocess-test to test, and nvpreprocess is opensource, please refer to /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess

[user53799]

Thanks.

1. I am glad to know there’s no limitation of ROI size.
2. I am trying to describe what I met with the preprocess pluging.
   My source is a h264 .mp4 file, frame rate is 23, duration around 120s.
   I placed a function to calculate ROI size in the preprocess code, that means, the plugin firstly got rect_params from the config file, but not actually use it, insteadly, when transforming ROI to user_meta for OSD, it is using ROI size which was calculated by my function.
   During the 120s’ source, in the beginning 80s time, the caculated ROI is about 600x400, and the infered box is located where it should be.
   But during the remainning 40s time, the calculated ROI is about 800x600, and the infered box is not located correctly, say, if expected box left, top, width, height should be (300, 300, 400,400), actually the box will be (400, 450, 400, 400).
   That means, box size is correct, but left and top values are bigger.
   And I tested in setting ROI size fixed, the same with entire frame, the result is the same with the above last 40s time.

Should I configure more memory for the ROI transform function?

[fanzh]

@user53799

But during the remainning 40s time, the calculated ROI is about 800x600, and the infered box is not located correctly, say, if expected box left, top, width, height should be (300, 300, 400,400), actually the box will be (400, 450, 400, 400).
That means, box size is correct, but left and top values are bigger.

could you provide simplified code to reproduce this issue? including input video and configuration file.

[user53799]

Hi,
The following is my code to access the raw frame in preprocess plugin:

else {
preprocess_frame = get_preprocess_frame_meta->second;

    roi_vector = preprocess_frame.roi_vector;

    GST_DEBUG_OBJECT (nvdspreprocess, "Group %d : Source ID %d : Got roi-vecsize = %ld\n",
        gcnt, source_id, roi_vector.size());

    // std::cout<<"preprocess_frame.roi_vector.size()=="<<preprocess_frame.roi_vector.size()<<std::endl; //==1
    for (guint n = 0; n < preprocess_frame.roi_vector.size(); n++) {
      roi_meta = roi_vector[n];

      if (preprocess_group->process_on_roi) {
        GST_DEBUG_OBJECT (nvdspreprocess, "Group %d : Source ID %d : Processing on ROIS\n", gcnt, source_id);

        /** Process on ROIs provided from config file */
        rect_params = roi_meta.roi;

        // NvDsMetaList * l_frame = NULL;
        // NvDsMetaList * l_user_meta = NULL;
        // cv::Mat out_mat, ima;
        // Get original raw data
        GstMapInfo in_map_info;
        char* src_data = NULL;
        NvBufSurface *surface = NULL;
        memset (&in_map_info, 0, sizeof (in_map_info));
      
        if (!gst_buffer_map (inbuf, &in_map_info, GST_MAP_READ)) {
              g_print ("Error: Failed to map gst buffer\n");
              gst_buffer_unmap (inbuf, &in_map_info);
              return GST_FLOW_ERROR;
          }
          surface = (NvBufSurface *)in_map_info.data;
          // NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (inbuf);
          
        // #ifdef PLATFORM_TEGRA
            NvBufSurfaceMap (surface, -1, -1, NVBUF_MAP_READ);
            NvBufSurfaceSyncForCpu (surface, -1, -1);
            // NvBufSurfaceUnMap (surface, -1, -1);
            // printf("surface->numFilled = %d\n",surface->numFilled);//===1
            for (uint frameIndex = 0; frameIndex < surface->numFilled; frameIndex++) {
            // void *src_data = NULL;
            // src_data = (char *)malloc(surface->surfaceList[frameIndex].dataSize);
            // if (src_data == NULL) {
            // g_print("Error: failed to malloc src_data \n");
            // }
            
            gint frame_width = (gint)surface->surfaceList[frameIndex].width;
            gint frame_height = (gint)surface->surfaceList[frameIndex].height;
            size_t frame_step = surface->surfaceList[frameIndex].pitch;
            // g_print("frame ch: %d\n",frame.channels()); //===1
            // g_print("H:   %d         ",frame_width); //====512
            // g_print("W:  %d\n",frame_height); //====800
            // printf("surface colorformat =%d\n", surface->surfaceList[frameIndex].colorFormat);

           
            frame = cv::Mat(frame_height, frame_width, 
            CV_8UC1, surface->surfaceList[frameIndex].mappedAddr.addr[0], 
            frame_step);

            
            // g_print("H:   %d",frame_width); //====512
            // g_print("      W:  %d\n",frame_height); //====800
            // g_print("      pitch:  %ld\n",frame_step); 
            // cv::Mat out_mat = cv::Mat(cv::Size(frame_width, frame_height), CV_8UC3);
            // cv::cvtColor(frame, out_mat, cv::COLOR_RGB2BGR);
            // char yuv_name[100] = "";
            // sprintf(yuv_name, "yuv_%5d.png", nump); 
            nump++;
            // cv::imwrite(yuv_name, out_mat);
            }
          p_roi = calc_roi(frame);
        
          NvBufSurfaceUnMap (surface, -1, -1);                         
          // printf("unmapped\n");
          // std::cout<<"frame ==\n"<<frame<<std::endl;

         if(src_data != NULL) {
                free(src_data);
                src_data = NULL;
                // delete *out_mat;
                // delete *frame;
          }
        
        // NvBufSurfaceUnMap (surface, -1, -1);
        
          // cv::cvtColor(out_mat, ima, cv::COLOR_BGR2GRAY);
          // std::cout<< "current_frame_width, heights: " <<out_mat.cols << "x"<<out_mat.rows << "channels:"<< out_mat.channels() <<std::endl;
       
          // p_roi = calc_roi(out_mat);
          
         rect_params.left = p_roi[0];  rect_params.top = p_roi[1]; rect_params.width = p_roi[2]; rect_params.height = p_roi[3];
          // // g_print("filling ROI left=%f top=%f width=%f height=%f\n",
          // rect_params.left, rect_params.top, rect_params.width, rect_params.height);

         gst_buffer_unmap (inbuf, &in_map_info);
          // NvBufSurfaceUnMap(in_surf, -1, -1);        
          // g_print("rect params: %f %f  %f %f ",rect_params.left, rect_params.top, rect_params.width, rect_params.height);

calc_roi() is my function to calculate a new “left, top, width, height” parameter for preprocess plugin to transform.

my config:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=800
height=512
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/…/streams/sample_1080p_h264.mp4
uri=file:///home/shisun/work/81fd5.mp4
#uri=file:///home/shisun/81fd5_crop3.mp4
num-sources=1
#drop-frame-interval=2
gpu-id=0

(0): memtype_device - Memory type Device

(1): memtype_pinned - Memory type Host Pinned

(2): memtype_unified - Memory type Unified

cudadec-memtype=0

[source1]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://192.168.55.100:554
#uri=file://…/…/streams/sample_1080p_h264.mp4
num-sources=1
gpu-id=0
#camera-width=1280
#camera-height=720
#camera-fps-n=60
#camera-fps-d=1
#camera-v4l2-dev-node=0

(0): memtype_device - Memory type Device

(1): memtype_pinned - Memory type Host Pinned

(2): memtype_unified - Memory type Unified

#nvbuf-memory-type=0
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 7=nv3dsink (Jetson only)
type=2
sync=1
source-id=0
gpu-id=0
nvbuf-memory-type=0

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=1
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000

Set muxer output width and height

width=1280
height=720
#width=800
#height=512
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

If set to TRUE, system timestamp will be attached as ntp timestamp

If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached

attach-sys-ts-as-ntp=1

[pre-process]
enable=1
#config-file=config_preprocess.txt
config-file=…/myconfigs/my_config_pre.txt

config-file property is mandatory for any gie section.

Other properties are optional and if set will override the properties set in

the infer config file.

[primary-gie]
enable=1
gpu-id=0
input-tensor-meta=1

model-engine-file=…/…/models/Primary_Detector/gh_fp16.trt
#Required to display the PGIE labels, should be added even when using config-file
#property
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
#Required by the app for SGIE, when used along with config-file property
gie-unique-id=1
nvbuf-memory-type=0
#config-file=config_infer_primary.txt
config-file=…/myconfigs/gh_config_infer_primary.txt

[tests]
file-loop=0

Thank you.

[user53799]

Hi,
Please ignore the above issue.
Now I found the problem is I had not pass all the parameters to the transformation function of the preprocess plugin.
Thanks

[fanzh]

Glad to know you fixed it, thanks for the update! If need further support, please open a new one. Thanks