• Hardware Platform (Jetson / GPU): GPU • DeepStream Version: 7.0 • JetPack Version (valid for Jetson only): NA • TensorRT Version: 8.6.1.6 • NVIDIA GPU Driver Version (valid for GPU only): 550.90.07 • Issue Type( questions, new requirements, bugs): questions • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
I have developed a DeepStream pipeline for object detection on video with ROI application. The pipeline consists of nvdspreprocess, which crops the ROI and normalizes the image, followed by nvinfer, which runs a custom YOLO model for detection.
The detection results are accurate for objects entirely within the ROI. However, my use case requires obtaining bounding box coordinates for objects that are partially outside the ROI. My model is trained to predict the shape of an object even when only part of it is visible. The model performs well outside the DeepStream pipeline and accurately predicts bounding boxes extending beyond the ROI boundaries.
Unfortunately, within the DeepStream pipeline, nvinfer clips the bounding boxes to the ROI size, despite setting the property crop-objects-to-roi-boundary=0. I believe this property should disable bounding box clipping, allowing them to be returned as predicted by the model.
Could you please advise on how to prevent bounding box clipping within nvinfer and obtain the full bounding box coordinates, even when they extend beyond the ROI?
For clarification, I have confirmed that my detection parsing function returns bounding boxes extending beyond the ROI boundaries, as expected. Therefore, it seems the clipping occurs somewhere within nvinfer.
do you mean setting crop-objects-to-roi-boundary to 0 nvinfer still clips the bounding boxes to the ROI size? if so, how did you set? please add crop-objects-to-roi-boundary=0 to nvinfer’s configuration file. you also can add log in attach_metadata_detector of \opt\nvidia\deepstream\deepstream-7.0\sources\gst-plugins\gst-nvinfer\gstnvinfer_meta_utils.cpp to check if nvinfer->crop_objects_to_roi_boundary is 0. especially you need to rebuild and replace the so after modifying.
I’ve set crop-objects-to-roi-boundary=0 in nvinfer’s configuration file and despite of that the bounding boxes are still clipped to the ROI size.
Following your debugging advice, I logged the value of nvinfer->crop_objects_to_roi_boundary within gstnvinfer_meta_utils.cpp. Surprisingly, it reads as 1, regardless of the value specified in the nvinfer config file.
To ensure the configuration file was parsed correctly, I explored several other nvinfer parameters in gstnvinfer_meta_utils.cpp, and their values matched my settings in the config file. This suggests that the config file parsing is functioning properly, but the crop-objects-to-roi-boundary setting is not being applied for some unknown reason.
I can’t reproduce this issue on dgpu with DS7.0. here is my test. is there any difference with your test?
without any code modification，I added crop-objects-to-roi-boundary=0 in dstest1_pgie_config.txt of /opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-test1.
add “printf(“nvinfer->crop_objects_to_roi_boundary:%d\n”, nvinfer->crop_objects_to_roi_boundary);” in attach_metadata_detector of /opt/nvidia/deepstream/deepstream-7.0/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp, then rebuild, then copy so to /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_infer.so
execute “./deepstream-test1-app /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264”, then the app printed “nvinfer->crop_objects_to_roi_boundary:0”.
please refer to my last comment. I used deepstream-test1 which does not include nvdspreprocess plugin. and I already used dstest1_pgie_config.txt because “./deepstream-test1-app /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264” will use dstest1_pgie_config.txt. did I miss something?
In my case, I use nvdspreprocess to crop several small ROIs from input video stream, which has high resolution (4k) in my application, while I need to make detection predictions on several smaller areas. At the same time, I need to have predicted bounding boxes edges beyond the ROI boundaries. I confirmed raw predictions from my model contains correct bounding boxes, so just looking for a way to propagate them through DeepStream pipeline.
@fanzh Thank you for your comments above. I’ll try it and, if still required, I’ll share a reproducible example with nvdspreprocess.
@fanzh After some further investigation, i believe the issue is in attach_metadata_detector of /opt/nvidia/deepstream/deepstream-7.0/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp
The bounding box clipping process in the function runs if the following condition is true: if(nvinfer->input_tensor_from_meta && nvinfer->clip_object_outside_roi). Note, it is clip_object_outside_roi (which has default value of True) not crop-objects-to-roi-boundary, so the predictions are clipped regardless of crop-objects-to-roi-boundary value, as I reported in the beginning of the thread.
I confirmed that if I manually set nvinfer->clip_object_outside_roi to false and recompile libnvdsgst_infer.so, the results are correct for my usecase, but it does not look like the proper way to control DeepStream.
Gst-nvinfer documentation does not mention clip_object_outside_roi. What is the correct way to set the value?
In addition, documentation on crop-objects-to-roi-boundary is confusing since it doesn’t affect the results, despite the description.
Thanks for the sharing! nvinfer plugin is opensource. clip_object_outside_roi is controlled by “clip-object-outside-roi” property.
crop-objects-to-roi-boundary’s ROI is related to nvifner’s roi-top-offset and roi-bottom-offset. it is not related to preprocess’s ROI.