Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) GeForce RTX 3080 • DeepStream Version 7.1 • JetPack Version (valid for Jetson only) • TensorRT Version • NVIDIA GPU Driver Version (valid for GPU only) 550.144.03 • Issue Type( questions, new requirements, bugs) question
**
I was debugging my SGIE inference and I wanted to make sure that have exactly the same results in DS and in python script.**
**I implemented nvinfer resize in python, it works, but for some sizes it still generates additional artefacts of resizing that change inference results.
Seems like some rounding in the resizing kernel or smthlike that, I tried to understand which sizes don’t give that effect (see heatmap). Clearly, even sizes have a much lower chance of having those artefacts, but still not 100%
I tried to roundup to the next bbox size that ends by 4 and got a second heatmap (with much lower differences, many zeros).
Can you give any advice on how to completely avoid the difference in resizing between python and nvinfer?**
I modified nvinfer to dump resized images before inference
To work in a secondary and to avoid any resize previous to nvinfer I created every /code/inference_images_fake/frame_%05d.png as a 800x600 black image, and added chess pattern at 0,0 of difference sizes similar to images i got in my production.
Then I create an object in metadata that has a bbox 0,0,w,h
Are you using OpenCV for resizing? nvinfer’s resizing is not compatible with OpenCV. Please refer to this formula.
I recommend using nvpreprocess to implement a custom CUDA kernel to ensure that DeepStream and your Python script have the same pre-processing. The pipeline will be like this
If nvinfer property input-tensor-meta is set to true, nvinfer’s image to tensor transformation will not be used, and the transformation process of nvdspreprocess will be used.
@junshengy I am using a custom python function that mimics nvinfer resize with padding, and as you see in the examples I have multiple images that have 0 difference after resizing for different crop sizes, so my question is if there are specific crop image sizes that I can respect to avoid those differences in resizing?
If your input image width and height are the same as the model input, then nvinfer will not scale it. You need to keep nvstreamux consistent with this, and nvstreammux will also scale the image. If avoid scaling by these plugins, I believe they should behave the same as python
I avoid resizing by nvstreammux, because every input image is exactly of nvstreammux size, and then I add a fake object in metadata with a bbox 0,0,w,h.
So the only resize is from nvinfer.
In the pairs of images in the initial post you see python_nvinfer_simulation, dump_from_deepstream, their difference
@junshengy After resizing bboxes to end on 4 (e.g., 78x61 → 84x64), I have the majority of crops that match the Python version. But then I have some of them with these artefacts in DS (images with non-zero diff)
In your pipeline, nvinfer is used as pgie. If the input frame width and height match the model width and height, no scaling occurs.
However, from your description, nvinfer is used as sgie, and the input is an object, so scaling is inevitable.
Are you saying that the difference between Python and nvinfer when using object input results in decreased accuracy?
2.If you are comparing object scaling, nvinfer will call NvBufSurfTransformAsync to use hardware scaling. How do you implement nvinfer resize using python?
3. If you want to be completely consistent in python and nvinfer, I think you can only use nvdspreprocess to customize CustomAsyncTransformation and CustomTensorPreparation. The former is scaling/extracting channels, and the latter is turning image into tensor.
4. Does your model suffer from a decrease in accuracy due to different scaling algorithms? Did you dump the image after scaling using nvinfer for comparison? Or did you use some other method?
No, it is in a secondary mode. As I explained in a couple of previous messages, I use nvinfer in secondary mode and I add objects to metadata in a Probe
the crops are small from 50x50 to 140x140, and sometimes the difference of resizing played an important role, but for many images it was small.
How do you implement nvinfer resize using python?
def nvinfer_emulated_scaling(
image,
target_width,
target_height,
maintain_aspect_ratio=True,
normalize=True,
symmetric_padding=False,
interpolation=cv2.INTER_NEAREST):
h, w = image.shape[:2]
if maintain_aspect_ratio:
# Compute scaling factor
scale = min(target_width / w, target_height / h)
new_w, new_h = int(w * scale), int(h * scale)
resized = cv2.resize(image, (new_w, new_h), interpolation=interpolation)
# Calculate padding
pad_w = target_width - new_w
pad_h = target_height - new_h
if symmetric_padding:
# Equal padding on both sides (if odd, adds extra padding to bottom/right)
top = pad_h // 2
bottom = pad_h - top
left = pad_w // 2
right = pad_w - left
else:
# Align top-left corner (padding mostly on bottom/right)
top, left = 0, 0
bottom, right = pad_h, pad_w
padded = cv2.copyMakeBorder(
resized,
top,
bottom,
left,
right,
borderType=cv2.BORDER_CONSTANT,
value=[0, 0, 0] # Black padding
)
else:
# Simple resize
padded = cv2.resize(image, (target_width, target_height), interpolation=cv2.INTER_NEAREST)
return padded
Does your model suffer from a decrease in accuracy due to different scaling algorithms? Did you dump the image after scaling using nvinfer for comparison? Or did you use some other method?
Yes, I dump already resized images from inside nvinfer to make comparison of preprocessed input images that are going to the model
As I mentioned above, the two are not full compatible.
If you need to be completely consistent with OpenCV, please use nvdspreprocess + custom cuda kernel.This may still cause slight differences due to some floating point operations, but usually does not cause loss of precision.
refer to /opt/nvidia/deepstream/deepstream-8.0/sources/gst-plugins/gst-nvdspreprocess/nvdspreprocess_lib
As I mentioned above, the two are not full compatible.
Can you explain to me why I have 90% of matching resized images between python resize and nvinfer resize when I control for the bbox size (I change size to finish on 4), but it is not 100% match, what could be the rule for crop size to get the same resize?
At this point, python resize and nvinfer resize are more compatible than not, but I am still have some crops that were resized differently and I need some ideas of why that happens. Maybe you can provide some ideas of why it is happening like that?
This is caused by the difference between the nvbufsurface scaling algorithm (CUDA) and opencv (CPU). These codes are not open source, so no more details can be provided. Please refer to the solution mentioned above.
@junshengy Thanks for the quick replies. I will update this thread if I identify a hack that will help me remove differences in 100% of resized crops, right now I still have differences in some (10-15%) of the resized crops.
