Can not run inference with bitmap image in Deepstream 6.0, TRT8.2 and JP4.6.3?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): Jetson
• DeepStream Version: 6.0
• JetPack Version (valid for Jetson only): 4.6.3
• TensorRT Version: 8.2
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I have bitmap images, I used repo for YOLOv7 https://github.com/NVIDIA-AI-IOT/yolo_deepstream/tree/main/deepstream_yolo but I encounter error when running inference on bitmap images as follow

**PERF:  FPS 0 (Avg)	
**PERF:  0.00 (0.00)	
** INFO: <bus_callback:194>: Pipeline ready

** ERROR: <cb_newpad:235>: Failed to link decodebin to pipeline
** INFO: <bus_callback:180>: Pipeline running

For jpeg images I could run inference. What I need to do?

2

When I running inference bitmap image on Jetpack5.1, DS6.2, TRT8.5 (XavierNX), different error was happen

Screenshot from 2023-07-17 17-50-37
Screenshot from 2023-07-17 17-50-371288×557 51.4 KB

4

Can you share your configure file and some bitmap images cause this error ?

5

I used repo https://github.com/NVIDIA-AI-IOT/yolo_deepstream/tree/main/deepstream_yolo and change these line for one image
#type=3 #uri=file:/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 type=2 uri=file://<path-to-image>
I checked for jpeg image with same config file, it worked, but it did not work with bitmap image.
test.bmp (798.8 KB)

6

Because jpeg is yuv420 usually. Hardware decoders usually support this format.

bmp format is BGR format, so need transform this format to nv12.

add the following item to [source0] group in your deepstream-app configure file

video-format=NV12

Thanks

7

Thanks.
Do I need to change model_color_format from 0 to 1 for bitmap images?

8

When I added this item to the [source0], the error is the same in JP4.6.3, TRT8.2, DS6.0 (Jetson) and PC (Docker TRT8.2, DS6.0)

Screenshot from 2023-07-19 08-58-18
Screenshot from 2023-07-19 08-58-181016×303 31.4 KB

video-format seem not supported in DS6.0. Here is the warning

** WARN: <parse_source:615>: Unknown key 'video-format' for group [source0]
9

No need, unless your model is trained using BGR.

Can you update to version 6.2 ? If you meet another bug, old versions are not well supported.

10

I checked in PC, I can run inference on bitmap image without video-format, but when I add video-format=NV12 for bitmap images, acc of mAP is changed much? How do you think what reason of difference?

I have not checked in Jetson board.

11

It relates with you model.
If you model trained by BGR，but you infer with NV12 .
I think this change is normal.

12

I trained model by RGB (jpeg images). I means that, when I use this model for 2 configs (one config with video-format=NV12, one config without video-format) on same bitmap images. The mAP difference is big.

13

When I input bmp image, If without video-format, It’s can’t run inference.

How did you compare the mAP? Is same of Input data?

Make surce use the target device.

14

Sorry for lack of information.
I use Docker container in PC with DS6.2 and TRT8.5, and I can run inference on bitmap images. But when setting with and without video-format=NV12 (otherwise is same), there is a large mAP difference. My model is trained with RGB images.

15

This is because of the difference in the data pipeline

with video-format=NV12
bmp(bgr24) → nv12 → rgb → nvinfer

without video-format=NV12
bmp(bgr24) → rgb → nvinfer

Different data processing led to discrepancies in the results

video-format=NV12 is necessary on Jetson.

If you want run it on GPU，reduce unnecessary data conversion.

16

Thank you so much for recommendation.

17

@junshengy
I have one more questions. If my image (jpeg) have size equal to size of engine model. Does Deepstream preprocess anything except for decoding?

jpeg => rgb (decoding, NVIDIA) close source) => nvinfer
I saw that my image before TRT inference is different with my original image (quality if lower). Thanks.

18

Due to hardware limitations, width alignment is necessary.

However I believe this does not result in lower image quality usually.

Did you dump the data into png ？try save as rgb ?

19

Did you dump the data into png ？

Yes, I dumped data into png.

try save as rgb ?

You means that I need to try dumping data to .jpg or jpeg to see the difference. Is that right?

20

I mean compression affects image quality. save to rgb avoid compression

21

Sorry, I am not much understand. I understand that we can dump data into image file, so that we can visualize and compare. I followed this guide https://forums.developer.nvidia.com/uploads/short-url/byXzsrCQMrPDx5qTGtfLiZhCUBO.txt This guide dumps data into .png format.