Output tensor values are different from Deepstream and keras, python TensorRt implementation

Accelerated Computing Intelligent Video Analytics DeepStream SDK
22

Thanks.
We are checking this internally.
Will share more information with you later.

23

sure, thanks

24

Hi,

We can run your source and reproduce the RMS issue internally.
Will share more information with you once we found anything.

Thanks.

26

sure, thank you

27

any update ?

28

Hi,

Thanks for your patience.

We can get the expected output from Deepstream with the source as below.
Would you mind checking this on your side as well?

    ...
    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        try:
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            l_user = frame_meta.frame_user_meta_list
            while l_user is not None:
                user_meta = pyds.NvDsUserMeta.cast(l_user.data)
                meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
                layer = pyds.get_nvds_LayerInfo(meta, 0)

                ptr = ctypes.cast(pyds.get_ptr(layer.buffer), ctypes.POINTER(ctypes.c_float))
                v = np.ctypeslib.as_array(ptr, shape=(3, 224,224))
                v = v.transpose((1, 2, 0))
                v = cv2.cvtColor(v, cv2.COLOR_RGB2BGR)
                #print(v)
                try:
                    l_user = l_user.next
                except StopIteration:
                    break

        except StopIteration:
            break

Thanks.

29

@AastaLLL thank you will implement and get back soon

30

@AastaLLL the code works fine i am getting the output right. But how is it done for a tensor in my case the 4 output logits of the mobilenetv2 model ?

31

Hi,

It should follow the NCHW rule.
Do you get the expected result with the following arrangement?

ptr = ctypes.cast(pyds.get_ptr(layer.buffer), ctypes.POINTER(ctypes.c_float))
v = np.ctypeslib.as_array(ptr, shape=(1,4))

Thanks.

32

will check and let you know

33

Hi,

Do you get the expected output with the update?
Thanks.

34

hey @AastaLLL didn’t get to check it. will do in a day or two and get back asap.

thank you

35

checked with the change and the RMSE is still high!

36

hey @AastaLLL can you replicate the issue on the jetson nano and try to infer and check if the problem is only from my end ?

37

Hi,

Sure. We are checking this to see the status of our environment.
Will share more information with you later.

38

Hi,

Sorry for the late reply.

Based on your scripts, we are able to get very close output between TensorRT and Deepstream.
Since the video decoder is different, it’s expected to have some minor differences and it won’t hurt the accuracy.

The experience is tested with the TensorRT default ResNet-50 model.
Please check the detailed steps below:

Generate data

$ /usr/src/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.trt
$ ffmpeg -loop 1 -i /usr/src/tensorrt/data/resnet50/tabby_tiger_cat.jpg -vframes 1000 -vcodec libx264 -crf 25 -pix_fmt yuv420p -t 5 test.h264

Generate output and compare

TRTInferance.py (2.3 KB)
deepstream_test_1.py (7.1 KB)
dstest1_pgie_config.txt (2.9 KB)
rmsmobinet1.py (585 Bytes)

$ python3 TRTInferance.py
$ python3 deepstream_test_1.py test.h264
$ python3 rmsmobinet1.py

Results

ds=
[3.6179506e-07 1.9918819e-01 4.8616490e-01 2.8783941e-05 5.0460119e-07
 3.1102151e-01 8.4696003e-06 1.3424859e-03 7.8613736e-04 1.4913004e-04]
trt=
[4.4984480e-07 8.2435668e-02 5.1276660e-01 2.0303991e-05 2.6858103e-07
 3.8613626e-01 4.7418266e-06 1.7026100e-02 9.7259018e-04 3.3475558e-04]
average rms :
0.0044365267020706955
max rms :
[0.00449751]

Thanks.

39

hey @AastaLLL thanks for the experiment. i will try to reproduce the same on my side with the script you have tried and will also try the same on mobilenetv2 as well.

thank you

41

hey @AastaLLL i replicated the same on my device and it seems to work fine with the resnet50 example model given from TensorRT but it gives a high rmse when i try it with the mobilenetV2 model.

thank you

42

Can you please verify if the keras to onnx conversion is right? and can you reproduce the same with MOBILENETV2 model that i have shared(with the onnx&trt conversion)?

43

Hi,

Actually, we want to try the model but it has been removed from the link (post on Nov 9).
So we use resnet50 instead as an example.
Would you mind sharing the model and the script to run it with TensorRT again?

The issue should not be related to the tf2onnx.
Since both TensorRT and Deepstream are based on the same ONNX file.

Thanks.