YOLOV4 - TensorRT int8 inference in Python

Please provide the following information when requesting support.

• Hardware (V100)
• Network Type (Yolo_v4-CSPDARKNET-19)
• TLT 3.0

Cuda - 11.1
Cudnn -8.0
TensorRT - 7.2.1
TensorRT-OSS - 7.2.1

I have trained and tested a TLT YOLOv4 model in TLT3.0 toolkit. I further converted the trained model into a TensorRT-Int8 engine.
So far, I’m able to successfully infer the TensorRT engine inside the TLT docker. However, when I try to infere the engine outside the TLT docker, I’m getting the below error.

Inference result outside TLT docker:

[TensorRT] VERBOSE: Registered plugin creator - ::GridAnchor_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::NMS_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Reorg_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Region_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Clip_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::LReLU_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::PriorBox_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Normalize_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::RPROI_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::BatchedNMS_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::FlattenConcat_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::CropAndResize version 1
[TensorRT] VERBOSE: Registered plugin creator - ::DetectionLayer_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Proposal version 1
[TensorRT] VERBOSE: Registered plugin creator - ::ProposalLayer_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::PyramidROIAlign_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::ResizeNearest_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Split version 1
[TensorRT] VERBOSE: Registered plugin creator - ::SpecialSlice_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::InstanceNormalization_TRT version 1
[TensorRT] INTERNAL ERROR: Assertion failed: d == a + length
batchedNMSPlugin.cpp:70
Aborting…

Aborted (core dumped)

Inference results inside the TLT docker:

[TensorRT] VERBOSE: Registered plugin creator - ::BatchTilePlugin_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::BatchedNMS_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::CoordConvAC version 1
[TensorRT] VERBOSE: Registered plugin creator - ::CropAndResize version 1
[TensorRT] VERBOSE: Registered plugin creator - ::CropAndResizeDynamic version 1
[TensorRT] VERBOSE: Registered plugin creator - ::DetectionLayer_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::FlattenConcat_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::GenerateDetection_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::GridAnchor_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::GridAnchorRect_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::InstanceNormalization_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::LReLU_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::NMS_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::NMSDynamic_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Normalize_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::PriorBox_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::ProposalLayer_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Proposal version 1
[TensorRT] VERBOSE: Registered plugin creator - ::ProposalDynamic version 1
[TensorRT] VERBOSE: Registered plugin creator - ::PyramidROIAlign_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Region_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Reorg_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::ResizeNearest_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::RPROI_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::SpecialSlice_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Split version 1
[TensorRT] VERBOSE: Deserialize required 2559933 microseconds.
(1, 3, 512, 512)
[TensorRT] VERBOSE: Allocated persistent device memory of size 46515712
[TensorRT] VERBOSE: Allocated activation device memory of size 25719296
[TensorRT] VERBOSE: Assigning persistent memory blocks for various profiles
OUTPUT
[Host:
[106]
Device:
<pycuda._driver.DeviceAllocation object at 0x7f11443b7bc0>, Host:
[0.05834961 0.5854492 0.16235352 0.62646484 0.36401367 0.5722656
0.41479492 0.6279297 0.6274414 0.5878906 0.7192383 0.6230469
0.43920898 0.40356445 0.48510742 0.42456055 0.23095703 0.72802734
0.25708008 0.75439453 0.4411621 0.38671875 0.4831543 0.4116211
0.44555664 0.3725586 0.47875977 0.3959961 0.20581055 0.72021484
0.27075195 0.7504883 0.23730469 0.7285156 0.27197266 0.7548828
0.02839661 0.86083984 0.06585693 0.9243164 0.79296875 0.8364258
0.8154297 0.8676758 0.61376953 0.60595703 0.7182617 0.6430664
0.41503906 0.4008789 0.49902344 0.43603516 0.43920898 0.40356445
0.48510742 0.42456055 0.45629883 0.38598633 0.49438477 0.41186523
0.19335938 0.21325684 0.26953125 0.25683594 0.46069336 0.37060547
0.4909668 0.39697266 0.62109375 0.7578125 0.72558594 0.7832031
0.05090332 0.8935547 0.09375 0.95214844 0.02662659 0.8857422
0.07006836 0.9482422 0.15539551 0.59472656 0.25 0.62402344
0. 0.6791992 0.06762695 0.74365234 0.02627563 0.5888672
0.05093384 0.6123047 0.26782227 0.44433594 0.3762207 0.47314453
0.16748047 0.20043945 0.23999023 0.2685547 0.19335938 0.21325684
0.26953125 0.25683594 0.1508789 0.2479248 0.25463867 0.27929688
0.21582031 0.7084961 0.25805664 0.76123047 0.23071289 0.7133789
0.25561523 0.7426758 0.16748047 0.20043945 0.23999023 0.2685547
0.265625 0.67871094 0.33007812 0.72558594 0.62939453 0.5864258
0.7163086 0.6245117 0.7734375 0.82910156 0.8183594 0.8847656
0.10571289 0.58935547 0.22241211 0.6225586 0.02593994 0.8334961
0.06640625 0.8911133 0.04418945 0.8701172 0.09790039 0.9189453
0.21081543 0.7167969 0.31079102 0.7519531 0.2915039 0.6791992
0.3618164 0.72509766 0.02378845 0.7080078 0.0748291 0.7519531
0.36816406 0.57373047 0.41308594 0.62646484 0.04589844 0.70654297
0.09729004 0.75634766 0.62890625 0.5883789 0.7207031 0.6225586
0.41796875 0.17150879 0.49072266 0.23181152 0.02655029 0.5751953
0.05072021 0.5986328 0.0241394 0.5830078 0.07092285 0.6298828
0.44555664 0.3725586 0.47875977 0.3959961 0.21044922 0.6816406
0.26220703 0.73046875 0.14794922 0.7167969 0.26367188 0.74902344
0.24169922 0.71435547 0.27148438 0.7426758 0.01754761 0.6044922
0.08483887 0.68359375 0.29760742 0.44970703 0.34204102 0.47265625
0.67089844 0.68115234 0.73535156 0.72216797 0.4411621 0.38671875
0.4831543 0.4116211 0.06329346 0.61035156 0.20751953 0.6484375
0.32910156 0.25708008 0.48242188 0.2932129 0.1508789 0.2479248
0.25463867 0.27929688 0.0869751 0.59228516 0.18017578 0.65966797
0.074646 0.57177734 0.15698242 0.6166992 0.79003906 0.8496094
0.8183594 0.88183594 0.2199707 0.68408203 0.30981445 0.7241211
0.2421875 0.44555664 0.35253906 0.47387695 0.23999023 0.16381836
0.29223633 0.24365234 0.7915039 0.83447266 0.85595703 0.87646484
0.2536621 0.7114258 0.33666992 0.75146484 0.4272461 0.37231445
0.4658203 0.39379883 0.20996094 0.21936035 0.3149414 0.24963379
0.44213867 0.39746094 0.5058594 0.42773438 0.2902832 0.70947266
0.36547852 0.75341797 0.0769043 0.89941406 0.13500977 0.9472656
0.16748047 0.20043945 0.23999023 0.2685547 0.40625 0.2475586
0.5 0.28173828 0.39257812 0.5932617 0.5073242 0.6245117
0.46069336 0.37060547 0.4909668 0.39697266 0.3684082 0.57421875
0.41381836 0.6269531 0.06774902 0.5878906 0.1628418 0.625
0.6298828 0.625 0.76953125 0.66308594 0.62109375 0.7578125
0.72558594 0.7832031 0.5361328 0.5957031 0.6845703 0.62597656
0. 0.6352539 0.10925293 0.67529297 0.08703613 0.4729004
0.19592285 0.5058594 0.36816406 0.57373047 0.41308594 0.62646484
0.44628906 0.37402344 0.51708984 0.40966797 0.40966797 0.38842773
0.45166016 0.4074707 0.45629883 0.38598633 0.49438477 0.41186523
0.30444336 0.8730469 0.39331055 0.94628906 0.05099487 0.68310547
0.09436035 0.73291016 0.35302734 0.33569336 0.42822266 0.3791504
0.57958984 0.75390625 0.7036133 0.7832031 0.63427734 0.6826172
0.70947266 0.7207031 0.28051758 0.44995117 0.32983398 0.4699707
0.2401123 0.21313477 0.27172852 0.23901367 0.22436523 0.21118164
0.2590332 0.24145508 0.32470703 0.6801758 0.390625 0.7241211
0.34106445 0.5629883 0.40795898 0.6118164 0.37109375 0.24902344
0.4741211 0.28320312 0.42114258 0.3876953 0.47143555 0.40722656
0.25195312 0.7446289 0.3388672 0.78271484 0.05682373 0.9169922
0.09564209 0.9921875 0.7734375 0.80029297 0.82128906 0.86279297
0.22229004 0.3083496 0.31347656 0.34985352 0.3803711 0.37182617
0.4711914 0.40649414 0.62890625 0.5883789 0.7207031 0.6225586
0.63378906 0.57177734 0.70996094 0.61572266 0.83691406 0.765625
0.8876953 0.82128906 0.16748047 0.20043945 0.23999023 0.2685547
0.6142578 0.7753906 0.72753906 0.8076172 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. ]
Device:
<pycuda._driver.DeviceAllocation object at 0x7f11443b7cb0>, Host:
[0.9970703 0.9970703 0.9892578 0.01747131 0.00734711 0.00676346
0.00633621 0.0062027 0.00588226 0.00531006 0.00504303 0.00455856
0.00411987 0.00385094 0.00380325 0.00324059 0.00309944 0.00309372
0.00284576 0.00273132 0.00263596 0.00263214 0.00252724 0.00251961
0.00245094 0.00232506 0.0023098 0.00229073 0.00218964 0.00218582
0.00215149 0.00214958 0.00214005 0.00206566 0.00191498 0.00189495
0.00186729 0.0018568 0.00180817 0.0018034 0.001791 0.00178719
0.0017786 0.00169182 0.00164127 0.00161839 0.00161362 0.00160408
0.00159836 0.00157833 0.00157452 0.00155449 0.00154877 0.00143814
0.00142956 0.00141716 0.00139809 0.00138378 0.00137806 0.0013628
0.0013237 0.00132084 0.00130177 0.00126934 0.00126648 0.00126457
0.00125885 0.00125694 0.00121021 0.001194 0.00118637 0.00118637
0.00118446 0.00118256 0.0011673 0.00116158 0.00115681 0.00115013
0.00113583 0.00113487 0.00113392 0.00113106 0.00112057 0.00111485
0.001091 0.00108147 0.0010767 0.00107384 0.00106907 0.00105858
0.00105762 0.00105572 0.00105 0.00104904 0.00104427 0.0010376
0.00103378 0.00103092 0.00102711 0.00101852 0.00101852 0.00101471
0.0010128 0.00100803 0.00100613 0.00100613 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0.
0. 0. ]
Device:
<pycuda._driver.DeviceAllocation object at 0x7f11443b7da0>, Host:
[ 5. 5. 5. 4. 5. 4. 4. 5. 5. 5. 5. 5. 4. 5. 4. 5. 4. 5.
5. 5. 5. 5. 5. 5. 5. 4. 4. 5. 5. 4. 5. 4. 5. 5. 5. 5.
5. 5. 5. 2. 5. 2. 5. 5. 5. 5. 5. 5. 5. 5. 5. 5. 5. 5.
4. 5. 5. 5. 5. 5. 4. 5. 5. 5. 4. 5. 4. 5. 5. 1. 4. 5.
5. 10. 2. 5. 4. 5. 5. 5. 7. 4. 4. 5. 7. 5. 4. 5. 5. 5.
5. 5. 5. 5. 4. 4. 5. 5. 5. 4. 4. 0. 5. 5. 2. 5. -1. -1.
-1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
-1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
-1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
-1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
-1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1. -1.
-1. -1.]
Device:
<pycuda._driver.DeviceAllocation object at 0x7f11443b7e90>]

I have used the same inference script posted in the the below link.

Hi, Please refer to the below links to perform inference in INT8

Thanks!