How to use the NGC's engine file in TensorRT directly?

This is a general problem I found.
I can use TLT/TAO to convert the NGC models, like traffic detection/license plate detection/ … into engine file and used them in deepstream.
But how to use the converted engine file in TensorRT directly? what’s the input parameters and what’s the output? and how to interpret the output?
Where can I find these definitions/experations?

Many thanks!!
Zhen

For example, I can run the traffic detection model’s engine file in TensorRT and get following results. But how to interpret the outputs? where can I find the definition of the outputs? I didn’t see such explanation/descriptions in NGC model zoo/card:

In the trafficDetect model, this is what I am getting:

binding: input_1

kFLOAT

dims: 3 x 368 x 640

vol: 706560, elementSize: 4, allocationSize: 2826240

binding: conv2d_bbox

kFLOAT

dims: 16 x 23 x 40

vol: 14720, elementSize: 4, allocationSize: 58880

binding: conv2d_cov/Sigmoid

kFLOAT

dims: 4 x 23 x 40

vol: 3680, elementSize: 4, allocationSize: 14720

Again, it’s not clear what the output represents. This is what the output looks like with one image with a few cars:

“conv2d_bbox” :

            [

                            [

                                            0.37553250789642334,

                                            0.25411617755889893,

                                            1.8081748485565186,

                                            2.4971177577972412,

                                            2.7246339321136475,

                                            5.2109041213989258,

                                            4.3916349411010742,

                                            3.0153656005859375,

                                            1.6900120973587036,

                                            0.99474477767944336,

                                            1.3673833608627319,

                                            2.4700007438659668,

                                            3.9251670837402344,

                                            5.4030637741088867,

                                            4.2155003547668457,

                                            5.1327691078186035,

                                            3.8731472492218018,

                                            4.7320461273193359,

                                            4.3721423149108887,

                                            5.6336498260498047,

                                            4.8304123878479004,

                                            5.3013443946838379,

                                            6.7608680725097656,

                                            7.622647762298584,

                                            7.4046239852905273,

                                            6.951876163482666,

                                            6.662114143371582,

                                            6.5420718193054199,

                                            6.9765019416809082,

                                            7.6317691802978516,

                                            7.026648998260498,

                                            4.4997396469116211,

                                            4.0014843940734863,

                                            3.0577478408813477,

                                            5.3856639862060547,

                                            5.5625028610229492,

                                            6.3134055137634277,

                                            3.8527646064758301,

                                            4.2653031349182129,

                                            3.6340911388397217

                            ],

                            [

                                            0.11145077645778656,

                                            0.47865772247314453,

                                            1.0091991424560547,

                                            1.4688435792922974,

                                            2.0099656581878662,

                                            3.3028495311737061,

                                            3.4658184051513672,

                                            3.2606766223907471,

                                            2.3121426105499268,

                                            1.9813642501831055,

                                            2.3886487483978271,

                                            3.7232904434204102,

                                            4.6479077339172363,

                                            5.8145737648010254,

                                            4.367802619934082,

                                            5.6700048446655273,

                                            3.7717795372009277,

                                            4.6876373291015625,

                                            4.6663732528686523,

                                            5.5668816566467285,

                                            3.965975284576416,

                                            3.5384566783905029,

                                            3.844569206237793,

                                            4.2550663948059082,

                                            3.8582050800323486,

                                            3.0537006855010986,

                                            2.7069411277770996,

                                            3.0914814472198486,

                                            3.6053018569946289,

                                            3.9712233543395996,

                                            3.8412570953369141,

                                            2.3562884330749512,

                                            2.2699246406555176,

                                            2.2648329734802246,

                                            2.3757293224334717,

                                            2.8585410118103027,

                                            3.6631553173065186,

                                            2.1824607849121094,

                                            1.6553515195846558,

                                            0.97408908605575562

                            ],

                            [

                                            0.16651234030723572,

                                            0.50424414873123169,

                                            1.0075459480285645,

                                            1.5075206756591797,

                                            2.0013504028320312,

                                            4.7068471908569336,

                                            5.766749382019043,

                                            5.2430353164672852,

                                            4.1564016342163086,

                                            3.5500133037567139,

                                            4.7792859077453613,

                                            5.3000750541687012,

                                            6.6395149230957031,

                                            6.6344137191772461,

                                            6.1651053428649902,

                                            6.6798558235168457,

                                            5.0381608009338379,

                                            5.1383776664733887,

                                            5.4317021369934082,

                                            5.4807519912719727,

                                            4.7112746238708496,

                                            3.8498964309692383,

                                            2.5966532230377197,

                                            1.6107935905456543,

                                            1.2945710420608521,

                                            0.76179730892181396,

                                            0.60269665718078613,

                                            1.3123809099197388,

                                            2.2958011627197266,

                                            3.0635495185852051,

                                            3.5925803184509277,

                                            1.8364815711975098,

                                            1.6862297058105469,

                                            1.4531872272491455,

                                            2.3856611251831055,

                                            2.1176724433898926,

                                            1.9974226951599121,

                                            2.1669998168945312,

                                            0.73970550298690796,

                                            0.2758115828037262

                            ],

                            [

                                            0.089869275689125061,

                                            0.50334018468856812,

                                            0.97577822208404541,

                                            1.5391831398010254,

                                            1.9722321033477783,

                                            2.8090779781341553,

                                            3.3160016536712646,

                                            3.8920249938964844,

                                            3.350888729095459,

                                            2.9540276527404785,

                                            3.7688748836517334,

                                            4.1934881210327148,

                                            4.9885177612304688,

                                            5.0805854797363281,

                                            4.9324016571044922,

                                            5.0166912078857422,

                                            3.7501306533813477,

                                            3.2604146003723145,

                                            3.34494948387146,

                                            3.2237546443939209,

                                            3.071000337600708,

                                            2.8689033985137939,

                                            2.0365254878997803,

                                            1.430745005607605,

                                            1.7589083909988403,

                                            1.8229328393936157,

                                            0.86348223686218262,

                                            2.0518219470977783,

                                            3.3266472816467285,

                                            3.1988863945007324,

                                            3.6416921615600586,

                                            1.704717755317688,

                                            2.0143857002258301,

                                            2.0278351306915283,

                                            2.0850400924682617,

                                            1.7135282754898071,

                                            1.8675398826599121,

                                            2.324976921081543,

                                            -0.073807142674922943,

                                            0.33885616064071655

                            ],

                            [

                                            0.049470987170934677,

                                            0.39533129334449768,

                                            0.87610602378845215,

                                            1.3367726802825928,

                                            1.984976053237915,

                                            3.5710234642028809,

                                            4.4019184112548828,

                                            4.8477401733398438,

                                            3.2135438919067383,

                                            3.3046224117279053,

                                            4.004511833190918,

                                            4.0938172340393066,

                                            3.8485541343688965,

                                            3.9832882881164551,

                                            3.7735247611999512,

                                            3.5560111999511719,

                                            3.4615559577941895,

                                            2.9695377349853516,

                                            2.9193885326385498,

                                            3.0738966464996338,

                                            3.3257119655609131,

                                            3.7752132415771484,

                                            3.4475436210632324,

                                            3.7348065376281738,

                                            3.9022555351257324,

                                            3.6569046974182129,

                                            2.6442482471466064,

                                            3.7659633159637451,

                                            3.2640130519866943,

                                            3.999577522277832,

                                            4.3788747787475586,

                                            3.6223740577697754,

                                            3.472090482711792,

                                            1.9565601348876953,

                                            1.7352296113967896,

                                            1.9727785587310791,

                                            1.9825735092163086,

                                            1.6170506477355957,

                                            0.056207694113254547,

                                            0.12673789262771606

                            ],

(others omitted)

   "conv2d_cov/Sigmoid" : 

   [

          [

                 0.00011001244274666533,

                 0.00017482467228546739,

                 0.00016120450163725764,

                 7.8516466601286083e-05,

                 6.7483269958756864e-05,

                 6.9306646764744073e-05,

                 0.00010452463902765885,

                 8.6162770458031446e-05,

                 0.00011173765960847959,

                 0.00010786755592562258,

                 6.7971021053381264e-05,

                 8.1910657172556967e-05,

                 0.00011511237971717492,

                 9.0498630015645176e-05,

                 0.00014222772733774036,

                 9.2852744273841381e-05,

                 0.00011915895447600633,

                 0.00011290013935649768,

                 8.6653039033990353e-05,

                 7.4095456511713564e-05,

                 4.4680178689304739e-05,

                 4.2047613533213735e-05,

                 3.2062063837656751e-05,

                 6.1399136029649526e-05,

                 6.07419824518729e-05,

                 4.6951670810813084e-05,

                 4.0828221244737506e-05,

                 2.6788027753354982e-05,

                 3.2976477086776868e-05,

                 5.5078613513614982e-05,

                 8.6014842963777483e-05,

                 9.6402363851666451e-05,

                 2.165514524676837e-05,

                 4.7765664930921048e-05,

                 3.6815432395087555e-05,

                 8.2502512668725103e-05,

                 0.00019901797350030392,

                 0.00055602152133360505,

                 0.00015995874127838761,

                 0.00027070645592175424

          ],

          [

                 8.7052409071475267e-05,

                 0.00020446772396098822,

                 0.00042453053174540401,

                 0.00011240707681281492,

                 0.00016903399955481291,

                 8.6330393969547004e-05,

                 0.00022339718998409808,

                 4.6142657083692029e-05,

                 3.4663960832403973e-05,

                 2.8023714548908174e-05,

                 2.7388612579670735e-05,

                 3.939199450542219e-05,

                 5.954323205514811e-05,

                 7.7573531598318368e-05,

                 5.3120646043680608e-05,

                 2.9723409170401283e-05,

                 2.6421263100928627e-05,

                 3.8776561268605292e-05,

                 3.3891494240378961e-05,

                 4.3938121962128207e-05,

                 5.60066728212405e-05,

                 2.117583790095523e-05,

                 2.0784378648386337e-05,

                 1.3355869668885134e-05,

                 3.7136171158635989e-05,

                 6.3561026763636619e-05,

                 5.4118634579936042e-05,

                 0.00024277940974570811,

                 5.247547960607335e-05,

                 1.8547325453255326e-05,

                 7.5064685916004237e-06,

                 8.0046238508657552e-06,

                 5.5486539167759474e-06,

                 7.6058437116444111e-05,

                 3.6896835808875039e-05,

                 4.2021074477816001e-05,

                 2.2917907699593343e-05,

                 2.1706089682993479e-05,

                 3.4076027077389881e-05,

                 0.00055832427460700274

          ],

See inference with triton server. You can find the github in Integrating TAO CV Models with Triton Inference Server — TAO Toolkit 3.21.11 documentation.
And for all the tensorrt engine, you can run polygraphy to check the dimension,etc.
$ python -m pip install colored
$ python -m pip install polygraphy --index-url https://pypi.ngc.nvidia.com
$ polygraphy inspect model xxx.engine

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.