[pluginV2Runner.cpp::execute::265] Error Code 2: Internal Error (Assertion status == kSTATUS_SCUESS failed. )

srsjd · October 19, 2022, 8:45am

Hi,
I upgraded my tensorRT 7 plugin c++ code so it’s compatible to tensorRT 8, and I keep getting this error while building engine.

I’m hoping someone can tell me more about this error msg, or point me to documents that can explain it. Does it mean that the building process failed due while processing a pluginV2?

Any suggestion is highly appreciated. Thanks in advance.

spolisetty · October 20, 2022, 1:36pm

Hi,

Could you please give us more details.
TensorRT version, GPU, Platform, CUDA version, Driver version.
Please try the following and share with us complete logs and if possible minimum issue repro ONNX model and scripts.

github.com/NVIDIA/TensorRT

[TensorRT] ERROR: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed.)

opened 02:33PM - 31 Mar 22 UTC

closed 07:43AM - 15 Jun 22 UTC

ssseeex

Component: Plugins triaged

## Description Hi, I have build a model that has three inputs with tensorrt p…ython api layer by layer, and I develop a 3d grid sample plugin to build voxel. Everything runs well when building an engine, but when the engine is used fo inference there's an error thrown, Is anyone knows what's the problem. Part of the log is below: Layer(Constant): (Unnamed Layer* 329) [Constant], Tactic: 0, -> (Unnamed Layer* 329) [Constant]_output[Float(1,32,1)] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 331) [ElementWise], (Unnamed Layer* 332) [ElementWise]), (Unnamed Layer* 333) [ElementWise]), (Unnamed Layer* 334) [ElementWise]), Tactic: 6, (Unnamed Layer* 327) [Reduce]_output[Float(1,32,1)], (Unnamed Layer* 329) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 328) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 330) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 325) [ElementWise]_output[Float(1,32,199680)] -> (Unnamed Layer* 334) [ElementWise]_output[Float(1,32,199680)] Layer(Scale): (Unnamed Layer* 336) [Scale], Tactic: 0, (Unnamed Layer* 335) [Shuffle]_output[Float(1,64,160,624)] -> (Unnamed Layer* 336) [Scale]_output[Float(1,64,160,624)] Layer(ElementWise): (Unnamed Layer* 338) [ElementWise] + (Unnamed Layer* 339) [Activation], Tactic: 1, (Unnamed Layer* 337) [Resize]_output[Float(1,64,160,624)], (Unnamed Layer* 336) [Scale]_output[Float(1,64,160,624)] -> (Unnamed Layer* 339) [Activation]_output[Float(1,64,160,624)] Layer(Reduce): (Unnamed Layer* 512) [Reduce], Tactic: 1, (Unnamed Layer* 511) [Shuffle]_output[Float(1,32,199680)] -> (Unnamed Layer* 512) [Reduce]_output[Float(1,32,1)] Layer(ElementWise): (Unnamed Layer* 513) [ElementWise], Tactic: 1, (Unnamed Layer* 511) [Shuffle]_output[Float(1,32,199680)], (Unnamed Layer* 512) [Reduce]_output[Float(1,32,1)] -> (Unnamed Layer* 513) [ElementWise]_output[Float(1,32,199680)] Layer(Constant): (Unnamed Layer* 518) [Constant], Tactic: 0, -> (Unnamed Layer* 518) [Constant]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 516) [Constant], Tactic: 0, -> (Unnamed Layer* 516) [Constant]_output[Float(1,32,1)] Layer(Reduce): (Unnamed Layer* 514) [ElementWise] + (Unnamed Layer* 515) [Reduce], Tactic: 2, (Unnamed Layer* 513) [ElementWise]_output[Float(1,32,199680)] -> (Unnamed Layer* 515) [Reduce]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 517) [Constant], Tactic: 0, -> (Unnamed Layer* 517) [Constant]_output[Float(1,32,1)] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 519) [ElementWise], (Unnamed Layer* 520) [ElementWise]), (Unnamed Layer* 521) [ElementWise]), (Unnamed Layer* 522) [ElementWise]), Tactic: 6, (Unnamed Layer* 515) [Reduce]_output[Float(1,32,1)], (Unnamed Layer* 517) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 516) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 518) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 513) [ElementWise]_output[Float(1,32,199680)] -> (Unnamed Layer* 522) [ElementWise]_output[Float(1,32,199680)] Layer(Scale): (Unnamed Layer* 524) [Scale], Tactic: 0, (Unnamed Layer* 523) [Shuffle]_output[Float(1,64,160,624)] -> (Unnamed Layer* 524) [Scale]_output[Float(1,64,160,624)] Layer(ElementWise): (Unnamed Layer* 526) [ElementWise] + (Unnamed Layer* 527) [Activation], Tactic: 1, (Unnamed Layer* 525) [Resize]_output[Float(1,64,160,624)], (Unnamed Layer* 524) [Scale]_output[Float(1,64,160,624)] -> (Unnamed Layer* 527) [Activation]_output[Float(1,64,160,624)] Layer(CaskConvolution): (Unnamed Layer* 528) [Convolution], Tactic: 2775507031594384867, (Unnamed Layer* 527) [Activation]_output[Float(1,64,160,624)] -> (Unnamed Layer* 528) [Convolution]_output[Float(1,32,160,624)] Layer(CaskConvolution): (Unnamed Layer* 340) [Convolution], Tactic: 2775507031594384867, (Unnamed Layer* 339) [Activation]_output[Float(1,64,160,624)] -> (Unnamed Layer* 340) [Convolution]_output[Float(1,32,160,624)] Layer(Reduce): (Unnamed Layer* 342) [Reduce], Tactic: 1, (Unnamed Layer* 341) [Shuffle]_output[Float(1,32,99840)] -> (Unnamed Layer* 342) [Reduce]_output[Float(1,32,1)] Layer(ElementWise): (Unnamed Layer* 343) [ElementWise], Tactic: 1, (Unnamed Layer* 341) [Shuffle]_output[Float(1,32,99840)], (Unnamed Layer* 342) [Reduce]_output[Float(1,32,1)] -> (Unnamed Layer* 343) [ElementWise]_output[Float(1,32,99840)] Layer(Constant): (Unnamed Layer* 348) [Constant], Tactic: 0, -> (Unnamed Layer* 348) [Constant]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 346) [Constant], Tactic: 0, -> (Unnamed Layer* 346) [Constant]_output[Float(1,32,1)] Layer(Reduce): (Unnamed Layer* 344) [ElementWise] + (Unnamed Layer* 345) [Reduce], Tactic: 2, (Unnamed Layer* 343) [ElementWise]_output[Float(1,32,99840)] -> (Unnamed Layer* 345) [Reduce]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 347) [Constant], Tactic: 0, -> (Unnamed Layer* 347) [Constant]_output[Float(1,32,1)] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 349) [ElementWise], (Unnamed Layer* 350) [ElementWise]), (Unnamed Layer* 351) [ElementWise]), (Unnamed Layer* 352) [ElementWise]), Tactic: 5, (Unnamed Layer* 345) [Reduce]_output[Float(1,32,1)], (Unnamed Layer* 347) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 346) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 348) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 343) [ElementWise]_output[Float(1,32,99840)] -> (Unnamed Layer* 352) [ElementWise]_output[Float(1,32,99840)] Layer(Scale): (Unnamed Layer* 354) [Scale], Tactic: 0, (Unnamed Layer* 353) [Shuffle]_output[Float(1,32,160,624)] -> (Unnamed Layer* 354) [Scale]_output[Float(1,32,160,624)] Layer(Reduce): (Unnamed Layer* 530) [Reduce], Tactic: 1, (Unnamed Layer* 529) [Shuffle]_output[Float(1,32,99840)] -> (Unnamed Layer* 530) [Reduce]_output[Float(1,32,1)] Layer(ElementWise): (Unnamed Layer* 531) [ElementWise], Tactic: 1, (Unnamed Layer* 529) [Shuffle]_output[Float(1,32,99840)], (Unnamed Layer* 530) [Reduce]_output[Float(1,32,1)] -> (Unnamed Layer* 531) [ElementWise]_output[Float(1,32,99840)] Layer(Constant): (Unnamed Layer* 536) [Constant], Tactic: 0, -> (Unnamed Layer* 536) [Constant]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 534) [Constant], Tactic: 0, -> (Unnamed Layer* 534) [Constant]_output[Float(1,32,1)] Layer(Reduce): (Unnamed Layer* 532) [ElementWise] + (Unnamed Layer* 533) [Reduce], Tactic: 2, (Unnamed Layer* 531) [ElementWise]_output[Float(1,32,99840)] -> (Unnamed Layer* 533) [Reduce]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 535) [Constant], Tactic: 0, -> (Unnamed Layer* 535) [Constant]_output[Float(1,32,1)] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 537) [ElementWise], (Unnamed Layer* 538) [ElementWise]), (Unnamed Layer* 539) [ElementWise]), (Unnamed Layer* 540) [ElementWise]), Tactic: 5, (Unnamed Layer* 533) [Reduce]_output[Float(1,32,1)], (Unnamed Layer* 535) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 534) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 536) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 531) [ElementWise]_output[Float(1,32,99840)] -> (Unnamed Layer* 540) [ElementWise]_output[Float(1,32,99840)] Layer(Scale): (Unnamed Layer* 542) [Scale], Tactic: 0, (Unnamed Layer* 541) [Shuffle]_output[Float(1,32,160,624)] -> (Unnamed Layer* 542) [Scale]_output[Float(1,32,160,624)] Layer(Resize): (Unnamed Layer* 558) [Resize], Tactic: 1, (Unnamed Layer* 542) [Scale]_output[Float(1,32,160,624)] -> (Unnamed Layer* 558) [Resize]_output[Float(1,32,320,1248)] Layer(Resize): (Unnamed Layer* 370) [Resize], Tactic: 1, (Unnamed Layer* 354) [Scale]_output[Float(1,32,160,624)] -> (Unnamed Layer* 370) [Resize]_output[Float(1,32,320,1248)] Layer(Reduce): (Unnamed Layer* 357) [Reduce], Tactic: 1, (Unnamed Layer* 356) [Shuffle]_output[Float(1,32,399360)] -> (Unnamed Layer* 357) [Reduce]_output[Float(1,32,1)] Layer(ElementWise): (Unnamed Layer* 358) [ElementWise], Tactic: 1, (Unnamed Layer* 356) [Shuffle]_output[Float(1,32,399360)], (Unnamed Layer* 357) [Reduce]_output[Float(1,32,1)] -> (Unnamed Layer* 358) [ElementWise]_output[Float(1,32,399360)] Layer(Constant): (Unnamed Layer* 363) [Constant], Tactic: 0, -> (Unnamed Layer* 363) [Constant]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 361) [Constant], Tactic: 0, -> (Unnamed Layer* 361) [Constant]_output[Float(1,32,1)] Layer(Reduce): (Unnamed Layer* 359) [ElementWise] + (Unnamed Layer* 360) [Reduce], Tactic: 2, (Unnamed Layer* 358) [ElementWise]_output[Float(1,32,399360)] -> (Unnamed Layer* 360) [Reduce]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 362) [Constant], Tactic: 0, -> (Unnamed Layer* 362) [Constant]_output[Float(1,32,1)] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 364) [ElementWise], (Unnamed Layer* 365) [ElementWise]), (Unnamed Layer* 366) [ElementWise]), (Unnamed Layer* 367) [ElementWise]), Tactic: 6, (Unnamed Layer* 360) [Reduce]_output[Float(1,32,1)], (Unnamed Layer* 362) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 361) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 363) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 358) [ElementWise]_output[Float(1,32,399360)] -> (Unnamed Layer* 367) [ElementWise]_output[Float(1,32,399360)] Layer(Scale): (Unnamed Layer* 369) [Scale], Tactic: 0, (Unnamed Layer* 368) [Shuffle]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 369) [Scale]_output[Float(1,32,320,1248)] Layer(ElementWise): (Unnamed Layer* 371) [ElementWise] + (Unnamed Layer* 372) [Activation], Tactic: 1, (Unnamed Layer* 370) [Resize]_output[Float(1,32,320,1248)], (Unnamed Layer* 369) [Scale]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 372) [Activation]_output[Float(1,32,320,1248)] Layer(Reduce): (Unnamed Layer* 545) [Reduce], Tactic: 1, (Unnamed Layer* 544) [Shuffle]_output[Float(1,32,399360)] -> (Unnamed Layer* 545) [Reduce]_output[Float(1,32,1)] Layer(ElementWise): (Unnamed Layer* 546) [ElementWise], Tactic: 1, (Unnamed Layer* 544) [Shuffle]_output[Float(1,32,399360)], (Unnamed Layer* 545) [Reduce]_output[Float(1,32,1)] -> (Unnamed Layer* 546) [ElementWise]_output[Float(1,32,399360)] Layer(Constant): (Unnamed Layer* 551) [Constant], Tactic: 0, -> (Unnamed Layer* 551) [Constant]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 549) [Constant], Tactic: 0, -> (Unnamed Layer* 549) [Constant]_output[Float(1,32,1)] Layer(Reduce): (Unnamed Layer* 547) [ElementWise] + (Unnamed Layer* 548) [Reduce], Tactic: 1, (Unnamed Layer* 546) [ElementWise]_output[Float(1,32,399360)] -> (Unnamed Layer* 548) [Reduce]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 550) [Constant], Tactic: 0, -> (Unnamed Layer* 550) [Constant]_output[Float(1,32,1)] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 552) [ElementWise], (Unnamed Layer* 553) [ElementWise]), (Unnamed Layer* 554) [ElementWise]), (Unnamed Layer* 555) [ElementWise]), Tactic: 6, (Unnamed Layer* 548) [Reduce]_output[Float(1,32,1)], (Unnamed Layer* 550) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 549) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 551) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 546) [ElementWise]_output[Float(1,32,399360)] -> (Unnamed Layer* 555) [ElementWise]_output[Float(1,32,399360)] Layer(Scale): (Unnamed Layer* 557) [Scale], Tactic: 0, (Unnamed Layer* 556) [Shuffle]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 557) [Scale]_output[Float(1,32,320,1248)] Layer(ElementWise): (Unnamed Layer* 559) [ElementWise] + (Unnamed Layer* 560) [Activation], Tactic: 1, (Unnamed Layer* 558) [Resize]_output[Float(1,32,320,1248)], (Unnamed Layer* 557) [Scale]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 560) [Activation]_output[Float(1,32,320,1248)] Layer(CudnnConvolution): (Unnamed Layer* 561) [Convolution], Tactic: 6, (Unnamed Layer* 560) [Activation]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 561) [Convolution]_output[Float(1,32,320,1248)] Layer(CudnnConvolution): (Unnamed Layer* 373) [Convolution], Tactic: 6, (Unnamed Layer* 372) [Activation]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 373) [Convolution]_output[Float(1,32,320,1248)] Layer(Reduce): (Unnamed Layer* 375) [Reduce], Tactic: 1, (Unnamed Layer* 374) [Shuffle]_output[Float(1,32,399360)] -> (Unnamed Layer* 375) [Reduce]_output[Float(1,32,1)] Layer(ElementWise): (Unnamed Layer* 376) [ElementWise], Tactic: 1, (Unnamed Layer* 374) [Shuffle]_output[Float(1,32,399360)], (Unnamed Layer* 375) [Reduce]_output[Float(1,32,1)] -> (Unnamed Layer* 376) [ElementWise]_output[Float(1,32,399360)] Layer(Constant): (Unnamed Layer* 381) [Constant], Tactic: 0, -> (Unnamed Layer* 381) [Constant]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 379) [Constant], Tactic: 0, -> (Unnamed Layer* 379) [Constant]_output[Float(1,32,1)] Layer(Reduce): (Unnamed Layer* 377) [ElementWise] + (Unnamed Layer* 378) [Reduce], Tactic: 1, (Unnamed Layer* 376) [ElementWise]_output[Float(1,32,399360)] -> (Unnamed Layer* 378) [Reduce]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 380) [Constant], Tactic: 0, -> (Unnamed Layer* 380) [Constant]_output[Float(1,32,1)] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 382) [ElementWise], (Unnamed Layer* 383) [ElementWise]), (Unnamed Layer* 384) [ElementWise]), (Unnamed Layer* 385) [ElementWise]), Tactic: 6, (Unnamed Layer* 378) [Reduce]_output[Float(1,32,1)], (Unnamed Layer* 380) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 379) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 381) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 376) [ElementWise]_output[Float(1,32,399360)] -> (Unnamed Layer* 385) [ElementWise]_output[Float(1,32,399360)] Layer(Scale): (Unnamed Layer* 387) [Scale] + (Unnamed Layer* 388) [Activation], Tactic: 0, (Unnamed Layer* 386) [Shuffle]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 388) [Activation]_output[Float(1,32,320,1248)] Layer(Reduce): (Unnamed Layer* 563) [Reduce], Tactic: 1, (Unnamed Layer* 562) [Shuffle]_output[Float(1,32,399360)] -> (Unnamed Layer* 563) [Reduce]_output[Float(1,32,1)] Layer(ElementWise): (Unnamed Layer* 564) [ElementWise], Tactic: 1, (Unnamed Layer* 562) [Shuffle]_output[Float(1,32,399360)], (Unnamed Layer* 563) [Reduce]_output[Float(1,32,1)] -> (Unnamed Layer* 564) [ElementWise]_output[Float(1,32,399360)] Layer(Constant): (Unnamed Layer* 569) [Constant], Tactic: 0, -> (Unnamed Layer* 569) [Constant]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 567) [Constant], Tactic: 0, -> (Unnamed Layer* 567) [Constant]_output[Float(1,32,1)] Layer(Reduce): (Unnamed Layer* 565) [ElementWise] + (Unnamed Layer* 566) [Reduce], Tactic: 2, (Unnamed Layer* 564) [ElementWise]_output[Float(1,32,399360)] -> (Unnamed Layer* 566) [Reduce]_output[Float(1,32,1)] Layer(Constant): (Unnamed Layer* 568) [Constant], Tactic: 0, -> (Unnamed Layer* 568) [Constant]_output[Float(1,32,1)] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 570) [ElementWise], (Unnamed Layer* 571) [ElementWise]), (Unnamed Layer* 572) [ElementWise]), (Unnamed Layer* 573) [ElementWise]), Tactic: 6, (Unnamed Layer* 566) [Reduce]_output[Float(1,32,1)], (Unnamed Layer* 568) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 567) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 569) [Constant]_output[Float(1,32,1)], (Unnamed Layer* 564) [ElementWise]_output[Float(1,32,399360)] -> (Unnamed Layer* 573) [ElementWise]_output[Float(1,32,399360)] Layer(Scale): (Unnamed Layer* 575) [Scale] + (Unnamed Layer* 576) [Activation], Tactic: 0, (Unnamed Layer* 574) [Shuffle]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 576) [Activation]_output[Float(1,32,320,1248)] Layer(CaskConvolution): (Unnamed Layer* 577) [Convolution], Tactic: -3946921629105938337, (Unnamed Layer* 576) [Activation]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 577) [Convolution]_output[Float(1,32,320,1248)] Layer(CaskConvolution): (Unnamed Layer* 389) [Convolution], Tactic: -3946921629105938337, (Unnamed Layer* 388) [Activation]_output[Float(1,32,320,1248)] -> (Unnamed Layer* 389) [Convolution]_output[Float(1,32,320,1248)] Layer(Constant): (Unnamed Layer* 616) [Constant], Tactic: 0, -> (Unnamed Layer* 616) [Constant]_output[Float(72,1)] Layer(Constant): (Unnamed Layer* 617) [Constant], Tactic: 0, -> (Unnamed Layer* 617) [Constant]_output[Float(72,1)] Layer(ElementWise): (Unnamed Layer* 618) [ElementWise], Tactic: 1, (Unnamed Layer* 617) [Constant]_output[Float(72,1)], (Unnamed Layer* 616) [Constant]_output[Float(72,1)] -> (Unnamed Layer* 618) [ElementWise]_output[Float(72,1)] Layer(PluginV2): (Unnamed Layer* 620) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 389) [Convolution]_output[Float(1,32,320,1248)], (Unnamed Layer* 577) [Convolution]_output[Float(1,32,320,1248)], (Unnamed Layer* 619) [Shuffle]_output[Float(1,72,1,1)] -> (Unnamed Layer* 620) [PluginV2DynamicExt]_output_0[Float(64,72,80,312)] Layer(CudnnConvolution): (Unnamed Layer* 622) [Convolution], Tactic: 57, (Unnamed Layer* 621) [Shuffle]_output[Float(1,64,72,80,312)] -> (Unnamed Layer* 622) [Convolution]_output[Float(1,32,72,80,312)] Layer(Constant): (Unnamed Layer* 623) [Constant], Tactic: 0, -> (Unnamed Layer* 623) [Constant]_output[Float(32,1)] Layer(Constant): (Unnamed Layer* 624) [Constant], Tactic: 0, -> (Unnamed Layer* 624) [Constant]_output[Float(32,1)] Layer(PluginV2): (Unnamed Layer* 625) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 622) [Convolution]_output[Float(1,32,72,80,312)], (Unnamed Layer* 623) [Constant]_output[Float(32,1)], (Unnamed Layer* 624) [Constant]_output[Float(32,1)] -> (Unnamed Layer* 625) [PluginV2DynamicExt]_output_0[Float(1,32,72,80,312)] Layer(PointWiseV2): PWN((Unnamed Layer* 626) [Activation]), Tactic: 8, (Unnamed Layer* 625) [PluginV2DynamicExt]_output_0[Float(1,32,72,80,312)] -> (Unnamed Layer* 626) [Activation]_output[Float(1,32,72,80,312)] Layer(CudnnConvolution): (Unnamed Layer* 627) [Convolution], Tactic: 57, (Unnamed Layer* 626) [Activation]_output[Float(1,32,72,80,312)] -> (Unnamed Layer* 627) [Convolution]_output[Float(1,32,72,80,312)] Layer(Constant): (Unnamed Layer* 628) [Constant], Tactic: 0, -> (Unnamed Layer* 628) [Constant]_output[Float(32,1)] Layer(Constant): (Unnamed Layer* 629) [Constant], Tactic: 0, -> (Unnamed Layer* 629) [Constant]_output[Float(32,1)] Layer(PluginV2): (Unnamed Layer* 630) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 627) [Convolution]_output[Float(1,32,72,80,312)], (Unnamed Layer* 628) [Constant]_output[Float(32,1)], (Unnamed Layer* 629) [Constant]_output[Float(32,1)] -> (Unnamed Layer* 630) [PluginV2DynamicExt]_output_0[Float(1,32,72,80,312)] Layer(PointWiseV2): PWN(PWN((Unnamed Layer* 631) [Activation]), (Unnamed Layer* 632) [ElementWise]), Tactic: 8, (Unnamed Layer* 630) [PluginV2DynamicExt]_output_0[Float(1,32,72,80,312)], (Unnamed Layer* 626) [Activation]_output[Float(1,32,72,80,312)] -> (Unnamed Layer* 632) [ElementWise]_output[Float(1,32,72,80,312)] Layer(CudnnConvolution): (Unnamed Layer* 633) [Convolution], Tactic: 1, (Unnamed Layer* 632) [ElementWise]_output[Float(1,32,72,80,312)] -> (Unnamed Layer* 633) [Convolution]_output[Float(1,64,36,40,156)] Layer(Constant): (Unnamed Layer* 634) [Constant], Tactic: 0, -> (Unnamed Layer* 634) [Constant]_output[Float(64,1)] Layer(Constant): (Unnamed Layer* 635) [Constant], Tactic: 0, -> (Unnamed Layer* 635) [Constant]_output[Float(64,1)] Layer(PluginV2): (Unnamed Layer* 636) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 633) [Convolution]_output[Float(1,64,36,40,156)], (Unnamed Layer* 634) [Constant]_output[Float(64,1)], (Unnamed Layer* 635) [Constant]_output[Float(64,1)] -> (Unnamed Layer* 636) [PluginV2DynamicExt]_output_0[Float(1,64,36,40,156)] Layer(PointWiseV2): PWN((Unnamed Layer* 637) [Activation]), Tactic: 8, (Unnamed Layer* 636) [PluginV2DynamicExt]_output_0[Float(1,64,36,40,156)] -> (Unnamed Layer* 637) [Activation]_output[Float(1,64,36,40,156)] Layer(CudnnConvolution): (Unnamed Layer* 638) [Convolution], Tactic: 57, (Unnamed Layer* 637) [Activation]_output[Float(1,64,36,40,156)] -> (Unnamed Layer* 638) [Convolution]_output[Float(1,64,36,40,156)] Layer(Constant): (Unnamed Layer* 639) [Constant], Tactic: 0, -> (Unnamed Layer* 639) [Constant]_output[Float(64,1)] Layer(Constant): (Unnamed Layer* 640) [Constant], Tactic: 0, -> (Unnamed Layer* 640) [Constant]_output[Float(64,1)] Layer(PluginV2): (Unnamed Layer* 641) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 638) [Convolution]_output[Float(1,64,36,40,156)], (Unnamed Layer* 639) [Constant]_output[Float(64,1)], (Unnamed Layer* 640) [Constant]_output[Float(64,1)] -> (Unnamed Layer* 641) [PluginV2DynamicExt]_output_0[Float(1,64,36,40,156)] Layer(PointWiseV2): PWN((Unnamed Layer* 642) [Activation]), Tactic: 8, (Unnamed Layer* 641) [PluginV2DynamicExt]_output_0[Float(1,64,36,40,156)] -> (Unnamed Layer* 642) [Activation]_output[Float(1,64,36,40,156)] Layer(CudnnConvolution): (Unnamed Layer* 643) [Convolution], Tactic: 1, (Unnamed Layer* 642) [Activation]_output[Float(1,64,36,40,156)] -> (Unnamed Layer* 643) [Convolution]_output[Float(1,64,18,20,78)] Layer(Constant): (Unnamed Layer* 644) [Constant], Tactic: 0, -> (Unnamed Layer* 644) [Constant]_output[Float(64,1)] Layer(Constant): (Unnamed Layer* 645) [Constant], Tactic: 0, -> (Unnamed Layer* 645) [Constant]_output[Float(64,1)] Layer(PluginV2): (Unnamed Layer* 646) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 643) [Convolution]_output[Float(1,64,18,20,78)], (Unnamed Layer* 644) [Constant]_output[Float(64,1)], (Unnamed Layer* 645) [Constant]_output[Float(64,1)] -> (Unnamed Layer* 646) [PluginV2DynamicExt]_output_0[Float(1,64,18,20,78)] Layer(PointWiseV2): PWN((Unnamed Layer* 647) [Activation]), Tactic: 8, (Unnamed Layer* 646) [PluginV2DynamicExt]_output_0[Float(1,64,18,20,78)] -> (Unnamed Layer* 647) [Activation]_output[Float(1,64,18,20,78)] Layer(CudnnConvolution): (Unnamed Layer* 648) [Convolution], Tactic: 1, (Unnamed Layer* 647) [Activation]_output[Float(1,64,18,20,78)] -> (Unnamed Layer* 648) [Convolution]_output[Float(1,64,18,20,78)] Layer(Constant): (Unnamed Layer* 649) [Constant], Tactic: 0, -> (Unnamed Layer* 649) [Constant]_output[Float(64,1)] Layer(Constant): (Unnamed Layer* 650) [Constant], Tactic: 0, -> (Unnamed Layer* 650) [Constant]_output[Float(64,1)] Layer(PluginV2): (Unnamed Layer* 651) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 648) [Convolution]_output[Float(1,64,18,20,78)], (Unnamed Layer* 649) [Constant]_output[Float(64,1)], (Unnamed Layer* 650) [Constant]_output[Float(64,1)] -> (Unnamed Layer* 651) [PluginV2DynamicExt]_output_0[Float(1,64,18,20,78)] Layer(PointWiseV2): PWN((Unnamed Layer* 652) [Activation]), Tactic: 8, (Unnamed Layer* 651) [PluginV2DynamicExt]_output_0[Float(1,64,18,20,78)] -> (Unnamed Layer* 652) [Activation]_output[Float(1,64,18,20,78)] Layer(CudnnDeconvolution): (Unnamed Layer* 653) [Deconvolution], Tactic: 0, (Unnamed Layer* 652) [Activation]_output[Float(1,64,18,20,78)] -> (Unnamed Layer* 653) [Deconvolution]_output[Float(1,64,35,39,155)] Layer(Constant): (Unnamed Layer* 654) [Constant], Tactic: 0, -> (Unnamed Layer* 654) [Constant]_output[Float(1,836160)] Layer(Reformat): (Unnamed Layer* 655) [Shuffle]_output copy, Tactic: 0, (Unnamed Layer* 655) [Shuffle]_output[Float(1,13540800)] -> (Unnamed Layer* 656) [Concatenation]_output[Float(1,13540800)] Layer(Reformat): (Unnamed Layer* 654) [Constant]_output copy, Tactic: 0, (Unnamed Layer* 654) [Constant]_output[Float(1,836160)] -> (Unnamed Layer* 656) [Concatenation]_output[Float(1,836160)] Layer(Constant): (Unnamed Layer* 658) [Constant], Tactic: 0, -> (Unnamed Layer* 658) [Constant]_output[Float(64,1)] Layer(Constant): (Unnamed Layer* 659) [Constant], Tactic: 0, -> (Unnamed Layer* 659) [Constant]_output[Float(64,1)] Layer(PluginV2): (Unnamed Layer* 660) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 657) [Shuffle]_output[Float(1,64,36,40,156)], (Unnamed Layer* 658) [Constant]_output[Float(64,1)], (Unnamed Layer* 659) [Constant]_output[Float(64,1)] -> (Unnamed Layer* 660) [PluginV2DynamicExt]_output_0[Float(1,64,36,40,156)] Layer(ElementWise): (Unnamed Layer* 661) [ElementWise] + (Unnamed Layer* 662) [Activation], Tactic: 1, (Unnamed Layer* 660) [PluginV2DynamicExt]_output_0[Float(1,64,36,40,156)], (Unnamed Layer* 642) [Activation]_output[Float(1,64,36,40,156)] -> (Unnamed Layer* 662) [Activation]_output[Float(1,64,36,40,156)] Layer(CudnnDeconvolution): (Unnamed Layer* 663) [Deconvolution], Tactic: 0, (Unnamed Layer* 662) [Activation]_output[Float(1,64,36,40,156)] -> (Unnamed Layer* 663) [Deconvolution]_output[Float(1,32,71,79,311)] Layer(Constant): (Unnamed Layer* 664) [Constant], Tactic: 0, -> (Unnamed Layer* 664) [Constant]_output[Float(1,1687072)] Layer(Reformat): (Unnamed Layer* 665) [Shuffle]_output copy, Tactic: 0, (Unnamed Layer* 665) [Shuffle]_output[Float(1,55820768)] -> (Unnamed Layer* 666) [Concatenation]_output[Float(1,55820768)] Layer(Reformat): (Unnamed Layer* 664) [Constant]_output copy, Tactic: 0, (Unnamed Layer* 664) [Constant]_output[Float(1,1687072)] -> (Unnamed Layer* 666) [Concatenation]_output[Float(1,1687072)] Layer(Constant): (Unnamed Layer* 668) [Constant], Tactic: 0, -> (Unnamed Layer* 668) [Constant]_output[Float(32,1)] Layer(Constant): (Unnamed Layer* 669) [Constant], Tactic: 0, -> (Unnamed Layer* 669) [Constant]_output[Float(32,1)] Layer(PluginV2): (Unnamed Layer* 670) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 667) [Shuffle]_output[Float(1,32,72,80,312)], (Unnamed Layer* 668) [Constant]_output[Float(32,1)], (Unnamed Layer* 669) [Constant]_output[Float(32,1)] -> (Unnamed Layer* 670) [PluginV2DynamicExt]_output_0[Float(1,32,72,80,312)] Layer(ElementWise): (Unnamed Layer* 671) [ElementWise], Tactic: 1, (Unnamed Layer* 632) [ElementWise]_output[Float(1,32,72,80,312)], (Unnamed Layer* 670) [PluginV2DynamicExt]_output_0[Float(1,32,72,80,312)] -> (Unnamed Layer* 671) [ElementWise]_output[Float(1,32,72,80,312)] Layer(Constant): (Unnamed Layer* 726) [Constant], Tactic: 0, -> (Unnamed Layer* 726) [Constant]_output[Float(1,20,304,288,3)] Layer(PluginV2): (Unnamed Layer* 727) [PluginV2DynamicExt], Tactic: 0, (Unnamed Layer* 671) [ElementWise]_output[Float(1,32,72,80,312)], (Unnamed Layer* 726) [Constant]_output[Float(1,20,304,288,3)] -> (Unnamed Layer* 727) [PluginV2DynamicExt]_output_0[Float(1,32,20,304,288)] Layer(Myelin): {ForeignNode[(Unnamed Layer* 730) [ElementWise]]}, Tactic: 0, (Unnamed Layer* 727) [PluginV2DynamicExt]_output_0[Float(1,32,20,304,288)], (Unnamed Layer* 729) [Identity]_output[Float(1,1,20,304,288)] -> Box[Float(1,32,20,304,288)] [TensorRT] INFO: [MemUsageSnapshot] Builder end: CPU 1563 MiB, GPU 9954 MiB [TensorRT] INFO: [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 1563, GPU 9955 (MiB) [TensorRT] INFO: Loaded engine size: 157 MB [TensorRT] INFO: [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1720 MiB, GPU 10113 MiB [TensorRT] VERBOSE: Using cublas a tactic source [TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1726, GPU 10256 (MiB) [TensorRT] VERBOSE: Using cuDNN as a tactic source [TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 1726, GPU 10266 (MiB) [TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1726, GPU 10261 (MiB) [TensorRT] VERBOSE: Deserialization required 512539 microseconds. [TensorRT] INFO: [MemUsageSnapshot] deserializeCudaEngine end: CPU 1726 MiB, GPU 10261 MiB [TensorRT] INFO: [MemUsageSnapshot] ExecutionContext creation begin: CPU 1532 MiB, GPU 9842 MiB [TensorRT] VERBOSE: Using cublas a tactic source [TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1532, GPU 9842 (MiB) [TensorRT] VERBOSE: Using cuDNN as a tactic source [TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +7, now: CPU 1532, GPU 9849 (MiB) [TensorRT] VERBOSE: Total per-runner device memory is 90608640 [TensorRT] VERBOSE: Total per-runner host memory is 109936 [TensorRT] VERBOSE: Allocated activation device memory of size 1228143616 [TensorRT] VERBOSE: myelinAllocCb allocated GPU (data-constants) 8 bytes at 0x226cc7500. [TensorRT] INFO: [MemUsageSnapshot] ExecutionContext creation end: CPU 1537 MiB, GPU 11109 MiB [TensorRT] VERBOSE: myelinAllocCb allocated GPU 63037440 bytes at 0x22ddea000. [TensorRT] ERROR: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed.) And the inference code snippet is: def do_inference(context, host_in, host_out): engine = context.engine #assert engine.num_bindings == 6 bindings = [] bytes = 0 stream = cuda.Stream() for i in range(len(host_in)): devide_in = cuda.mem_alloc(host_in[i].nbytes) cuda.memcpy_htod_async(devide_in, host_in[i], stream) bindings.append(int(devide_in)) devide_out = cuda.mem_alloc(host_out.nbytes) bindings.append(int(devide_out)) context.execute_async(bindings=bindings, stream_handle=stream.handle) cuda.memcpy_dtoh_async(host_out, devide_out, stream) stream.synchronize() runtime = trt.Runtime(TRT_LOGGER) assert runtime with open(ENGINE_PATH, "rb") as f: engine = runtime.deserialize_cuda_engine(f.read()) assert engine context = engine.create_execution_context() assert context BATCH_SIZE = 1 image_size = IMAGE_BATCH_SIZE * IMAGE_CHANNELS * IMAGE_HEIGHT * IMAGE_WIDTH host_in = [] left_image = cuda.pagelocked_empty(image_size, dtype=np.float32) right_image = cuda.pagelocked_empty(image_size, dtype=np.float32) calib = cuda.pagelocked_empty(BATCH_SIZE * 3 * 4, dtype=np.float32) data_left = cv2.imread("./liga/test/left/000000.png") data_right = cv2.imread("./liga/test/right/000000.png") calib_project = np.random.rand(12) data_left = cv2.resize(data_left, (IMAGE_HEIGHT, IMAGE_WIDTH)).flatten() data_right = cv2.resize(data_right, (IMAGE_HEIGHT, IMAGE_WIDTH)).flatten() np.copyto(left_image, data_left.ravel()) np.copyto(right_image, data_right.ravel()) np.copyto(calib, calib_project.ravel()) host_in.append(left_image) host_in.append(right_image) host_in.append(calib) host_out = cuda.pagelocked_empty(BATCH_SIZE * 32 * 20 * 304*288, dtype=np.float32) do_inference(context, host_in, host_out) print(f'Output: \n{host_out[:5000]}\n{host_out[-10:]}') ## Environment **TensorRT Version**: 8.0 **NVIDIA GPU**: AGX Xavier **NVIDIA Driver Version**: **CUDA Version**: 10.2 **CUDNN Version**: 8.2 **Operating System**: JetPack 4.6 **Python Version (if applicable)**: 3.6 **Tensorflow Version (if applicable)**: **PyTorch Version (if applicable)**: **Baremetal or Container (if so, version)**: ## Relevant Files ## Steps To Reproduce

Thank you.

srsjd · October 24, 2022, 6:56am

mainboard.log.INFO.20221017-160406.3087681 (1.9 MB)
Here is the full log. The CUDA environment is installed through Jetpack 5.0. There is no repo onnx model available since the network is constructed using tensorrt c++ API and pytorch pt model.

As for the reference, the issue posted in github happens during inference phase and the building phase was successfil, but my error occured during the building phase. So I don’t think checking enqueue function can solve my issue.

Let me know if you have any suggestion. Many thanks.

TRT: 8.4.0.9
GPU: Orin
CUDA: 11.4.166

spolisetty · November 2, 2022, 1:55pm

Hi,

Based on the logs, the error timestamp looks different than other log messages.

I1017 16:05:05.494969 3087681 rt_net.cc:37] =============== Computing costs for 
I1017 16:05:05.494993 3087681 rt_net.cc:37] *************** Autotuning format combination: Float(414720,11520,8,1) -> Float(414720,11520,8,1) ***************
I1017 16:05:05.495082 3087681 rt_net.cc:37] --------------- Timing Runner: cls_pred_prob (PluginV2)
I1017 16:05:05.546500 3087681 rt_net.cc:37] Deleting timing cache: 849 entries, served 641 hits since creation.
I1017 13:59:37.946795 1477744 rt_net.cc:37] [mainboard]2: [pluginV2Runner.cpp::execute::265] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. )

Could you please share with us the minimal issue repro script/model/steps to try from our end for better debugging.

Thank you.

duanchengwen07 · August 10, 2023, 1:47pm

Did you solve it in the end? how to solve