Cuda Error in executeInternal: 700 (an illegal memory access was encountered)

[AastaLLL]

Hi,

You can enable the verbose output to get more information about the error.

bool SampleDynamicReshape::build()
{
    ...
    sample::gLogger.setReportableSeverity(nvinfer1::ILogger::Severity::kVERBOSE);
    return buildPredictionEngine(builder) /*&& buildPreprocessorEngine(builder)*/;
}

We got the following log when deploying the app with batchsize=4096.

It looks like the error occurs when TensorRT wants to write out the output tensor.
So please double-check the output buffer of enqueue function:

...
[11/12/2021-13:30:58] [V] [TRT] Layer(Reformat): StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_1/classify/conv1d__464:0 copy, Tactic: 0, StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_1/classify/conv1d__464:0[Float(128,1)] -> tf_op_layer_concat[Float(128,1)]
[11/12/2021-13:30:58] [V] [TRT] Layer(Reformat): StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_2/classify/conv1d__372:0 copy, Tactic: 0, StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_2/classify/conv1d__372:0[Float(128,1)] -> tf_op_layer_concat[Float(128,1)]
[11/12/2021-13:30:58] [V] [TRT] Layer(Reformat): StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_3/classify/conv1d__280:0 copy, Tactic: 0, StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_3/classify/conv1d__280:0[Float(128,1)] -> tf_op_layer_concat[Float(128,1)]
 Preparation time: 73061951044[ns]
[11/12/2021-13:30:58] [E] [TRT] engine.cpp (986) - Cuda Error in executeInternal: 700 (an illegal memory access was encountered)
[11/12/2021-13:30:58] [E] [TRT] FAILED_EXECUTION: std::exception
 failed executeV2
&&&& FAILED TensorRT.tester_onnx_RADA # ./tester_onnx_rada --batch=4096
[11/12/2021-13:30:58] [E] [TRT] engine.cpp (179) - Cuda Error in ~ExecutionContext: 700 (an illegal memory access was encountered)
[11/12/2021-13:30:58] [E] [TRT] INTERNAL_ERROR: std::exception
[11/12/2021-13:30:58] [E] [TRT] Parameter check failed at: ../rtSafe/safeContext.cpp::terminateCommonContext::155, condition: cudnnDestroy(context.cudnn) failure.
[11/12/2021-13:30:58] [E] [TRT] Parameter check failed at: ../rtSafe/safeContext.cpp::terminateCommonContext::165, condition: cudaEventDestroy(context.start) failure.
[11/12/2021-13:30:58] [E] [TRT] Parameter check failed at: ../rtSafe/safeContext.cpp::terminateCommonContext::170, condition: cudaEventDestroy(context.stop) failure.
[11/12/2021-13:30:58] [E] [TRT] ../rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 700 (an illegal memory access was encountered)
terminate called after throwing an instance of 'nvinfer1::CudaError'
  what():  std::exception
Aborted (core dumped)

Thanks.