Conv3D with strides = 2 and user defined padding

Hello,

I get an error when trying to do a 3d convolution with strides = 2 and postPadding = 1. The terminal says
WARNING: Setting layouts of network and plugin input/output tensors to linear, as 3D operators are found and 3D non-linear IO formats are not supported, yet.
ERROR: C:\source\builder\cudnnBuilderUtils.cpp (364) - Cuda Error in nvinfer1::cudnn::findFastestTactic: 77 (an illegal memory access was encountered)
ERROR: C:\source\rtSafe\safeRuntime.cpp (32) - Cuda Error in nvinfer1::internal::DefaultAllocator::free: 77 (an illegal memory access was encountered)

Below is a minimal example for the error

#include <NvInfer.h>
#include <iostream>
#include <vector>

using namespace std;
using namespace nvinfer1;

class StdOutLogger : public ILogger
{
public:
  StdOutLogger(ILogger::Severity severity)
    : mReportedSeverity(severity)
  {}

  void log(ILogger::Severity severity, char const* message) override
  {
    if (severity > mReportedSeverity)
      return;

    switch (severity)
    {
    case Severity::kINTERNAL_ERROR: std::cerr << "INTERNAL_ERROR: "; break;
    case Severity::kERROR: std::cerr << "ERROR: "; break;
    case Severity::kWARNING: std::cerr << "WARNING: "; break;
    case Severity::kINFO: std::cerr << "INFO: "; break;
    default: std::cerr << "UNKNOWN: "; break;
    }

    std::cerr << message << std::endl;
  }

private:
  ILogger::Severity mReportedSeverity;
};

Weights getKernelWeights(int inputChannel, int filter, int kernelsize)
{
  Weights weights;
  weights.count = inputChannel * filter * kernelsize * kernelsize * kernelsize;
  weights.type = DataType::kFLOAT;
  vector<float> values(weights.count);
  for (auto i = 0u; i < values.size(); ++i)
  {
    values[i] = i;
  }
  weights.values = values.data();
  return weights;
}

Weights getBiasWeights()
{
  Weights weights;
  weights.count = 2;
  weights.type = DataType::kFLOAT;
  vector<float> values{ -1.f, -2.f };
  weights.values = values.data();
  return weights;
}

int main()
{
  StdOutLogger logger(ILogger::Severity::kINFO);
  auto builder = createInferBuilder(logger);
  auto network = builder->createNetwork();

  auto convBiasWeights = getBiasWeights();

  auto input = network->addInput("input", DataType::kFLOAT, Dims4{ 1, 64, 64, 64 });
  auto conv = network->addConvolutionNd(*input, 2, Dims3{ 3, 3, 3}, getKernelWeights(1, 2, 3), convBiasWeights);
  conv->setPrePadding(Dims3{ 1, 1, 1 });
  conv->setPostPadding(Dims3{ 1, 1, 1 });

  auto conv2 = network->addConvolutionNd(*conv->getOutput(0), 2, Dims3{ 3, 3, 3 }, getKernelWeights(2, 2, 3), convBiasWeights);
  conv2->setStrideNd(Dims3{ 2, 2, 2 });
  conv2->setPrePadding(Dims3{ 0, 0, 0 });
  conv2->setPostPadding(Dims3{ 1, 1, 1 });
  
  network->markOutput(*conv2->getOutput(0));

  builder->setMaxBatchSize(1);
  builder->setMaxWorkspaceSize(static_cast<size_t>(1) << 33);

  auto engine = builder->buildCudaEngine(*network);

  engine->destroy();
  network->destroy();
  builder->destroy();

  return 0;
}

PS: Maybe this issue is related to https://devtalk.nvidia.com/default/topic/1064564/tensorrt/dynamic-input-and-convolution/

Hi Xalanot,

I was able to repro your issue and have escalated to the engineering team for more details.

Thanks,
NVIDIA Enterprise Support

I have encountered the exact same problem. I converted R3D-18 from Pytorch (https://pytorch.org/docs/stable/_modules/torchvision/models/video/resnet.html#r3d_18 which also has stride=2, padding=1 layers) to onnx and then used onnx-tensorrt parser.

I am using:
TensorRT 6.0, CUDA 10.0, Cudnn 7.6.4, 2080 Ti, ONNX=1.6.0, Pytoch’s onnx export with opset=9

I uploaded a sample model here in .onnx format: https://drive.google.com/file/d/1XlO79gSPpfzGq_xq-OTYxSxxJW6ZTzfF/view?usp=sharing

It fails somewhere in onnx_tensorrt.backend.prepare(model).

[TensorRT] WARNING: Setting layouts of network and plugin input/output tensors to linear, as 3D operators are found and 3D non-linear IO formats are not supported, yet.
[TensorRT] ERROR: ../builder/cudnnBuilderUtils.cpp (354) - Cuda Error in findFastestTactic: 77 (an illegal memory access was encountered)
[TensorRT] ERROR: ../rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 77 (an illegal memory access was encountered)
terminate called after throwing an instance of 'nvinfer1::CudaError'
  what():  std::exception
[19]    5681 abort (core dumped)  python onnx_trt_test.py
1 Like

Hi Xalanot,

This is either the same solution as https://devtalk.nvidia.com/default/topic/1064564/tensorrt/dynamic-input-and-convolution/

or

Either way, this should be fixed in the next release.

Thanks,
NVIDIA Enterprise Support

Is there any one solve this problem, i have this trouble too…