Network has dynamic or shape inputs but no optimization profiles have been defined

yashkhokarale · July 11, 2020, 6:44pm

Description

I have been stuck on this error for the past 3 days. Looks minor but still unable to figure out. Anyone kindly helps me get past this.

I have trained my model on CIFAR10 on TensorFlow & then exported to ONNX. Do I need to play around with some dynamic shapes while exporting? Also, I have exported the whole “.pb” I haven’t frozen any “graph or ckpt”. Is fine. If freezing a graph or something is required kindly shed some light on that (with links).
Also, I have attached the netron output do let me know if it’s correct.onnx-model|75x500

Also, I am referring to “sample_dynamic_reshape.cpp”.

github.com

NVIDIA/TensorRT/blob/9ee84508cfaf9f9e0fd0cd7dffd951f6345cd9a3/samples/opensource/sampleDynamicReshape/sampleDynamicReshape.cpp#L177


      
          profileCalib->setDimensions(input->getName(), OptProfileSelector::kOPT, Dims4{calibBatchSize, 1, 28, 28});
          profileCalib->setDimensions(input->getName(), OptProfileSelector::kMAX, Dims4{calibBatchSize, 1, 28, 28});
          preprocessorConfig->setCalibrationProfile(profileCalib);
          
          
std::unique_ptr<IInt8Calibrator> calibrator;
          if (mParams.int8)
          {
              preprocessorConfig->setFlag(BuilderFlag::kINT8);
              const int nCalibBatches{10};
              MNISTBatchStream calibrationStream(
                  calibBatchSize, nCalibBatches, "train-images-idx3-ubyte", "train-labels-idx1-ubyte", mParams.dataDirs);
              calibrator.reset(
                  new Int8EntropyCalibrator2<MNISTBatchStream>(calibrationStream, 0, "MNISTPreprocessor", "input"));
              preprocessorConfig->setInt8Calibrator(calibrator.get());
          }
          
          
mPreprocessorEngine = makeUnique(builder->buildEngineWithConfig(*preprocessorNetwork, *preprocessorConfig));
          if (!mPreprocessorEngine)
          {
              sample::gLogError << "Preprocessor engine build failed." << std::endl;
              return false;

What are these formats for the images & how do I pass my CIFAR10 in such format? My CIFAR10 images are available in batches when downloaded in a binary file. How can I feed it in here ie in which format & how many images?
“train-images-idx3-ubyte”, “train-labels-idx1-ubyte”

Is it necessary to pass image in PGM / PPM? Aren’t there other ways to pass an image.

github.com

NVIDIA/TensorRT/blob/9ee84508cfaf9f9e0fd0cd7dffd951f6345cd9a3/samples/opensource/sampleDynamicReshape/sampleDynamicReshape.cpp#L329


      
          //! It runs inference for using a random image from the MNIST dataset as an input.
          //!
          bool SampleDynamicReshape::infer()
          {
              // Load a random PGM file into a host buffer, then copy to device.
              std::random_device rd{};
              std::default_random_engine generator{rd()};
              std::uniform_int_distribution<int> digitDistribution{0, 9};
              int digit = digitDistribution(generator);
          
          
    Dims inputDims = loadPGMFile(locateFile(std::to_string(digit) + ".pgm", mParams.dataDirs));
              mInput.deviceBuffer.resize(inputDims);
              CHECK(cudaMemcpy(
                  mInput.deviceBuffer.data(), mInput.hostBuffer.data(), mInput.hostBuffer.nbBytes(), cudaMemcpyHostToDevice));
          
          
    // Set the input size for the preprocessor
              CHECK_RETURN_W_MSG(mPreprocessorContext->setBindingDimensions(0, inputDims), false, "Invalid binding dimensions.");
          
          
    // We can only run inference once all dynamic input shapes have been specified.
              if (!mPreprocessorContext->allInputDimensionsSpecified())
              {

If yes what are the ways?
If No, then how do I convert each of my images in this format? I have my CIFAR10 images in NumPy array.

Here’s the GitHub link for my code & GitHub - yashraj02/Tensor-RT

Tensorflow- 2.2.0
onnx-1.7.0
tf2onnx-1.6.2

Using TensorRT Conatianer Image (20.06) Latest
CUDA <<11.0.167>>
CUDA <<11.0.167>>
<<TensorRT 7.1.2>>
Method : TernsorRT C++ API for inference
Which samples(from TensorRT C++ API) should be used for my task?

PS:
Also, a suggestion if anyone is reading from TensorRT team. Kindly add numbering (1,2,…) & sub-numbering [a,b,…] to the TensorRT GitHub Readme section. Its a nightmare for a beginner like me to get started. There are certain sections which are optional certain important can’t differentiate easily. It’s just a suggestion.

A clear and concise description of the bug or issue.

Environment

TensorRT Version 7.1.3:
GPU Type Tesla V100:
Nvidia Driver Version 450.:
CUDA Version 8.0:
CUDNN Version:
Operating System + Version Ubuntu 18.04:
Python Version (if applicable) 3.6:
TensorFlow Version (if applicable) 2.2:
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered
error1919×492 41.7 KB

SunilJB · July 11, 2020, 8:19pm

When using runtime dimensions, you must create at least one optimization profile at build time. Please refer below link:

https://github.com/NVIDIA/TensorRT/blob/master/samples/opensource/sampleDynamicReshape/sampleDynamicReshape.cpp#L153

For ONNX model generation for saved model, checkpoint or using graphdef format, please refer below link:

Supported data format in TRT:

For pre-processing of input image for additional format please refer below link, examples are provided for streaming from live camera feed and processing images

Thanks

yashkhokarale · July 12, 2020, 3:42am

I guess you haven’t refereed my code from my github link provided above.
I have used the same example as that of sample_dynamic_shape.
Hence,
“auto profile = builder->createOptimizationProfile();”
this is already in my code.

yashkhokarale · July 12, 2020, 3:48am

Understand my concern correctly!!
Your are talking about int8,fp16 etc while I am asking is image formats ie .png,jpg etc

What formats are accepted from jpg,png,etc & how do I pass my CIFAR10 images in such format to onnx model (any working example or github link)?
My CIFAR10 images are available in batches (batch1, batch2,…) when downloaded, in a binary file. How can I feed it in here ie in which format & how many images?
https://www.cs.toronto.edu/~kriz/cifar.html

SunilJB · July 13, 2020, 1:17pm

Hi @yashkhokarale,

Input name used in your code is incorrect (Similarly output tensor name needs to be updated)

github.com

yashraj02/Tensor-RT/blob/master/cifar.cpp#L415


      
              infile.read(reinterpret_cast<char*>(fileData.data()), vol);
          
              // Print an ascii representation
              sample::gLogInfo << "Input:\n";
              for (size_t i = 0; i < vol; i++)
              {
                  sample::gLogInfo << (" .:-=+*#%@"[fileData[i] / 26]) << (((i + 1) % w) ? "" : "\n");
              }
              sample::gLogInfo << std::endl;
          
              // Normalize and copy to the host buffer.
              mInput.hostBuffer.resize(inputDims);
              float* hostDataBuffer = static_cast<float*>(mInput.hostBuffer.data());
              std::transform(fileData.begin(), fileData.end(), hostDataBuffer,
                  [](uint8_t x) { return 1.0 - static_cast<float>(x / 255.0); });
              return inputDims;
          }
          
          //!
          //! \brief Checks whether the model prediction (in mOutput) is correct.
          //!

As per your model it should be " conv2d_input:0"

I ran your onnx model using trtexec command line tool and i am able to successfully generate the TRT engine file:
trtexec --onnx=cifar.onnx --explicitBatch --minShapes=conv2d_input:0:1x32x32x3 --optShapes=conv2d_input:0:16x32x32x3 --maxShapes=conv2d_input:0:32x32x32x3 --shapes=conv2d_input:0:5x32x32x3 --verbose
[07/13/2020-12:59:32] [I] min: 0.0390015 ms
[07/13/2020-12:59:32] [I] max: 0.0667419 ms
[07/13/2020-12:59:32] [I] mean: 0.0411783 ms
[07/13/2020-12:59:32] [I] median: 0.0410156 ms
[07/13/2020-12:59:32] [I] percentile: 0.0432129 ms at 99%
[07/13/2020-12:59:32] [I] total compute time: 2.42643 s
&&&& PASSED TensorRT.trtexec # trtexec --onnx=cifar.onnx --explicitBatch --minShapes=conv2d_input:0:1x32x32x3 --optShapes=conv2d_input:0:16x32x32x3 --maxShapes=conv2d_input:0:32x32x32x3 --shapes=conv2d_input:0:5x32x32x3 --verbose

https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec#example-4-running-an-onnx-model-with-full-dimensions-and-dynamic-shapes

You can refer to this link, it has multiple example on formatting the input image including jpg and camera input. On similar line you can perform the input image pre-processing.

Thanks

yashkhokarale · July 14, 2020, 3:57pm

Thanks for your valuable feed back. I am able to run using the model using trtexec but the issue persists while I make file & run in bin as ./sample_cifar.
I have changed the input & output names as suggested by you.

github.com

yashraj02/Tensor-RT/blob/d22722d8c1f467acc0e614396c7d7c08f199ca0b/cifar.cpp#L461


      
              if (args.dataDirs.empty()) //!< Use default directories if user hasn't provided directory paths
              {
                  params.dataDirs.push_back("data/cifar/");
                  params.dataDirs.push_back("data/samples/cifar/");
              }
              else //!< Use the data directory provided by the user
              {
                  params.dataDirs = args.dataDirs;
              }
              params.onnxFileName = "cifar.onnx";
              params.inputTensorNames.push_back("conv2d_input:0");
              params.outputTensorNames.push_back("Identity:0");
              params.int8 = args.runInInt8;
              params.fp16 = args.runInFp16;
              return params;
          }
          
          
//!
          //! \brief Prints the help information for running this sample
          //!
          void printHelpInfo()

SunilJB · July 16, 2020, 10:25am

Hi,

You can’t directly port same code to run your model.
You were getting the error because buildPredictionEngine doesn’t have any optimization profile setting and buildPreprocessorEngine code is adding additional input to create a dynamic input case in sample code.

In order to support your model, you can try removing the buildPreprocessorEngine related codes and update the buildPredictionEngine function similar to below code to generate a TRT model.

const auto explicitBatch = 1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
    auto network = makeUnique(builder->createNetworkV2(explicitBatch));
    auto parser = nvonnxparser::createParser(*network, gLogger.getTRTLogger());
    bool parsingSuccess = parser->parseFromFile(
        locateFile(mParams.onnxFileName, mParams.dataDirs).c_str(), static_cast<int>(gLogger.getReportableSeverity()));
    if (!parsingSuccess)
    {
        throw std::runtime_error{"Failed to parse model"};
    }

    mPredictionInputDims = network->getInput(0)->getDimensions();
    mPredictionOutputDims = network->getOutput(0)->getDimensions();

    // Create a builder config
    auto preprocessorConfig = makeUnique(builder->createBuilderConfig());

    // Create an optimization profile so that we can specify a range of input dimensions.
    auto profile = builder->createOptimizationProfile();

    profile->setDimensions("conv2d_input:0", OptProfileSelector::kMIN, Dims4{1, 32, 32, 3});
	gLogInfo << "Passed min" << std::endl;
    profile->setDimensions("conv2d_input:0", OptProfileSelector::kOPT, Dims4{16, 32, 32, 3});
	gLogInfo << "Passed mid" << std::endl;
    profile->setDimensions("conv2d_input:0", OptProfileSelector::kMAX, Dims4{32, 32, 32, 3});
	gLogInfo << "Passed max" << std::endl;
    preprocessorConfig->addOptimizationProfile(profile);
	
    preprocessorConfig->setMaxWorkspaceSize(100_MiB);
    // Build the prediciton engine.
    mPredictionEngine = makeUnique(builder->buildEngineWithConfig(*network, *preprocessorConfig));

Please refer to below documentation as well for more details

Thanks

Topic		Replies	Views
Network has dynamic or shape inputs, but no optimization profile has been defined TensorRT tensorrt	6	861	April 29, 2023
[TensorRT] ERROR: input: dynamic input is missing dimensions in profile 0 TensorRT	11	6978	October 12, 2021
Some PyTorch model with slicing operation fails on inference TensorRT tensorrt , pytorch , onnx , deepstream	2	1440	January 7, 2022
Dynamic Shapes TensorRT	6	4242	June 26, 2020
ONNX to TensorRT Python module doesn't generate dynamic batch size engine TensorRT tensorrt , cudnn , onnx	3	1070	October 20, 2023
I do not get any performance improvement after using TensorRT provider for object detection model Jetson Nano tensorrt , onnx	7	1399	July 12, 2022
Built engine failed to include optimization profile with dynamic input shapes #2166 TensorRT	3	476	July 22, 2022
[TensorRT] ERROR: Network must have at least one output TensorRT tensorrt	29	2355	September 30, 2021
Failed to used TensorRT Engine file in deepstream DeepStream SDK	16	2750	October 12, 2021
TensorRT6 OnnxParser could not support dynamic shape. TensorRT	11	3250	November 8, 2019

Description

Environment

Relevant Files

Steps To Reproduce

Related topics