Get intermediate layer output

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I am using the Python API and I would like to get the output of an hidden layer after the execution of an engine. Is this possible?

3

Hi @mfoglio,

We can set the layer as a network output and bind a buffer to it to collect the output.

Thank you.

4

The engine is built in C++ and run in Python. Can I do that while loading the engine in Python or should I do that when building it in C++? You can refer to this repository: tensorrtx/yolov5 at master · wang-xinyu/tensorrtx · GitHub

5

Hi @mfoglio,

We cannot do while loading. We need to set the layer as a network output and regenerate the engine.

Thank you.

6

Hello. Do you have an example? Thank you

7

So rebuilding the engine means redoing the torch->onnx->trt? Cut the output I want from the existing torch model. Is it possible to output multiple layers? Do I have to make this separately?

8

Hi all. I am in a quite similar situation. I need to print the output of a specific layer. My code works well (C++/TRT) when I choose the last layer (it correctly displays my segmentation mask, ie a matrix of 0 and non zero values). But when I choose the first layer, it only display zero values matrix (and I use the same code to extract data). What i am doing:

  • To set the a specific intermediate layer output as model output
    network->markOutput(*network->getLayer(i)->getOutput(j));

  • The engine bindings are well updated
    “ENGINE binding:0/1 input_2:0 {1,3,256,256}”
    “ENGINE binding:1/1 UnetMobileNetV3Small/rescaling_1/mul:0 {1,3,256,256}”

  • To display the output data:

std::vector<float> outputCpuDataDebug;
const size_t  outputCpuDataDebugSize = engine->getBindingDimensions(1).d[0] * engine->getBindingDimensions(1).d[1] * engine->getBindingDimensions(1).d[2] * engine->getBindingDimensions(1).d[3];
outputCpuDataDebug.resize(outputCpuDataDebugSize);
cudaMemcpyAsync(outputCpuDataDebug.data(), buffers[1], sizeof(float) * outputCpuDataDebugSize, cudaMemcpyDeviceToHost, stream);
cudaStreamSynchronize(stream);
std::cout << "outputCpuDataDebug" << std::endl;
for (int i = 0; i < engine->getBindingDimensions(1).d[1]; i++) {
  for (int j = 0; j < engine->getBindingDimensions(1).d[2]; j++) {
    for (int k = 0; k < engine->getBindingDimensions(1).d[3]; k++) {
      std::cout << " " << outputCpuDataDebug[i *  engine->getBindingDimensions(1).d[2] * engine->getBindingDimensions(1).d[3] + j * engine->getBindingDimensions(1).d[3] + k];
    }
    std::cout << std::endl ;
  }
  std::cout << std::endl << "--------------" << std::endl;
}
  • The result: a zero-matrix! But if a set the last layer output as the model output, I get the right matrix with non zero and zero values.