Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) GPU Tesla T4
• DeepStream Version 5.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 450.119.03
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Trying to implement an ensemble model consisting of 3 models - Face Detection, Python Backend model that preprocesses image according to bounding boxes, and a Multi-Label Classifier model, in this sequence.
This ensemble model runs fine in tritonserver:21.06-py3 container with client script that I run from tritonserver:21.06-py3-sdk container.
But on deploying in deepstream I get this error -
ERROR: infer_trtis_server.cpp:258 Triton: TritonServer response error received., triton_err_str:Unsupported, err_msg:in ensemble 'ENSEMBLE_MODEL', failed to get input buffer in CPU memory
The complete error log:
Creating Pipeline
Creating streamux
inputs: ['file:///opt/nvidia/deepstream/deepstream-5.1/sources/deepstream_python_apps/apps/DEEPSTREAM_RGB_INGESTION_PRIMARY_INFERENCE/SAMPLE_RGB_STREAM/1.avi']
Creating source_bin 0
Creating source bin
source-bin-00
Creating Pgie
Creating nvvidconv
Creating nvvidconv2
Creating nvosd
Creating capsfilter
WARNING: Overriding infer-config batch-size 0 with number of sources 1
Creating mp4 Encoder
Creating mp4 Container
Creating Filesink
Adding elements to Pipeline
(python3:3240): GStreamer-WARNING **: 16:23:49.420: Name 'queue6' is not unique in bin 'pipeline0', not adding
Linking elements in the Pipeline
In the probe
Now playing...
Number of Input streams: 1
0 : file:///opt/nvidia/deepstream/deepstream-5.1/sources/deepstream_python_apps/apps/DEEPSTREAM_RGB_INGESTION_PRIMARY_INFERENCE/SAMPLE_RGB_STREAM/1.avi
Starting pipeline
2021-06-29 16:23:52.625964: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-29 16:23:53.160714: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2499995000 Hz
2021-06-29 16:23:53.161177: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f29049caa70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-06-29 16:23:53.161210: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-06-29 16:23:53.161323: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-06-29 16:23:53.161453: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.161984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:1e.0
2021-06-29 16:23:53.162017: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-29 16:23:53.162084: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-06-29 16:23:53.162112: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-06-29 16:23:53.162135: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-06-29 16:23:53.162167: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-06-29 16:23:53.162194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-06-29 16:23:53.162222: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-06-29 16:23:53.162327: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.162856: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.163321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0
2021-06-29 16:23:53.163463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-29 16:23:53.163484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0
2021-06-29 16:23:53.163494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N
2021-06-29 16:23:53.163617: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.164180: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.164772: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.165289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9952 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5)
2021-06-29 16:23:53.166889: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f29063c76f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-06-29 16:23:53.166918: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2021-06-29 16:23:53.167432: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.167976: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:1e.0
2021-06-29 16:23:53.168013: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-29 16:23:53.168033: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-06-29 16:23:53.168050: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-06-29 16:23:53.168067: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-06-29 16:23:53.168079: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-06-29 16:23:53.168095: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-06-29 16:23:53.168119: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-06-29 16:23:53.168198: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.168744: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.169216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0
2021-06-29 16:23:53.169246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-29 16:23:53.169262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0
2021-06-29 16:23:53.169272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N
2021-06-29 16:23:53.169380: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.169946: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-06-29 16:23:53.170454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9952 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5)
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: ENSEMBLE_MODEL
2021-06-29 16:23:54.340138: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-06-29 16:23:56.259073: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
ERROR: infer_trtis_server.cpp:258 Triton: TritonServer response error received., triton_err_str:Unsupported, err_msg:in ensemble 'ENSEMBLE_MODEL', failed to get input buffer in CPU memory
ERROR: infer_trtis_backend.cpp:586 TRTIS server failed to parse response with request-id:0 model:
ERROR: infer_trtis_backend.cpp:341 failed to specify dims after running inference failed on model:ENSEMBLE_MODEL, nvinfer error:NVDSINFER_TRTIS_ERROR
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Aborted (core dumped)
Any solution to this error?
Will be glad to send more information if needed.
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
