TensorRT Error: Can't identify the cuda device. Running on device 0

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
1

Description

I am trying to perform inference of an SSD_MobileNet_V2 frozen graph inside a docker container (tensorflow:19.12-tf1-py3) . Here is the code that I have used to run load the model.

with tf.Session() as sess:
    # First deserialize your frozen graph:
    with tf.gfile.GFile(path, 'rb') as f:
        frozen_graph = tf.GraphDef()
        frozen_graph.ParseFromString(f.read())
    # Now you can create a TensorRT inference graph from your
    # frozen graph:
    converter = trt.TrtGraphConverter(
        input_graph_def=frozen_graph,
        nodes_blacklist=output_node_names, #output nodes
        is_dynamic_op=True) 
    trt_graph = converter.convert()
#     saved_model_dir_trt = "/tensorrt-exp/frozen_graph_sets/tensorrt_model.trt"
#     converter.save(saved_model_dir_trt)
    # Import the TensorRT graph into a new graph and run:
    output_node = tf.import_graph_def(
        trt_graph,
        return_elements=output_node_names)
    sess.run(output_node)

When I run it, I am getting warning messages and errors which are listed below

2021-01-07 09:12:15.870404: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 440.100.0
2021-01-07 09:12:15.870433: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 440.100.0
2021-01-07 09:12:15.870444: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 440.100.0
2021-01-07 09:12:15.895578: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz
2021-01-07 09:12:15.897850: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4dd0210 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-07 09:12:15.897908: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-01-07 09:12:15.913938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2021-01-07 09:12:16.136353: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2021-01-07 09:12:16.137217: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-01-07 09:12:16.358155: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 402 ops of 53 different types in the graph that are not converted to TensorRT: Sum, Where, Split, TopKV2, Assert, Equal, Select, Greater, Size, GatherV2, Fill, GreaterEqual, TensorArrayWriteV3, AddV2, Exit, NoOp, Pack, LoopCond, Merge, NextIteration, Switch, TensorArraySizeV3, TensorArrayV3, Placeholder, Maximum, StridedSlice, Cast, Minimum, Shape, TensorArrayGatherV3, NonMaxSuppressionV5, TensorArrayScatterV3, Reshape, TensorArrayReadV3, FusedBatchNormV3, Enter, Squeeze, ConcatV2, Unpack, Less, Range, ZerosLike, Transpose, RealDiv, Tile, Pad, LogicalAnd, Slice, Mul, Const, Sub, ExpandDims, Identity, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2021-01-07 09:12:16.363051: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:633] Number of TensorRT candidate segments: 40
2021-01-07 09:12:16.493432: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:696] Couldn't get current device: unknown error
2021-01-07 09:12:16.493471: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493476: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0 
2021-01-07 09:12:16.493539: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_0 added for segment 0 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.493586: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493591: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0 
2021-01-07 09:12:16.493627: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_1 added for segment 1 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.493683: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493690: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0 
2021-01-07 09:12:16.493745: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_2 added for segment 2 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.493808: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493818: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0 
2021-01-07 09:12:16.493872: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_3 added for segment 3 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.493931: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493938: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0 
2021-01-07 09:12:16.493985: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_4 added for segment 4 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.494030: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.494035: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0 
2021-01-07 09:12:16.494078: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_5 added for segment 5 consisting of 9 nodes succeeded.
2021-01-07 09:12:16.494115: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.494121: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0

Environment

TensorRT Version: TensorRT version: (6, 0, 1)
GPU Type: X86 GTX 1660 Ti
Nvidia Driver Version: 440.100
CUDA Version: 10.2
CUDNN Version:
Operating System + Version: Ubuntu + 20.04
Python Version (if applicable): 3.6
TensorFlow Version (if applicable): 1.15
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/tensorflow:19.12-tf1-py3

3

Hi @sk.ahmed401,

Seems to be you are using trt older version (6). Please use latest trt release.
You can use tf-trt or tf-onnx-trt.
For your reference.

Thank you.

4

@spolisetty, thanks for the response. The model I am trying to optimize is from tf.1.15.0. So I am using the same version container for the optimization.

If I want to check the above paths, which one do you suggest to try.

5

@sk.ahmed401,
You can download latest release from
https://developer.nvidia.com/nvidia-tensorrt-7x-download.
And you can choose any of the approach as per your convenience.

Thank you.

1 Like