Hi all, I’ve installed the official package of Tensorflow for Jetson Xavier on my Xavier board, and I’m having issues initializing CUPTI to perform some GPU runtime metric collection.
CUDA toolkit version: nvcc --version says “Cuda compilation tools, release 10.0, V10.0.117”
Tensorflow (within a virtualenvironment): tensorflow-gpu==1.13.0+nv19.2
Error from Tensorflow application in which I enable GPU and CPU execution metric collection using code like:
sess.run(g, options=config_pb2.RunOptions( trace_level=config_pb2.RunOptions.FULL_TRACE),...)
and this triggers dynamically loading the CUPTI object file (script output log below):
Parsing Inputs... 2019-02-24 13:39:39.129051: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcupti.so.10.0 locally 2019-02-24 13:39:39.130273: E tensorflow/core/platform/default/device_tracer.cc:142] Failed to initialize CUPTI: 15 ... ==================Model Analysis Report====================== Incomplete shape. Found accelerator operation but misses accelerator stream stats! It's likely a gpu tracing issue rather than tf-profiler issue. If you found your operation missing accelerator time, consider filing a bug to xprof-dev@!
I found in the CUPTI API docs https://docs.nvidia.com/cupti/Cupti/r_main.html that CUPTI will throw errno 15 (Failed to init) when “… the CUDA driver version is not compatible with the CUPTI version.”
Is this an issue with the official Tensorflow Package for Jetson Xavier? Any help would be greatly appreciated!