Jetson Xavier Official Tensorflow Package - can't initialize CUPTI

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hi all, I’ve installed the official package of Tensorflow for Jetson Xavier on my Xavier board, and I’m having issues initializing CUPTI to perform some GPU runtime metric collection.

My setup:
CUDA toolkit version: nvcc --version says “Cuda compilation tools, release 10.0, V10.0.117”
Tensorflow (within a virtualenvironment): tensorflow-gpu==1.13.0+nv19.2

Error from Tensorflow application in which I enable GPU and CPU execution metric collection using code like:

sess.run(g,
               options=config_pb2.RunOptions(
                   trace_level=config_pb2.RunOptions.FULL_TRACE),...)

and this triggers dynamically loading the CUPTI object file (script output log below):

Parsing Inputs...                                                                                                                                                       
2019-02-24 13:39:39.129051: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcupti.so.10.0 locally                         
2019-02-24 13:39:39.130273: E tensorflow/core/platform/default/device_tracer.cc:142] Failed to initialize CUPTI: 15 
...
==================Model Analysis Report======================                                                                                                           
Incomplete shape.                                                                                                                                                       
                                                                                                                                                                        
                                                                                                                                                                        
Found accelerator operation but misses accelerator stream stats!                                                                                                        
                                                                                                                                                                        
It's likely a gpu tracing issue rather than tf-profiler issue.                                                                                                          
If you found your operation missing accelerator time, consider filing a bug to xprof-dev@!

I found in the CUPTI API docs https://docs.nvidia.com/cupti/Cupti/r_main.html that CUPTI will throw errno 15 (Failed to init) when “… the CUDA driver version is not compatible with the CUPTI version.”

Is this an issue with the official Tensorflow Package for Jetson Xavier? Any help would be greatly appreciated!

2

Hi,

Do you meet the same error without collecting trace?

We will try to reproduce this issue internally.
Will update more information with you later.

Thanks.

3

Hi AastaLLL, thanks for looking into this. I do not get this same error when not collecting trace as above.

4

Hi,

We can reproduce this issue in our side and already pass it to our internal team for investigation.
Will update information with you for any progress.

Thanks.

5

You need CUPTI path setting to use trace.

export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:${LD_LIBRARY_PATH}

You can run it after you build tensorflow.

6

Hi naisy, unfortunately I’m not building tensorflow on my Xavier board.

7

Hi,

Sorry for keeping you waiting.

This issue can be solved if you execute TensorFlow with root authority.
Due to our recent permission restrictions, all the CUDA tools (cuda-gdb and cupti) need to be executed with root authority.

Thanks.

1 Like