AGX Xavier profiling issue with PyProf + pytorch1.7 docker

[r05943077]

I’m trying to profile my pytorch GPU application with PyProf, with the pytorch application executed in docker environment.

However, I got the error message:
(venv) amros@amros-desktop:~/Desktop$ python -m pyprof.prof --csv net.dict
“Idx”,“Direction”,“Sub”,“Module”,“Op”,“Kernel”,“Params”,“Sil(ns)”
no such table: CUPTI_ACTIVITY_KIND_KERNEL
Traceback (most recent call last):
File “/usr/lib/python3.6/runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “/usr/lib/python3.6/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File “/home/amros/Desktop/venv/lib/python3.6/site-packages/pyprof/prof/main.py”, line 21, in
main()
File “/home/amros/Desktop/venv/lib/python3.6/site-packages/pyprof/prof/prof.py”, line 213, in main
kernel = eval(line)
File “”, line 1
no such table: CUPTI_ACTIVITY_KIND_KERNEL
^
SyntaxError: invalid syntax

the issue may be similar to this post:

SyntaxError

Experiment environment:
The pytorch 1.7 docker image is l4t-pytorch:r32.5.0-pth1.7-py3:

NVIDIA L4T PyTorch | NVIDIA NGC

Also I try python 1.9 in docker, but not works
Jetpack version is 4.5
CUDA version is 10.2

Steps to reproduce the issue:
1.
Install PyProf as described in: GitHub - NVIDIA/PyProf: A GPU performance profiling tool for PyTorch models
2.
(profile the whole docker)
nsys profile -f true -o net --export sqlite (run docker to execute_python_file)
or
(run profile in docker)
(run docker) nsys profile -f true -o net --export sqlite execute_python_file
3.
python -m pyprof.parse net.sqlite > net.dict
4.
python -m pyprof.prof --csv net.dict

it seems step 1.~3. goes without error, but in step 4. I got the error message:
no such table: CUPTI_ACTIVITY_KIND_KERNEL
^
SyntaxError: invalid syntax

The pytorch code is as in https://github.com/NVIDIA/PyProf：
1.
import torch.cuda.profiler as profiler
import pyprof
pyprof.init()

some pytorch code in GPU
or
2.
import torch.cuda.profiler as profiler
import pyprof
pyprof.init()
with torch.autograd.profiler.emit_nvtx():
some pytorch code in GPU

Also I try the lenet.py example in the above PyProf git. But none of the above work.

Is there some other way to profile Tensor core/GPU usage in AGX Xavier + pytorch1.7 docker?
Thanks a lot!

[AastaLLL]

Hi,

Thanks for reporting this issue.
We are checking this internally.

To profile GPU tasks, you can also try our nvprof tool.
For example:

$ sudo /usr/local/cuda-10.2/bin/nvprof python3 my.py

Thanks.