Dlprof unable to create dlprof_dldb.sqlite

Environment:

  • 8 gpus machine with 2080ti
  • cuda 11.4
  • using the docker image nvcr.io/nvidia/tensorflow:21.11-tf2-py3

Command:

dlprof --reports=detail --mode=tensorflow2 <COMMAND>

Model trains fine, limited the number of steps such that it trains for a couple of minutes only.

Error:

Creating final output files...
Processing [===============================================================100%]
Saved report file to "/tmp/nsys-report-e176-9bcd-0e6a-6a6a.qdrep"
Exporting 26015306 events: [===============================================100%]

Exported successfully to
/tmp/nsys-report-e176-9bcd-0e6a-6a6a.sqlite
Report file moved to "/app/./nsys_profile.qdrep"
Report file moved to "/app/./nsys_profile.sqlite"

[DLProf-17:26:01] DLprof completed system call successfully
[DLProf-17:26:04] Initializing Nsight Systems database
[DLProf-17:28:34] Reading System Information from Nsight Systems database
[DLProf-17:28:34] Reading Domains from Nsight Systems database
[DLProf-17:28:34] Error Occurred:
[DLProf-17:28:34] Nsight Systems did not detect any NVTX traces.  Please check your script and try again.

nsys_profile.qdrep and nsys_profile.sqlite are created correctly.

Hi Thomas, please be advised that DLProf has been sunset and is no longer being supported. The recommendation is to directly use nsight systems, or use the native tensorflow profiler.