Dlprof unable to create dlprof_dldb.sqlite

Development Tools Nsight Systems Profiling Linux Targets
,
#1

Environment:

  • 8 gpus machine with 2080ti
  • cuda 11.4
  • using the docker image nvcr.io/nvidia/tensorflow:21.11-tf2-py3

Command:

dlprof --reports=detail --mode=tensorflow2 <COMMAND>

Model trains fine, limited the number of steps such that it trains for a couple of minutes only.

Error:

Creating final output files...
Processing [===============================================================100%]
Saved report file to "/tmp/nsys-report-e176-9bcd-0e6a-6a6a.qdrep"
Exporting 26015306 events: [===============================================100%]

Exported successfully to
/tmp/nsys-report-e176-9bcd-0e6a-6a6a.sqlite
Report file moved to "/app/./nsys_profile.qdrep"
Report file moved to "/app/./nsys_profile.sqlite"

[DLProf-17:26:01] DLprof completed system call successfully
[DLProf-17:26:04] Initializing Nsight Systems database
[DLProf-17:28:34] Reading System Information from Nsight Systems database
[DLProf-17:28:34] Reading Domains from Nsight Systems database
[DLProf-17:28:34] Error Occurred:
[DLProf-17:28:34] Nsight Systems did not detect any NVTX traces.  Please check your script and try again.

nsys_profile.qdrep and nsys_profile.sqlite are created correctly.

#2

Hi Thomas, please be advised that DLProf has been sunset and is no longer being supported. The recommendation is to directly use nsight systems, or use the native tensorflow profiler.