Environment:
- 8 gpus machine with 2080ti
- cuda 11.4
- using the docker image
nvcr.io/nvidia/tensorflow:21.11-tf2-py3
Command:
dlprof --reports=detail --mode=tensorflow2 <COMMAND>
Model trains fine, limited the number of steps such that it trains for a couple of minutes only.
Error:
Creating final output files...
Processing [===============================================================100%]
Saved report file to "/tmp/nsys-report-e176-9bcd-0e6a-6a6a.qdrep"
Exporting 26015306 events: [===============================================100%]
Exported successfully to
/tmp/nsys-report-e176-9bcd-0e6a-6a6a.sqlite
Report file moved to "/app/./nsys_profile.qdrep"
Report file moved to "/app/./nsys_profile.sqlite"
[DLProf-17:26:01] DLprof completed system call successfully
[DLProf-17:26:04] Initializing Nsight Systems database
[DLProf-17:28:34] Reading System Information from Nsight Systems database
[DLProf-17:28:34] Reading Domains from Nsight Systems database
[DLProf-17:28:34] Error Occurred:
[DLProf-17:28:34] Nsight Systems did not detect any NVTX traces. Please check your script and try again.
nsys_profile.qdrep
and nsys_profile.sqlite
are created correctly.