Deep Learning Institute profiling error

Community Information Forum Feedback
#1

Hi,

I’m in the course, Accelerating CUDA C++ Applications with Concurrent Streams, on part 4: Nsight Systems Setup. I follow the instructions and type the command “make profile”. When it tries to import the qdstrm file, I get errors.

error1
error13839×1979 679 KB

I’m not sure where to post this error. There’s no obvious place to go for help with DLI. I tried to open a ticket here, https://www.nvidia.com/en-us/support/consumer/ which requires me to login even through I’m already logged in. However, the login doesn’t work.

Thanks for your help.

#2

Hi @FractalMath,

Welcome to the NVIDIA Developer forums. Unfortunately, there is no DLI forum at the moment. Let me see if I can find someone from the DLI team to help here.

Best,
Tom

#3

Hi @FractalMath, thanks so much for reaching out for support. I maintain this training and hope that I can be of service.

I attempted to duplicate your error and was unable to do so. When I visit the training and do make profile from within section 04, everything works as expected.

That said, I believe that if you spin up the content again, you will likely find it working as expected, though please let me know if that is not the case.

I was able to find a bit more information on the error you displayed on this Developer Forum Thread. It looks like there is some kind of timeout occurring with QdstrmImporter which can often be caused by either a massive qstrm file, which I have no reason to believe is true in your case, or perhaps some underlying issue on the VM you were working on. If it were the latter, it is my hope that restarting the training will solve the issue for you.

Again, please do let me know if restarting the content does not solve your issue and I’ll be eager to help get you unblocked. Thanks so much for your interest in the course.

#4

Hi @jwyatt, I spun up the content two additional times and the same problem remains. The file baseline-report.qdstrm is 57k. The QdstrmImporter takes a very long time during the import before it errors out.

Thanks for your help.

#5

Hi @FractalMath, sorry to hear that did not resolve the issue for you.

I was able to replicate the error, though I had to generate the report 3 times before running into it. I’m still not certain what the root cause is (though I will report it back to you when I know), though I do have a solution that will get you unstuck.

Following the advice of this forum post, we are able to turn off CPU sampling and only trace the CUDA APIs (which is what we are interested in for this material) by adding the nsys flags --sample=none --trace=cuda. Thus, a full profile command would look something like nsys profile --stats=true --force-overwrite=true --sample=none --trace=cuda -o baseline-report ./baseline. I tried this out and it works, and, gives me everything I need within the Nsight Systems GUI.

I will look to resolve this issue more at the root when I’m able, and will also queue up edits to the content so that the use of Makefile is enabled once again without error.

Please let me know if you have any issues going forward, and thanks again for reaching out.

#6

Adding those flags worked.

Thank you for your help.

#7

Happy to help, and sorry again for the inconvenience.

I’ve been able to discover that this issue was a known issue with earlier versions of Nsight Systems and has since been addressed. Therefore I updated the course to use the very latest version of Nsight Compute (version 2021.1) and I can confirm that it solves the issue you reported.

Again, thanks so much for reporting the issue, your feedback helps us make sure the course is in its best possible configuration.

Please let us know if we can be of further support.

1 Like