nv-nsight-cu-cli + pytorch throws "free(): invalid pointer" error


My system specs are:
Driver: 440.33.01
CUDA: 10.2
Docker: pytorch:19.12-py3

For a simple python file


import torch

When I run

$ python foo.py

the application runs fine.

However, when I run

$ nv-nsight-cu-cli python foo.py

I get,

free(): invalid pointer
==ERROR== The application returned an error code (6)
==WARNING== No kernels were profiled
==WARNING== Profiling kernels launched by child processes requires the --target-processes all option

However, if I comment out the “import torch” statement, and run

$ nv-nsight-cu-cli python foo.py

the error goes away. This indicates that there is some weird interaction between nv-nsight-cu-cli and pytorch. I read a few related issues on github (https://github.com/pytorch/pytorch/issues/19739) and it appears this error could be because of the differences in the CXX_ABI flag (D_GLIBCXX_USE_CXX11_ABI) used to compile different libraries / binaries.

Can you please look into this?

Thanks for the report!
We are looking into it.

Just to add on this, I get the same error with Keras (Tensorflow Backend) as well. My environment is the same, that is, Driver version: 440.33.01, CUDA version: 10.2, Tensorflow version: 1.13

Is there any update on this? Is it fixed?

We isolated and fixed the issue.
A fixed version will be shipped in the next container release.

OK That’s great thanks. BTW may I know what was causing it, Because I tried to install pytorch from source in the container and it also didn’t work. However I installed a nightly pytroch binary and it worked.
For my purposes I actually need to make some changes in pytorch source and use that version. But this issue prevents me to do that. Is there any way?