The target application terminated with signal 11

Developer Tools Nsight Systems Profiling Linux Targets
1

./nsys/bin/nsys profile -t cuda,osrt,nvtx -o /debug_info/261153-mphrp-worker-0–1 -w true python /opt/src/main.py

then the program terminated, the error msg:

The target application terminated with signal 11 (SIGSEGV)
Processing events...

The target application terminated with signal 11 (SIGSEGV)
Processing events...
Saving temporary "/tmp/nsys-report-268d-15cb-3b96-1580.qdstrm" file to disk..

But, if no nsys profile, the program is ok. The nsys version is 2021.3.1, and the cuda version is cuda-11.2

2

Hi @lwk111111
Can you please try to upgrade nsys to the latest version and check if the issue persist?
See Nsight Systems - Get Started | NVIDIA Developer

3

I have upgrade the latest ver, but the problem exists.
The problem occurs in the container, but there is no problem on the host machine

4

Hi @Guy_Sz thanks for your response
I have upgrade the latest ver, but the problem exists.
The problem occurs in the container, but there is no problem on the host machine

5

the python program like this

def calculate_sum(n):
    sum = 0
    for i in range(n):
        sum += i
    return sum

def main():
    result = calculate_sum(1000000000)
    print("Result:", result)

if __name__ == '__main__':
    print("start")
    main()
    print("end")

When i use nsys profile like this below.
./nsys profile -t cuda -o baseline -w true --force-overwrite true python main.py
the program only print start and exited, not run the main function.

6

Can you please share the generated report file?

7

baseline_2.txt (146.7 KB)

this forum does not allow to upload qdrep file， so i change the qdrep file to txt

8

Can you please zip the report file instead of changing its type?

9

OK
baseline_2.zip (47.6 KB)

10

The report shows that the target application was profiled with nsys 2021.3.1
Can you please double-check that you are using the latest nsys? (2024.4)

image
image679×844 22.2 KB

11

I have used the latest version, the report file
baseline_3.zip (42.2 KB)

12

I believe I managed to reproduce your issue on my local machine (ubuntu 22.04).
The script you provided runs fine when I use Python3, with and without nsys.
With Python2.7 (like your use-case), the program gets killed due to memory exhaustion (regardless if I run with nsys or not).

I also see from the provided report that the program exited with signal 9 (kill), which is consistent with the findings.

On your setup, the script doesn’t crush when running without nsys?
If you reduce the number you pass to “calculate_sum()”, does the issue persist?

13

thanks for your response.
The problem only occurs in docker container, the file below is generated by my production environment.
from the debug info i find that the program does not execute the main and exited.
dzdebug-from-1261153-ssvvo-worker-2.zip (56.6 KB)

14

@rknight can you see if you can figure out the Docker portion of this issue?

15

Can you share your Docker run command that you used to launch the Docker container?

16

The cmd like this:
./nsys/bin/nsys profile -t cuda,osrt,nvtx -o /debug_info/dzdebug-from-1261153-mphrp-worker-0–1 -w true python /opt/src/main.py

17

Hi lwk111111,

Thanks for sharing your nsys command line. Can you also share the command you used to launch the Docker container? I’m wondering if that command can be adjusted to give the container access to more memory.