about mpirun + nvprof profiling


When executing my cuda program with openmpi in my Jetson nano(TX1), I want to check GPU usage.

So I refer to here(https://devblogs.nvidia.com/cuda-pro-tip-profiling-mpi-applications/) and here(http://cs.colby.edu/courses/S14/cs336/online_materials/CUDA_Profiler_Users_Guide.pdf).

But, there is not executed with error.

psl@jetsonmaster-desktop:~/Sharedir$ mpirun -np 2 --hostfile address.txt --map-by node nvprof -o result.%q{OPMI_COMM_WORLD_RANK}.nvprof ./a.out
[jetsonmaster-desktop:11644] [[36386,0],0] usock_peer_send_blocking: send() to socket 29 failed: Broken pipe (32)
[jetsonmaster-desktop:11644] [[36386,0],0] ORTE_ERROR_LOG: Unreachable in file oob_usock_connection.c at line 316
[jetsonmaster-desktop:11644] [[36386,0],0]-[[36386,1],0] usock_peer_accept: usock_peer_send_connect_ack failed
mpirun was unable to find the specified executable file, and therefore
did not launch the job.  This error was first reported for process
rank 1; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command
      line parameter option (remember that mpirun interprets the first
      unrecognized command line token as the executable).

Executable: /usr/local/cuda-10.0/bin/nvprof

How can I solve this?? or How can I execute openmpi with cuda ??


We could not reproduce the issue at our end. The common reason of this kind of error is that the nvprof executable cannot be found.

Please make sure that nvprof is in the PATH environment.

Are you able run nvprof as standalone?
$ nvprof

Please check:
$ which nvprof