NVSHMEM Setup

Accelerated Computing CUDA CUDA Setup and Installation
1

Is it possible to prototype NVSHMEM solutions using a GTX3090 ? I have been testing installations but I get a problem when trying to use nvshmem_malloc. Is it recommended to use MPI + OpenShmem instead ? Do you have any specific tutorial to understand the problem ?

Error as seen:

WARN: GDRCopy open call failed, falling back to not using GDRCopy
src/mem/mem.cpp:416: non-zero status: 101 cuMemCreate failed
src/team/team_internal.cu:287: NULL value nvshmemi_psync_pool allocation failed
src/init/init.cu:762: non-zero status: 2 team setup failed
src/init/init.cu:796: non-zero status: 7 nvshmemi_common_init failed …src/init/init_device.cu:nvshmemi_check_state_and_init:44: nvshmem initialization failed, exiting
src/util/cs.cpp:21: non-zero status: 16: No such file or directory, exiting… mutex destroy failed

2

From here:

NVSHMEM requires the following hardware:
NVIDIA Data Center GPU of the NVIDIA Volta™ GPU architecture or later.

RTX3090 is not a datacenter GPU.

There are additional documentation and tutorial resources linked from here.
Scroll down to the bottom for the “Resources” section.

3

Hi Robert,

Thank you for the reply, it wasn’t clear that the errors were due the card not being a datacentre card. I take that the 3090 would not be fit to develop a library that uses nvshmem, what are the recommended alternatives ?

Regards,

Guilherme

4

https://www.nvidia.com/content/dam/en-zz/solutions/data-center/data-center-gpu-portfolio-line-card.pdf

5

Thank you Robert, I was trying to find a solution to use the same codebase on NVSHMEM for single gpu and multi-gpu. I take that the solution is to use ifdefs to skip nvshmem calls.