NVSHMEM Setup

Is it possible to prototype NVSHMEM solutions using a GTX3090 ? I have been testing installations but I get a problem when trying to use nvshmem_malloc. Is it recommended to use MPI + OpenShmem instead ? Do you have any specific tutorial to understand the problem ?

Error as seen:

WARN: GDRCopy open call failed, falling back to not using GDRCopy
src/mem/mem.cpp:416: non-zero status: 101 cuMemCreate failed
src/team/team_internal.cu:287: NULL value nvshmemi_psync_pool allocation failed
src/init/init.cu:762: non-zero status: 2 team setup failed
src/init/init.cu:796: non-zero status: 7 nvshmemi_common_init failed …src/init/init_device.cu:nvshmemi_check_state_and_init:44: nvshmem initialization failed, exiting
src/util/cs.cpp:21: non-zero status: 16: No such file or directory, exiting… mutex destroy failed

From here:

NVSHMEM requires the following hardware:
NVIDIA Data Center GPU of the NVIDIA Volta™ GPU architecture or later.

RTX3090 is not a datacenter GPU.

There are additional documentation and tutorial resources linked from here.
Scroll down to the bottom for the “Resources” section.

Hi Robert,

Thank you for the reply, it wasn’t clear that the errors were due the card not being a datacentre card. I take that the 3090 would not be fit to develop a library that uses nvshmem, what are the recommended alternatives ?

Regards,

Guilherme

Thank you Robert, I was trying to find a solution to use the same codebase on NVSHMEM for single gpu and multi-gpu. I take that the solution is to use ifdefs to skip nvshmem calls.