Written data wraps when Submitting writes > max_direct_io_size_kb

Accelerated Computing GPU-Accelerated Libraries
1

Hi

I’m using GDS in O_DIRECT mode with the async API. My problem is that if I submit writes with cuFileWriteAsync() with a size > max_direct_io_size_kb the written data wraps at max_direct_io_size_kb.

I.e. when i read the written data it is correct until max_direct_io_size_kb where data from source buffer offset 0 is found and not from source buffer offset max_direct_io_size_kb.

The behavior is the same for different values of max_direct_io_size_kb .

As long as a don’t submit any writes > max_direct_io_size_kb all the written data is correct.
Is this expected?

I see no errors with gds_stat or in cufile log, and no cuFile calls returns any error.

2

This looks like a bug from your description.
Can you please share the cuda toolkit and driver version you are using and output of tool gdscheck -p.

3

Thanks for the quick reply, see software details in spoilers below.

Driver Version: 555.42.02 CUDA Version: 12.5

$ /usr/local/cuda/gds/tools/gdscheck -p

Summary

GDS release version: 1.10.0.4
nvidia_fs version: 2.20 libcufile version: 2.12
Platform: x86_64

ENVIRONMENT:

=====================
DRIVER CONFIGURATION:

NVMe : Supported
NVMeOF : Unsupported
SCSI : Unsupported
ScaleFlux CSD : Unsupported
NVMesh : Unsupported
DDN EXAScaler : Unsupported
IBM Spectrum Scale : Unsupported
NFS : Unsupported
BeeGFS : Unsupported
WekaFS : Unsupported
Userspace RDMA : Unsupported
–Mellanox PeerDirect : Disabled
–rdma library : Not Loaded (libcufile_rdma.so)
–rdma devices : Not configured
–rdma_device_status : Up: 0 Down: 0

CUFILE CONFIGURATION:

properties.use_compat_mode : false
properties.force_compat_mode : false
properties.gds_rdma_write_support : true
properties.use_poll_mode : true
properties.poll_mode_max_size_kb : 16384
properties.max_batch_io_size : 128
properties.max_batch_io_timeout_msecs : 5
properties.max_direct_io_size_kb : 16384
properties.max_device_cache_size_kb : 131072
properties.max_device_pinned_mem_size_kb : 33554432
properties.posix_pool_slab_size_kb : 4 1024 16384
properties.posix_pool_slab_count : 128 64 32
properties.rdma_peer_affinity_policy : RoundRobin
properties.rdma_dynamic_routing : 0
fs.generic.posix_unaligned_writes : false
fs.lustre.posix_gds_min_kb: 0
fs.beegfs.posix_gds_min_kb: 0
fs.weka.rdma_write_support: true
fs.gpfs.gds_write_support: false
profile.nvtx : false
profile.cufile_stats : 0
miscellaneous.api_check_aggressive : false
execution.max_io_threads : 8
execution.max_io_queue_depth : 128
execution.parallel_io : true
execution.min_io_threshold_size_kb : 16384
execution.max_request_parallelism : 8
properties.force_odirect_mode : true
properties.prefer_iouring : false

GPU INFO:

GPU index 0 NVIDIA RTX A2000 bar:1 bar size (MiB):8192 supports GDS, IOMMU State: Disabled
GPU index 1 NVIDIA RTX A2000 bar:1 bar size (MiB):8192 supports GDS, IOMMU State: Disabled

PLATFORM INFO:

Found ACS enabled for switch 0000:80:03.1
Found ACS enabled for switch 0000:c0:03.1
IOMMU: disabled
Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
Cuda Driver Version Installed: 12050
Platform: Super Server, Arch: x86_64(Linux 6.5.0-21-generic)
Platform verification succeeded

$ apt list --installed | grep nvidia

Summary

libnvidia-cfg1-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
libnvidia-common-555/unknown,now 555.42.02-0ubuntu1 all [installed,automatic]
libnvidia-compute-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
libnvidia-compute-555/unknown,now 555.42.02-0ubuntu1 i386 [installed,automatic]
libnvidia-decode-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
libnvidia-decode-555/unknown,now 555.42.02-0ubuntu1 i386 [installed,automatic]
libnvidia-encode-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
libnvidia-encode-555/unknown,now 555.42.02-0ubuntu1 i386 [installed,automatic]
libnvidia-extra-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
libnvidia-fbc1-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
libnvidia-fbc1-555/unknown,now 555.42.02-0ubuntu1 i386 [installed,automatic]
libnvidia-gl-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
libnvidia-gl-555/unknown,now 555.42.02-0ubuntu1 i386 [installed,automatic]
nvidia-compute-utils-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
nvidia-dkms-555-open/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
nvidia-driver-555-open/unknown,now 555.42.02-0ubuntu1 amd64 [installed]
nvidia-firmware-555-555.42.02/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
nvidia-fs-dkms/unknown,now 2.20.5-1 amd64 [installed,automatic]
nvidia-fs/unknown,now 2.20.5-1 amd64 [installed,automatic]
nvidia-gds-12-5/unknown,now 12.5.0-1 amd64 [installed]
nvidia-kernel-common-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
nvidia-kernel-source-555-open/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
nvidia-prime/jammy,jammy,now 0.8.17.1 all [installed,automatic]
nvidia-settings/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
nvidia-utils-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]
xserver-xorg-video-nvidia-555/unknown,now 555.42.02-0ubuntu1 amd64 [installed,automatic]

1 Like
4

Thanks for the information. This issue is fixed in CUDA toolkit version 12.6 and above.