Host-pinned memory bookeeping?

Accelerated Computing CUDA CUDA Programming and Performance
1

I will try to describe this the best I can.

I have an application that utilizes host-pinned memory (only) to provide some kernel internal information, akin to a performance profiling tool. This tool can be activated and deactivated, and in the latter case, host-pinned memory is not allocated. In an edge-case, a kernel that consumed almost all device memory (leaving only a few MBs free) was launched. If I test this workload with the tool deactivated, works. If I activate my tool, it OOMs when trying to allocate a huge in-device array (which happens after host-pinnede memory is allocated).

Question is, does host-pinned memory has any hidden cost in terms of device memory? I am thinking bookeeping or any other similar cost.

2

There may be a “hidden cost” in terms of device page table entries. I don’t know the exact structure of the device page table but I know it exists.

3

This is interesting, thank you. I assume there should be some internal data structures to perform some tasks like the ones you suggest, question is how big they are and how do they scale, i.e., fixed size or dependent on some dimensions. Do you have an idea of where to look at for this type f documentation?

4

I don’t think it is documented anywhere.

5

I see. Well, I think I can try some benchmarking and checking this overhead in specific scenarios. Thanks!

6

Here is one datapoint. It’s really not conclusive due to CUDA lazy allocation. You don’t really know what is happening under the hood. But it is at least consistent with the idea that a host-pinned allocation may require some device memory:

# cat t312.cu
#include <iostream>

int main(){

  cudaSetDevice(0);
  size_t tot, fre;
  cudaMemGetInfo(&fre, &tot);
  std::cout << "total: " << tot << " free: " << fre << std::endl;
  char *d, *d1;
  size_t s = 1048576ULL * 4096;
  cudaHostAlloc(&d, s, cudaHostAllocDefault);
  cudaMemGetInfo(&fre, &tot);
  std::cout << "total: " << tot << " free: " << fre << std::endl;
  cudaHostAlloc(&d1, s, cudaHostAllocDefault);
  cudaMemGetInfo(&fre, &tot);
  std::cout << "total: " << tot << " free: " << fre << std::endl;
}
# nvcc -o t312 t312.cu
# compute-sanitizer ./t312
========= COMPUTE-SANITIZER
total: 23583784960 free: 23344316416
total: 23583784960 free: 23335927808
total: 23583784960 free: 23327539200
========= ERROR SUMMARY: 0 errors
#

Could be around 10MB for a 4GB allocation.

1 Like
7

Great data point, and useful as orientation, Thanks again.

8

To finalize this discussion, I tested with power of two sizes and a single host allocation. Device memory consumption starts scaling with a factor of two starting from 1G allocs, so in the model I tested (A100), 1GB of host-pinned memory consumes around 2MB of device memory, 2GB consumes 4MB, and so on.

9

So one 8 byte page entry per 4KB host page?

(+ in theory higher-level page directories?)