For anyone at NVIDIA.
Would it be possible to add CUDA driver support for changing the mappings of allocated memory regions? For example, say I have allocated two regions using something like cudaMalloc, where A* points to region A and B* points to region B. Would it be possible to include a function that would change the mappings such that A* points to region B and B* points to region A without changing the actual values of A* or B*? Even the ability to call cudaMalloc and specify the address being allocated to would be sufficient. I am hoping that GPUs implement some form of virtual memory and that this would be possible by changing some page table entries in the driver.
We are currently trying to add support for context switching from one GPU to another in Ocelot and this would significantly make my life easier. The main problem we are having is that after we copy all of the state from one GPU to another, any existing pointers to cudaMalloced regions are no longer valid. We are currently handling this via pointer analysis, but this only works in cases where pointers are passed as parameters to kernels and not embedded in other memory regions. More generally we could handle it by allocating all memory using the zero-copy mechanism, but we would like to avoid the performance overhead.