Cuda Run time library unload

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,

I am running into cudaErrorCudartUnloading error in my application. I found that this occurs because I am calling cudaFree from destructors of global static variables, and the destructors may be getting called after the cuda run time library is unloaded.

Is there a robust way to force my application to keep the cuda run time library loaded, until after I call all the necessary cudaFree functions. Can I control/delay the unloading of cuda run time library by any means ? For instance, can I have my class maintain certain variables/handles that will force cuda run time library to stay loaded.

Thanks!

2

No. It is a bad design practice to put calls to the CUDA runtime API in constructors that may run before main and destructors that may run after main.

3

The run time API I am calling is essentially a cudaFree command. I want to deallocate device memory pointers owned by my static object, in the destructor of the object. And yes, I understand that the destructors of the static variable may be called in any order during application exit.

Is there a way to check in my destructor if the run time module is already unloaded, and if so suppress calls to cudaFree. I guess that if the module is already unloaded and the context is cleared, any memory should be getting implicitly deleted anyway, and I don’t have to use explicit cudaFree calls.

4

You’re already doing that. The error code returned by the runtime API call is exactly that indication.
In my view, you’re exploring UB as far as CUDA is concerned. I don’t see explicit use of UB as a good design practice. You may think I’m fundamentally wrong or disagree with me. That’s fine; it’s the nature of community. Anyway, the error code you received is the only thing you should expect after CUDA runtime shuts off. Any call into the CUDA runtime at that point would return that error code, even a call that tells you if the CUDA runtime is available or not (there is no such call to my knowledge, but the entire CUDA runtime API is documented: http://docs.nvidia.com/cuda/cuda-runtime-api/index.html#axzz4nzXQGo3P )

5

Hi @Robert_Crovella, could you please elaborate why it is a bad design practice to put calls to the CUDA runtime API in constructors that may run before main and destructors that may run after main?

If I put cudaDeviceReset() in constructor of a global object, it seems ok for the process to correctly finish, and no errors.

6

Here is a general write up. Before deciding that everything is OK with your code, make sure you are doing proper CUDA error checking, including on the calls in the constructor/destructors. The easiest to demonstrate hazard that I have seen is when CUDA calls are in destructors of global objects.

7

@Robert_Crovella I have this problem too, My main application dlopens a library which is linked with a compiled .cu file. I can not have any cuda code linked to my main application. I am using cagra from cuvs. I make several API calls that first create dev_resources and an index. Then several more API calls come in that batch load (150 chunks) data into the input for the cagra build.

The CU file has this:

struct cagra_context_s {
raft::device_resources dev_resources;
cuvs::neighbors::cagra::index_params index_params;
cuvs::neighbors::cagra::search_params search_params;
std::optional<raft::device_matrix<float, int64_t>> dataset_d;
std::optional<cuvs::neighbors::cagra::index<float, uint32_t>> index;

void allocate_dataset(int64_t num_vectors, int64_t vec_size) {
if (!dataset_d) {
dataset_d.emplace(raft::make_device_matrix<float, int64_t>(dev_resources, num_vectors, vec_size));
}
}

void set_index(cuvs::neighbors::cagra::index<float, uint32_t>&& idx) {
index.emplace(std::move(idx)); // moves into the optional
}
};
// the key is the mem_id from CUVS.cpp
std::unordered_map<unsigned long long, cagra_context_s> cagra_context;

If I manually eliminate this context before application exits the application exit is fine. But If I forget to tear down the context and just exit I get the cudaErrorCudartUnloading when atexit runs the destructor for the map.

I tried calling std:atexit in both the main and the library and also tried:

#if 0
// Doesn’t actually work for a halt
// The cudart is getting unloaded before the cagra destructors so they fail
void gpu_cleanup() noexcept {
// stop work, sync, free device buffers, clear containers, etc.
// e.g., for your map of contexts:

for (auto& kv : cagra_context) {
auto key = kv.first;
isccu_free_gpu_mem(key);
}

}

attribute((destructor)) static void lib_fini() { gpu_cleanup(); }

But not matter what I get during exit:

CUDA Error detected. cudaErrorCudartUnloading driver shutting down
app: /home/pmilosla/projects/cuda/cuvs_install/include/rmm/mr/device/cuda_memory_resource.hpp:80: virtual void rmm::mr::cuda_memory_resource::do_deallocate(void*, std::size_t, rmm::cuda_stream_view): Assertion `status__ == cudaSuccess’ failed.
Aborted (core dumped)

What’s a good approach to this? I cant put anything in main, i Can only have some library context which I’ve chose to have as a variable at the top of the file so its available to multiple library API calls

I am guessing what is happening is main app calls dlclose before it runs atexit. I am unclear what is triggering the cudart unload. I havent been able to find it with gdb breakpoints.

8

I guess I don’t really understand the requirements and restrictions.

and

strikes me as an insoluble problem. (In fact I don’t really understand it even from a programming perspective.) Perhaps others will have ideas, and I am certainly not a CAGRA or Rapids expert. Rapids has its own community support system which is separate from these forums, so if no one has ideas here, you might try there as well (i.e. ask the Rapids experts).

Again, I don’t really understand what “I can’t put anything in main” means (yes, I understand the words; I don’t understand it as a design restriction.) But if we leave that alone, then the solution path would seem to me to be to (re-)consider your other position: " I’ve chose to have as a variable at the top of the file"

So move that to within a function scope somewhere. Pass the context around via a function parameter. I think in C++ it is not controversial to say non-constant global scope variables are frowned on, but I acknowledge other viewpoints are possible and valid.

9

If you could point me to a cagra forum thats usable, I would use it, but I think they want everything as a github issue, and I tried asking stuff in issues but I feel its a bit of poor form and lacking in response.

So the main app (surprise its a database) can not know anything about cuda or cagra. It cant be linked against it, and it cant have access to the headers. This is typical when you are writing libraries. The library here is just generically passing data to a cuda aware layer but in stages, because there is a lot of data and it takes a while to decide to use it, build the index and search. So I can pass a pointer to the memory of the, lets say cagra index back, but it cant be a typed pointer and the interface is actually limited to C and not C++ (but both sides are C++). Which I think means the cuda driver unloading will take place anyway when the infrastructure decides to dlclose.

The only thing I have been able to think of so far is to ask for the database to execute a callback on option 2) below

  1. controlled cuda aware exit (i.e. its decided that it happens in the cuda flow)

  2. controlled but non cuda aware exit (i.e. the data manager decided to exit what they were doing and forgot to issue “clean up cuda”)

  3. an error (not anticipated by any user)

10

I asked a question here: [QST] how have a planned but not CUDA/CAGRA aware exit() while CAGRA is in a library? · Issue #1295 · rapidsai/cuvs · GitHub

11

For the moment a I have a callback before the exit in flow 2.