How to hook APIs in libnvvm.so Using LD_PRELOAD?

Accelerated Computing CUDA CUDA NVCC Compiler
1

I am using CUDA version 11.5 and want to utilize LD_PRELOAD to hook APIs in libnvvm.so to implement my own version. However, my interception attempt has not been successful.
I wrote the following code to intercept the nvvmAddModuleToProgram function in libnvvm.so:
nvcc.cpp：

#include "/usr/include/nvvm.h"
#include <dlfcn.h>
#include <filesystem>
#include <iostream>
#include <cstdio>
#include <list>
#include <string>
#include <sstream>
#include <vector>
#include <fstream>

#define LIBNVVM "libnvvm.so"
static void* libnvvm = NULL;
std::ofstream logFile("log.txt", std::ios::app);

#define bind_lib(lib) \
if (!libnvvm) { \
    libnvvm = dlopen(lib, RTLD_NOW | RTLD_GLOBAL); \
    if (!libnvvm) { \
        fprintf(stderr, "Error loading %s: %s\n", lib, dlerror()); \
        abort(); \
    } \
}

#define bind_sym(handle, sym, retty, ...) \
typedef retty (*sym##_func_t)(__VA_ARGS__); \
static sym##_func_t sym##_real = NULL; \
if (!sym##_real) { \
    sym##_real = (sym##_func_t)dlsym(handle, #sym); \
    if (!sym##_real) { \
        fprintf(stderr, "Error loading %s: %s\n", #sym, dlerror()); \
        abort(); \
    } \
}

nvvmResult nvvmAddModuleToProgram(nvvmProgram prog, const char *bitcode, size_t size, const char *name) {
    logFile << "here" << std::endl;
    bind_lib(LIBNVVM);
    bind_sym(libnvvm, nvvmAddModuleToProgram, nvvmResult, nvvmProgram, const char*, size_t, const char*);
    return nvvmAddModuleToProgram_real(prog, bitcode, size, name);   
}

I compiled nvcc.cpp into a shared library libnvcc.so using the following command:

g++ -g -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -fPIC nvcc.cpp -shared -o libnvcc.so -ldl

Then, I executed the command:

LD_PRELOAD=/home/qzy/myfile/testcuda/libnvcc.so nvcc -o test test.cu -keep

I expected the nvvmAddModuleToProgram function to hook the original function in libnvvm.so, and if the hook was successful, it should print “here” in log.txt. However, log.txt remains empty, indicating that the hook has failed.
Could you please advise me on how to correctly hook the APIs in libnvvm.so? Thank you!

2

The LD_PRELOAD method only works if the executable you are running explicitly dynamically links against a library providing that symbol/entry point. I’m fairly certain nvcc does not fit that description. For confirmation my suggestion is to run ldd on whichever nvcc you are using, and see what it links against. Identify which one you think actually provides a nvvm symbol/entry point.

I don’t have a recipe to suggest for you. nvcc is a compiler-driver. You might want to start by understanding the detail steps it follows using nvcc --verbose .... or similar. Perhaps you would then look at all the sub-tools that are spawned, learn how to invoke those, and then perhaps use ldd again to see if any of those sub-tools link against a library that contains the symbols/entry points you are attempting to hook.

FWIW its entirely possible that there is no feasible application of LD_PRELOAD method to perform such hooking. Two possibilities are that the libraries involved are statically linked to one of the sub-tools, and/or the library involved is runtime-loaded using e.g. a dlopen/dlsym method to runtime load and runtime link specific entry points.

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.