Hello. I’ve used gstreamer-1.20 for a long time with no problem on cuda 11.4 in docker (image nvidia/cudagl:11.4.0-devel-ubuntu20.04). But now i want to update gstreamer to version 1.22 due to some useful changes.
Up to now my application that successfully works on 1.20 + 11.4 doesn’t work on 1.22 + 11.4. I tried cuda-memcheck and this is what i saw there:
Program hit CUDA_ERROR_NOT_FOUND (error 500) due to “named symbol not found” on CUDA API call to cuGetProcAddress. - 4 times or even more
ERROR: 1: [cudaResources.cpp::ScopedCudaStream::37] Error Code 1: Cuda Runtime (invalid device context)
========= Program hit cudaErrorDeviceUninitialized (error 201) due to “invalid device context” on CUDA API call to cudaMalloc.
Program hit cudaErrorDeviceUninitialized (error 201) due to “invalid device context” on CUDA API call to cudaMallocHost.
and so on.
I see device in nvidia-smi and see some gpu utilization at start until the crash.
Could anybody give me an advice what to do? I will appreciate any help)
Is there any documentation from the gstreamer team about what versions of CUDA are supported for 1.22? Is it possible that 1.22 requires a newer version of cuda?
Finally i think CUDA_ERROR_INVALID_CONTEXT is a problem we should solve to continue. I’ve added a processing of cuda operations return value and now i see that all calls in docker with gstreamer 1.22 return not cudaSuccess while calls in docker with gstreamer 1.20 return cudaSuccess.
I didn’t say that both environments are the same except gstreamer version: the same hardware, the same docker base image, the same docker file except 8 lines of gstreamer and all of its plugins installation.
If i use in gstreamer pipeline avdec_h264 as decoding block - all is ok. But when i use nvh264dec, after pipeline initialization i get 201 error when i make cudaMalloc, cudaMallocHost etc. If i use some threads working with cuda only first of them reports error in making cuda api calls. But i think engines are not correct too. They are being built after gstreamer initialization. So maybe gstreamer does some strange actions with cuda. Maybe there are some blocking mechanisms or something like that…
Thanks for the details. It sounds to me like this isn’t so much an issue with the CUDA-MEMCHECK tool (this forum) but either for the gstreamer support team or CUDA Programming forum. Since the problem isn’t in your own source code, memcheck probably won’t help you fix it. I don’t think the folks on this forum will know how to fix the issue your running into. I’d recommend posting on those support channels.