Gstreamer-1.22 with Cuda 11.4

Developer Tools CUDA Developer Tools CUDA-MEMCHECK
,
1

Hello. I’ve used gstreamer-1.20 for a long time with no problem on cuda 11.4 in docker (image nvidia/cudagl:11.4.0-devel-ubuntu20.04). But now i want to update gstreamer to version 1.22 due to some useful changes.

Up to now my application that successfully works on 1.20 + 11.4 doesn’t work on 1.22 + 11.4. I tried cuda-memcheck and this is what i saw there:

Program hit CUDA_ERROR_NOT_FOUND (error 500) due to “named symbol not found” on CUDA API call to cuGetProcAddress. - 4 times or even more

ERROR: 1: [cudaResources.cpp::ScopedCudaStream::37] Error Code 1: Cuda Runtime (invalid device context)
========= Program hit cudaErrorDeviceUninitialized (error 201) due to “invalid device context” on CUDA API call to cudaMalloc.

Program hit cudaErrorDeviceUninitialized (error 201) due to “invalid device context” on CUDA API call to cudaMallocHost.

and so on.

I see device in nvidia-smi and see some gpu utilization at start until the crash.

Could anybody give me an advice what to do? I will appreciate any help)

2

Is there any documentation from the gstreamer team about what versions of CUDA are supported for 1.22? Is it possible that 1.22 requires a newer version of cuda?

3

I tried to find this information but i didn’t find anything.
I found that in 1.22 gstreamer has " * New gst-cuda library" and that’s all about cuda…

4

I see here an advice to ignore “error: 201 CUDA_ERROR_INVALID_CONTEXT” - Error when running optimized code but runs fine with debug

Is situation described in that topic related to mine or not? Should I pay attention to this error?

But nevertheless the main thing here is seg fault. I think that it’s because of this error but am i right?

5

Finally i think CUDA_ERROR_INVALID_CONTEXT is a problem we should solve to continue. I’ve added a processing of cuda operations return value and now i see that all calls in docker with gstreamer 1.22 return not cudaSuccess while calls in docker with gstreamer 1.20 return cudaSuccess.

I didn’t say that both environments are the same except gstreamer version: the same hardware, the same docker base image, the same docker file except 8 lines of gstreamer and all of its plugins installation.

6

The new observation.

If i use in gstreamer pipeline avdec_h264 as decoding block - all is ok. But when i use nvh264dec, after pipeline initialization i get 201 error when i make cudaMalloc, cudaMallocHost etc. If i use some threads working with cuda only first of them reports error in making cuda api calls. But i think engines are not correct too. They are being built after gstreamer initialization. So maybe gstreamer does some strange actions with cuda. Maybe there are some blocking mechanisms or something like that…

7

Thanks for the details. It sounds to me like this isn’t so much an issue with the CUDA-MEMCHECK tool (this forum) but either for the gstreamer support team or CUDA Programming forum. Since the problem isn’t in your own source code, memcheck probably won’t help you fix it. I don’t think the folks on this forum will know how to fix the issue your running into. I’d recommend posting on those support channels.

8

Thank you. Asked my question on both forums (cuda and gstreamer) but haven’t received any answer yet ((