Hi, I have a setup where I am streaming multiple(4) GSML2 cameras (IMX390) into my app, doing a little opencv processing then using the dustynv libraries for a webrtc stream to a locally hosted webpage. My work can all be done in GPU memory and I quickly noted that in my original implementation that a lot of time was spent reading in the frames from the openCv videosource (this copies into cpu memory, that later gets copied again into gpu).
I went down the path of trying to use gstreamer directly in my c++ code. I setup my gstreamer pipline/appsink and tied to convert into the EGLImage but many of the examples I followed made use of nvbuf_utils, which is deprecated in Jetpack 5.1.2 and I never got this working.
From my reading/understanding I needed to use the newer approach with NvBufSurface and surface transforms. Going this route while setting up the NvBufSurface I found out that my version of jetpack it doesn’t support NVBUF_MEM_CUDA_UNIFIED for memory type (Jetpack 6 feature I believe?). It only really appears to support the NVBUF_MEM_DEAFAULT, which if I’m not mistaking, will cause it to hit the CPU again.
Sorry if this post is confusing but my head is spinning a little trying to figure out what is the correct approach. All I really want is to read in camera frames to gpu memery such that I can have the cv::cuda::GpuMat point to it without hitting the cpu. Reading other forum posts it appears possible, I just haven’t pieced it all together yet and was hopping for some help, or an example.
I’m looking at Honey_Patouceul’s example in your second link their I noticed that nvbuf_utils.h is still being imported (this doesn’t appear to exist in jetpack 5.1.2). Commenting out that include however, I get the project to compile. but running any of the sample commands I always get the ERROR **: : Failed to go into PLAYING state.
I tested some gstreamer commands, running gst-launch-1.0 and see everything connect and run from command line, but the same setup is failing when running the code.
I’ll continue testing, perhaps try and merge this example with one of earlier iterations where I thought I was close and get back to you.
Okay, So following the example in your first link I was able to get everything up and running (had to use appsink as nv3dsink wasn’t working for me, even though gst-inspect-1.0 nv3dsink showed the sink was available)
Unfortunately to my surprise running this code was no faster than my original implementation, showing simular cpu utilization with all cores running about 50%. This surprised me as my old implimentation used gstreamer with openCV’s VideoCapture.read(). This method I thought would avoid the cpu bottleneck that occurs when reading frames with opencv by keeping everything in GPU.
Reading other posts I found several discussions around cuGraphicsEGLRegisterImage, cuGraphicsResourceGetMappedEglFrame, and cuGraphicsUnregisterResource running every frame, consuming a good chunk of resources. For my application I timed cuGraphicsEGLRegisterImage() and see it alone runs for ~2.6+ ms (total from all 4 cameras). The discussion didn’t have a satisfying conclusion and I was wondering if there is anymore information that can be provided on working around or avoiding registering on every frame? Can I just register the memory block once somehow?
Here is the other Discussion:
Below is my code (reads in 4 camera frames and sends them out over webrtc). Sorry, my code is very messy as I am using it as a sandbox at this point. Running quick tests, adding pieces to as it gets working, so structure and cleanliness are severely lacking.
I’ve been having difficulty implementing the cuGraphicsEGL-related functions in the initial and destroy phases. It seams that when I try to cache the EGLImages I run into issues with cuGraphicsMapResources. I checked and each argus camera appears to be cycling through 3-4 buffer descripters (I guess Argus has a small pool of NVMM surfacess?). I tried creating a new EGLImage for each buffer and camera combination (say 12 in total) then using the appropriate one based on the egleImage, but this also fails at the resourceGetMapped step. (note for the first frame it works, its the second frame on each camera that fails).
So I think I need a little more help in understanding if I can avoid the cuGraphicsEGLRegisterImage() on every frame, and if so what do I need to do inorder to properly cache these resources for re-use. Perhaps their is some context switching I need to better manage
Okay, so I figured out why it was failing when creating an EGL image cache for each camera/buffer. I needed to call cudaFree(0) once in each thread (each pad probe callback) before I do any cuda access. Correcting this mistake the code runs, and only registers once per buffer per camera (in my case it looks like 4 buffers per camera so I end up with 16 sets of CUgraphicsResource).
Profiling I am still a little disapointed in cpu usage, it has gotten a little better but for some reason I hoped for more. Perhaps switching from the padProbe to an appsink callback itself might help? I was reading a bit about the extra overhead of thread management in the pad probe callback might hurt cpu ussage… Unsure, but hoping for some direction on where else I could look to gain some performance? Perhaps others have taken a different routes at reading in frames thats more efficient?
for those that are curious here is my slightly cleaned up (though still has a lot of timers) code:
I do have a different version where I do cudaAsync for copy the the images into my final frame which reduces the COPY: to sub 100 us. But this isn’t the final state of the code, other operations will occur, so I’m not overly interested in optimizing this portion of the code. I mainly want to ensure I’m getting data to a cv::cuda::GpuMat as efficiently as possible (the step before I start merging everything) what happens after that will change with my application specifics, for now I just have this combining of 4 frames to send out so I can verify all streams are working.
I think this is a fairly efficient pipeline, if we ignore the COPY and the Render cost, the bulk of time appears to be spent with EGLMap, EGLUnmap and only consume 0.9 ms per camera, which isn’t too bad. I am more so wondering about how I am probing the gstreamer with the pad probe. If there is a better way to setup my pipeline. Perhaps to access at the appsink instead (Some reading I was doing suggested there would be less overhead juggling threads at this level, but it wasn’t clear if this was true or if I was misinterpreting). Or some other approach I should look at?
Okay, so the bulk of what I was struggling with was resolved with AastaLLL’s first post (links to Honey’s script) along with Aastall’s suggestion attempting to only call some of the cuGraphicsEGL on init destroy. For how this was done I put a sample of my code in my other responses (for those that might be interested).