added cudaFromGstSample to jetson-inference python bindings

aconchillo · August 8, 2019, 8:14am

Hi!

I tried to get jetson-inference to analyze a video file (H.264 encoded) using python and GStreamer but I found some road blocks like what to do with the GstSample obtained from GStreamer and how to copy it (since it’s NVMM memory) to CUDA so it can then be used by jetson-inference.

After a couple of nights I ended up being able to load and analyze a video file using jetson-inference. I created a new cudaFromGstSample that can be used from directly from python. This is not optimal since a memcpy needs to happen, but it was good enough for my use case.

I’m not 100% sure that what I did was the right thing, so I’d love to hear any comments.

Here’s the pull request for the jetson-utils repo with more details:

github.com/dusty-nv/jetson-utils

python: add initial cudaFromGstSample support

dusty-nv:master ← aconchillo:add-pygst-support

opened 07:58AM - 08 Aug 19 UTC

aconchillo

+641 -7

This patch adds initial bindings support for GStreamer. It currently only provi…des a function `cudaFromGstSample`. `cudaFromGstSample` expects a GstSample as a single argument. The GstSample needs to be in NV12 format and stored in host memory (this means there's no zero-copy). The function will then copy the frame to CUDA memory and convert it to RGBA (float4) directly in the GPU. A usage example could be: def on_new_sample(appsink): sample = appsink.emit('pull-sample') img, width, height = jetson.utils.cudaFromGstSample(sample) # classify the image class_idx, confidence = net.Classify(img, width, height) When using GStreamer for, let's say, decoding a video encoded in H.264, it is very likely that OpenMAX GStreamer plugins are used (i.e. `omxh264dec`). This will create a `video/x-raw(memory:NVMM)` frame that is mapped into NVMM memory (i.e. DMA buffer). Unfortunately, it's not straight forward to convert NVMM memory to CUDA, that's why `cudaFromGstSample` expects the frame to be stored in the host memory. To copy an NVMM mapped frame to host memory, your pipeline needs to include the `nvvidconv` plugin which will convert an NVMM mapped frame to a regular `video/x-raw` frame.

Aleix

AastaLLL · August 12, 2019, 6:52am

Hi,

Thanks for your contribute.
I will forward this information to Dusty : )