Jetson.utils load numpy array into GPU memory

Hey Jetson-folks!
I’m currently using third-party camera controler to grab a frame from a PoE camera which is transformed into a 3 channel numpy array. My current process is to save this array using cv2.imwrite to the local disk in order to use jetson.utils.loadImageRGBA for loading this image from the disk. That is extremely inefficient but I wasn’t able to figure out how to load the image array more directly into the GPU memory. Any ideas and hints?

hello klat,

please check Jetson Linux Multimedia API Reference, there’s Multimedia API Sample Applications to demonstrate how to use the Multimedia API and other libraries.
you may also refer to 12_camera_v4l2_cuda to capture images and share the image stream with CUDA engines.
thanks

Hi Jerry,
Thx for the reference! I’ve been able to load the image array with the cuda-from-numpy.py module in Jetson utils but had to transform the RGB image to an RGBA first for the right shape. Just out of curiosity, why was the RGBA shape chosen for cuda over RGB? Another question: How would I handle greyscale images? I couldn’t find a module or mentioning on how to load greyscale images for a network that has been trained using DIGITS with greyscale data setting. Thx for your support!

hello klat,

it’s by default using RGBA color formats for processing.

please refer to Canny edge detector samples, which expects a single channel 8-bit grayscale input image.

BTW,
please also check CUDA Toolkit Documentation for more details.
thanks

Hi Jerry,

Thx for the Canny edge hint! I’ll check it out.

I experience this strage problem with loading images into cuda:
When loading a RGBA array directly into cuda using jetson.utils.cudaFromNumpy() I get very different inference results than saving the array with cv2.imwrite to disk and then loading it with jetson.utils.loadImageRGBA(). While the inference works well with the disk saving workaround, it doesn’t work at all with the direct injection using .cudaFromNumpy().

Further investigations I did:
I checked whether the image is broken somehow when loaded directly into cuda. Used jetson.utils.saveImageRGBA() to save it and to compare it to the version save with cv2.imwrite(). The image look the same and the resolution is also the same. But the image saves with .saveImageRGBA() is about twice the size and the system performance drops significantly when saving the image with this method.
Any ideas why inference results vary so heavily depending on the way the image is injected to cuda and why is saving with jetson.utils.saveImageRGBA() double image size and low performance?

Thanks for you support! I’m happy to share code and other resources. Just let me know.

The imageNet/detectNet/segNet classes from jetson-inference expect an RGBA float4 image, with pixel values between 0.0-255.0. So you will want to make sure your numpy array has 4 channels and those pixel ranges before converting it with cudaFromNumpy().

There is a quality parameter to the base saveImageRGBA(), which effects the compressed size. I see that I forgot to implement this quality parameter in the Python bindings, sorry about that. As a workaround, you could change the default quality on this line:

And then run:

cd jetson-inference/build
make
sudo make install

Terrific dusty_nv! Thanks for adding this to the Python bindigs :)

Is there sample code for loading grayscale images into cuda using jetson.utils.cudaFromNumpy or another method? Color is irrelevant for my use case and I want to do inferencing as efficient as possible with processing single channel images. I used DIGITS to train a caffe model with grayscale images but still can’t figure out how to load single channel images into cuda on the jetson. All reference examples are using RGBA. I assume inferencing single channel images is possible with jetson, is it?

Yes, if your network was trained for grayscale images and the pre-processing code supports it, then no problem. There isn’t anything specific to Jetson that limits it to only RGBA, ect. My library is just setup for that since it is most common - you would need to change the pre-processing code inside imageNet.cpp/ect.

Instead you might want to look at one of the TensorRT mnist python examples, since I believe mnist uses grayscale. You can find them under /usr/src/tensorrt/samples/python