Fusion of RGB + Thermal

Please provide complete information as applicable to your setup.

**• Hardware Platform - Orin NX
**• DeepStream Version- 6.2
**• JetPack Version (valid for Jetson only)- 5.1.0
I want to take 2 input streams, RGB and Thermal. I want to concatenate the feeds to make a single 4 channels(RGBT) output which I will then send to nvinfer.
How can I merge both the input streams?

Can you tell us what is your model’s inputs?

How will you collect the thermal data? Does the thermal data fluctuate frame by frame?

My model is designed to process images with four channels. I use two types of input data: RGB stream, which has three channels, and thermal stream, which has a single channel. I need to combine these two data streams frame by frame to create four-channel images, which I will then feed into nvinfer for object detection. How can I concatenate the two streams?

For dataset collection:
RGB Resolution: 3840x2160 FPS-20 FPS
Thermal Resolution: 640x512 FPS-9 FPS

I am not sure what you mean by “Does the thermal data fluctuate frame by frame?”.

You have the RGB stream with 20 FPS while the thermal data is 9 FPS. How do the two types of data combined together? For every RGB image frame, there should be a corresponding thermal data for it, right?

Can you describe the model input layers in detail? How many input layers and the data type and dimension of each layer

Yes, for every RGB image there is a thermal image. I am taking the streams simultaneously. I am clear with the model inferencing part but I do not know how to combine both the data to send it to my model which accepts 4 channels image. So, my question is, I use two types of input data: RGB stream, which has three channels, and thermal stream, which has a single channel. I need to combine these two data streams frame by frame to create four-channel images. How can I combine them

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

DeepStream is based on GStreamer which is mainly for traditional standard multimedia (video, audio, images). The RGB image such as jpg files, bmp files, or RGB frames from camera devices(CSI, DVB,…) can be treated by DeepStream directly. But the thermal data is not any standard media data and we don’t know how did you get the thermal data( from files? from some type of devices?..), then we don’t know how to suggest you to adapt to the DeepStream APIs.

Suppose the RGB images are local RGB data files while the thermal data is stored in some kind of binary files, suppose you model has only one input layer for the 4 channel data, a relatively easy way is to add pad probe function after the nvstreammux in your video(images) pipeline, you can read the thermal data and put the data into the corresponding frame metadata’s user metadata. Then you need to customize a nvpreprocess library to get the frame data and thermal data and combine them into new tensor data with CUDA. Then the tensor data can be accepted by the gst-nvinfer with “input-tensor-meta” parameter enabled. Gst-nvinfer — DeepStream documentation 6.4 documentation

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.