Fit custom model with different inputs into sample pipeline Deepstream 3D Action Recognition App


I have custom model which I want to fit into DeepStream 3D Action Recognition App

In DeepStream sample application model has input shape as NCDHW(NCSHW). e.g (4,3,32,224,224). Lets call this Tensor 1.

But my model takes 3 input tensors. One of them is exactly same as above.

The other two tensors are like Tensor 2. (31, 3, 224, 224) and Tensor 3. (31, 3, 224, 224) of type (S,C,H,W)

Both above tensors have same data as Tensor 1. Only difference is that Tensor 2 does not have last frame (Check S is 31 instead of 32) and similarly Tensor 3 does not have first frame.

I have explored existing application and found that whenever 32 frames are processed, batched and then sent to gst-nvinfer for inferencing as per this DeepStream 3D Action Recognition App — DeepStream 6.0 Release documentation ( 3D Action Recognition App

My question is that is it possible to fit custom model into DeepStream 3D Action Recognition App?

If yes,
From what I have understood, to fit my custom model into existing application I need to make changes into:

  1. custom_sequence_preprocess (sub directory of app) → sequence_image_process.cpp
  2. gst-nvinfer plugin

Please correct me if any other plugins or module needs to be updated to fit custom model into this app.
Would request you to please help me with the high level flow.

Sorry for delay!

Will check and get back to you!

Not a problem. Thanks.

Hi @dusty_nv Can you please look at this?

Thanks in advance.

Sorry @miteshp.patel, wish I could help but this isn’t my area of expertise. @mchi will follow-up.

Hi, Gentle reminder.

Hey @mchi We would really appreciate your suggestion wrt above approach. Awaiting your response for the same.