Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
• DeepStream Version: 7.1
• TensorRT Version: 10.13
• NVIDIA GPU Driver Version (valid for GPU only): Ada Lovelace
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
I am trying to build a deepstream app to process a specific spatio-temporal model we have built in our lab. The model takes in two input tensors:
- Custom tensor sequence of length 32 exactly as is used in deepstream-3d-action-recognition for n_sources of RTSP streams [n_sources,3,32,256,256]
- A custom tensor that takes the output of a pgie people net model which prepares the tensor based on the output of resnet34_peoplenet_int8.onnx and makes a tensor in the format [total_num_bbox_for_the_batch, [source_id_it_belongs_to, x1, y1, x2, y2]] where x1, y1, x2, y2 represent the bboxes for each person coming from pgie1 (object detection of people net). Another note is that that these are to be taken for only the middle/key frame of the sequence, i.e. frame 16/32, although I have not even attempted to retrieve this frame yet.
Here is a screenshot of the input and output tensor dimensions in netron.app for further details.
I have gotten the first tensor to work with a version of the model without the bboxes, (similar to recognition net i assume but trained by us). I cannot seem to engineer the custom tensor in nvdspreprocess_lib without it failing to load the library, no matter what I try. Here is the code I tried so far in my nvdspreprocess_lib.cpp function:
NvDsPreProcessStatus
PrepareBboxTensor(CustomCtx *ctx, NvDsPreProcessBatch *batch, NvDsPreProcessCustomBuf *&buf,
CustomTensorParams &tensorParam, NvDsPreProcessAcquirer \*acquirer)
{
NvDsPreProcessStatus status = NVDSPREPROCESS_TENSOR_NOT_READY;
// Acquire buffer from tensor pool
buf = acquirer->acquire();
float \*pDst = (float\*)buf->memory_ptr;
int units = batch->units.size();
int total_objects = 0;
// Iterate over all frames in the batch
guint batch_id = 0;
for (int i = 0; i < units; ++i) {
GstBuffer \*inbuf = (GstBuffer \*)batch->inbuf;
NvDsBatchMeta \*batch_meta = gst_buffer_get_nvds_batch_meta(inbuf);
if (!batch_meta) continue;
for (NvDsMetaList \*l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next, batch_id++) {
NvDsFrameMeta \*frame_meta = (NvDsFrameMeta \*)(l_frame->data);
for (NvDsMetaList \*l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) {
NvDsObjectMeta \*obj_meta = (NvDsObjectMeta \*)l_obj->data;
// Only PGIE objects (optionally filter by class_id if needed)
float x1 = obj_meta->rect_params.left;
float y1 = obj_meta->rect_params.top;
float x2 = obj_meta->rect_params.left + obj_meta->rect_params.width;
float y2 = obj_meta->rect_params.top + obj_meta->rect_params.height;
// Write \[batch_id, x1, y1, x2, y2\] to tensor
pDst\[0\] = (float)batch_id;
pDst\[1\] = x1;
pDst\[2\] = y1;
pDst\[3\] = x2;
pDst\[4\] = y2;
pDst += 5;
total_objects++;
}
}
}
status = NVDSPREPROCESS_SUCCESS;
return status;
}
CustomCtx* initLib()
{
CustomCtx\* ctx = new CustomCtx();
// Set tensor parameters for \[n_objects, 5\] (batch_id, x1, y1, x2, y2)
ctx->tensor_num_dims = 2;
ctx->tensor_shape\[0\] = \_MAX_OBJECT_NUM\_; // Maximum number of objects per batch (define as needed)
ctx->tensor_shape\[1\] = 5; // \[batch_id, x1, y1, x2, y2\]
ctx->tensor_dtype = NVDSPREPROCESS_DT_FLOAT32;
// Calculate total elements and bytes
ctx->tensor_num_elements = ctx->tensor_shape\[0\] \* ctx->tensor_shape\[1\];
ctx->tensor_bytes = ctx->tensor_num_elements \* sizeof(float);
ctx->cpu_tensor = new float\[ctx->tensor_num_elements\];
memset(ctx->cpu_tensor, 0, ctx->tensor_bytes);
return ctx;
}
Here is the corresponding config file:
config_preprocess_stdet.txt (2.0 KB)
I tried to rebuild this into the deepstream-pose-classification app as this is the most similar to mine, just to run the pipeline as follows:
gst_bin_add_many(GST_BIN(pipeline), pgie, tracker, preprocess1, nvtile, nvvidconv, nvosd, sink, nvdslogger, NULL);
The end intended pipeline will be: gst_bin_add_many(GST_BIN(pipeline), preprocess0, pgie, tracker, preprocess1, sgie, nvmsgconv, nvmsgbroker, fakesink, NULL); Here preprocess0 should construct the custom tensor for the sequence (exactly as in deepstream-3d-action-recognition) and preprocess1 the second bbox tensor, which is where it is failing.
I have attached and config files for reference. Every time I run this configuration, I receive the error:
Running…
ERROR from element preprocess-plugin: Could not open custom library
Error details: gstnvdspreprocess.cpp(524): gst_nvdspreprocess_start (): /GstPipeline:deepstream_pose_classfication_app/GstNvDsPreProcess:preprocess-plugin
Returned, stopping playback
Could anyone point me in the right direction of how to build the custom tensor for this? I have tried looking at all of the examples, but I don’t want to modify the ROI itself, I just want the bboxes from the centre image and their batch ids in one tensor.