Hi,
Orin NX 5.1.2 DS 6.3
I’m using an Action Recognition model trained in TAO. not getting the same results as in TAO.
I want to validate the preprocess images sent to the Action recognition.
getting error:
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:4339: => Surface type not supported for transformation NVBUF_MEM_CUDA_UNIFIED
compiled makefile WITH_OPENCV:=1
and set define DUMP_ROIS
Anything else I missed?
Thanks,
Yaniv
which sample are you referring to or testing? did you add custom code or configurations?
Hi @fanzh ,
I use custom code. but for simplicity, I also tried now the original deepstream-3d-action-recognition sample with the same results (on OrinAGX):
./deepstream-3d-action-recognition -c deepstream_action_recognition_config.txt
num-sources = 4
Now playing: file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_walk.mov, file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_ride_bike.mov, file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_run.mov, file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_push.mov,
0:00:00.118756256 3059496 0xaaab00917a70 WARN nvinfer gstnvinfer.cpp:887:gst_nvinfer_start:<primary-nvinference-engine> warning: NvInfer output-tensor-meta is enabled but init_params auto increase memory (auto-inc-mem) is disabled. The bufferpool will not be automatically resized.
0:00:02.132655968 3059496 0xaaab00917a70 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.3/sources/apps/sample_apps/deepstream-3d-action-recognition/resnet18_3d_rgb_hmdb5_32.etlt_b4_gpu0_fp16.engine
INFO: [FullDims Engine Info]: layers num: 2
0 INPUT kFLOAT input_rgb 3x32x224x224 min: 1x3x32x224x224 opt: 4x3x32x224x224 Max: 4x3x32x224x224
1 OUTPUT kFLOAT fc_pred 5 min: 0 opt: 0 Max: 0
0:00:02.274426944 3059496 0xaaab00917a70 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.3/sources/apps/sample_apps/deepstream-3d-action-recognition/resnet18_3d_rgb_hmdb5_32.etlt_b4_gpu0_fp16.engine
0:00:02.294572064 3059496 0xaaab00917a70 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:config_infer_primary_3d_action.txt sucessfully
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:494, [INFO: CUSTOM_LIB] 3D custom sequence network info(NCSHW), [N: 4, C: 3, S: 32, H: 224, W:224]
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:522, [INFO: CUSTOM_LIB] Sequence preprocess buffer manager initialized with stride: 1, subsample: 0
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:526, [INFO: CUSTOM_LIB] SequenceImagePreprocess initialized successfully
Using user provided processing height = 224 and processing width = 224
Decodebin child added: source
Decodebin child added: decodebin0
Decodebin child added: source
Decodebin child added: decodebin1
Decodebin child added: source
Decodebin child added: decodebin2
Decodebin child added: source
Decodebin child added: decodebin3
Running...
WARNING from element primary-nvinference-engine: NvInfer output-tensor-meta is enabled but init_params auto increase memory (auto-inc-mem) is disabled. The bufferpool will not be automatically resized.
Warning: NvInfer output-tensor-meta is enabled but init_params auto increase memory (auto-inc-mem) is disabled. The bufferpool will not be automatically resized.
Decodebin child added: qtdemux0
Decodebin child added: qtdemux1
Decodebin child added: qtdemux2
Decodebin child added: qtdemux3
Decodebin child added: multiqueue0
Decodebin child added: multiqueue1
Decodebin child added: multiqueue2
Decodebin child added: multiqueue3
Decodebin child added: h264parse0
Decodebin child added: h264parse1
Decodebin child added: h264parse2
Decodebin child added: capsfilter1
Decodebin child added: capsfilter0
Decodebin child added: capsfilter2
Decodebin child added: h264parse3
Decodebin child added: capsfilter3
Decodebin child added: aacparse0
Decodebin child added: aacparse1
Decodebin child added: aacparse2
Decodebin child added: aacparse3
Decodebin child added: faad2
Decodebin child added: faad0
Decodebin child added: faad3
Decodebin child added: faad1
Decodebin child added: nvv4l2decoder0
Decodebin child added: nvv4l2decoder3
Decodebin child added: nvv4l2decoder1
Decodebin child added: nvv4l2decoder2
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:4339: => Surface type not supported for transformation NVBUF_MEM_CUDA_UNIFIED
0:00:02.785967680 3059496 0xaaab0009eaa0 ERROR nvdspreprocess gstnvdspreprocess.cpp:1156:batch_transformation: NvBufSurfTransform failed with error -1
ERROR from element preprocess-plugin: Transformation from plugin failed with error
Error details: gstnvdspreprocess.cpp(1282): group_transformation (): /GstPipeline:preprocess-test-pipeline/GstNvDsPreProcess:preprocess-plugin
Returned, stopping playback
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:586, [INFO: CUSTOM_LIB] SequenceImagePreprocess is deinitializing
Deleting pipeline
when compiling preprocess with WITH_OPENCV:=0 and without DUMP_ROI runs ok.
thanks for the sharing! from the error “Surface type not supported for transformation NVBUF_MEM_CUDA_UNIFIED”, it is related with memorytype. can you try
allocator_info.memory_type = NVBUF_MEM_DEFAULT; instead of allocator_info.memory_type = NVBUF_MEM_CUDA_UNIFIED in opt\nvidia\deepstream\deepstream\sources\gst-plugins\gst-nvdspreprocess\gstnvdspreprocess.cpp? especially you need to rebuild and replace libnvdsgst_preprocess.so.
OK. now its working and saves the rois butsome of them are total black and other with bad color format :
how ca I fix that?
./deepstream-3d-action-recognition -c deepstream_action_recognition_config.txt
num-sources = 4
Now playing: file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_walk.mov, file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_ride_bike.mov, file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_run.mov, file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_push.mov,
0:00:00.119459136 3082124 0xaaab26061a70 WARN nvinfer gstnvinfer.cpp:887:gst_nvinfer_start:<primary-nvinference-engine> warning: NvInfer output-tensor-meta is enabled but init_params auto increase memory (auto-inc-mem) is disabled. The bufferpool will not be automatically resized.
0:00:02.158322528 3082124 0xaaab26061a70 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.3/sources/apps/sample_apps/deepstream-3d-action-recognition/resnet18_3d_rgb_hmdb5_32.etlt_b4_gpu0_fp16.engine
INFO: [FullDims Engine Info]: layers num: 2
0 INPUT kFLOAT input_rgb 3x32x224x224 min: 1x3x32x224x224 opt: 4x3x32x224x224 Max: 4x3x32x224x224
1 OUTPUT kFLOAT fc_pred 5 min: 0 opt: 0 Max: 0
0:00:02.301358624 3082124 0xaaab26061a70 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.3/sources/apps/sample_apps/deepstream-3d-action-recognition/resnet18_3d_rgb_hmdb5_32.etlt_b4_gpu0_fp16.engine
0:00:02.320316352 3082124 0xaaab26061a70 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:config_infer_primary_3d_action.txt sucessfully
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:494, [INFO: CUSTOM_LIB] 3D custom sequence network info(NCSHW), [N: 4, C: 3, S: 32, H: 224, W:224]
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:522, [INFO: CUSTOM_LIB] Sequence preprocess buffer manager initialized with stride: 1, subsample: 0
/dvs/git/dirty/git-master_linux/deepstream/sdk/apps/deepstream/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess/sequence_image_process.cpp:526, [INFO: CUSTOM_LIB] SequenceImagePreprocess initialized successfully
Using user provided processing height = 224 and processing width = 224
Decodebin child added: source
Decodebin child added: decodebin0
Decodebin child added: source
Decodebin child added: decodebin1
Decodebin child added: source
Decodebin child added: decodebin2
Decodebin child added: source
Decodebin child added: decodebin3
Running...
WARNING from element primary-nvinference-engine: NvInfer output-tensor-meta is enabled but init_params auto increase memory (auto-inc-mem) is disabled. The bufferpool will not be automatically resized.
Warning: NvInfer output-tensor-meta is enabled but init_params auto increase memory (auto-inc-mem) is disabled. The bufferpool will not be automatically resized.
Decodebin child added: qtdemux1
Decodebin child added: qtdemux0
Decodebin child added: qtdemux2
Decodebin child added: qtdemux3
Decodebin child added: multiqueue3
Decodebin child added: multiqueue2
Decodebin child added: multiqueue0
Decodebin child added: multiqueue1
Decodebin child added: h264parse0
Decodebin child added: h264parse1
Decodebin child added: h264parse2
Decodebin child added: h264parse3
Decodebin child added: capsfilter0
Decodebin child added: capsfilter1
Decodebin child added: capsfilter2
Decodebin child added: capsfilter3
Decodebin child added: aacparse0
Decodebin child added: aacparse2
Decodebin child added: aacparse1
Decodebin child added: aacparse3
Decodebin child added: faad0
Decodebin child added: faad2
Decodebin child added: faad1
Decodebin child added: faad3
Decodebin child added: nvv4l2decoder0
Decodebin child added: nvv4l2decoder1
Decodebin child added: nvv4l2decoder3
Decodebin child added: nvv4l2decoder2
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
In cb_newpad
nvbufsurface: Wrong buffer index (0)
nvbufsurface: Wrong buffer index (1)
nvbufsurface: Wrong buffer index (2)
nvbufsurface: Wrong buffer index (3)
nvbufsurface: Wrong buffer index (0)
nvbufsurface: Wrong buffer index (1)
nvbufsurface: Wrong buffer index (2)
nvbufsurface: Wrong buffer index (3)
nvbufsurface: Wrong buffer index (0)
nvbufsurface: Wrong buffer index (1)
nvbufsurface: Wrong buffer index (2)
nvbufsurface: Wrong buffer index (3)
nvbufsurface: Wrong buffer index (0)
nvbufsurface: Wrong buffer index (1)
Hi @fanzh ,
changing dump_rois:
in_mat =
cv::Mat (nvdspreprocess->processing_height, nvdspreprocess->processing_width,
CV_8UC4, outsurf->surfaceList[i].mappedAddr.addr[0],
outsurf->surfaceList[i].pitch);
from CV_8UC3 to CV_8UC4 fixed the color format.
But there are still issues with mixing with src_id. for example, all of these images are src_id 0 but actually show different sources:
also, some images were saved from different parts of different sources or the same sources from different frame numbers:
this probably caused the model to give wrong inferences.
what am I missing?
Thanks
are there any wrong inferences? because they are batch data, all of these images from different sources.
Thanks for the sharing! I can reproduce this issue on Orin with DS6.4. some images were saved from different parts of different sources or the same sources from different frame numbers. we are investigating.
Thanks @fanzh
sorry for the late reply. here is the reason for sources mixing issue.
deepstream-3d-action-recognition did not set custom-input-transformation-function in configuration. the default transformation of nvpreprocess plugin is using NvBufSurfTransformAsync to do buffertransform. NvBufSurfTransformAsync is async function. we need to dump data after NvBufSurfTransformSyncObjWait ending.
here is a workaround.
gboolean ret = dump_rois(nvdspreprocess, batch.get(), memory->surf);
in gst_nvdspreprocess_on_frame.
2. after
for(auto sync_object: batch->sync_objects){
NvBufSurfTransformSyncObjWait(sync_object, -1);
NvBufSurfTransformSyncObjDestroy(&sync_object);
}
in gst_nvdspreprocess_output_loop, add the following code:
dump_rois(nvdspreprocess, batch.get(), &nvdspreprocess->batch_outsurf);
no, from the analysis above, the data is correct after NvBufSurfTransformSyncObjWait.
Hi @fanzh,
Now its ok.
Thanks!
Yaniv
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.