• Jetson AGX Xavier
• DeepStream 6.1
• JetPack Version 5.0.2-b231
• TensorRT Version 8.4.1.5-1
I would like to use the deepstream-test1-usbcam from https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-test1-usbcam
with a custom model .etlt
emotionnet_onnx.etlt (4.3 MB)
can i use the .etlt like this or do I have to transform it before ? how ?
how should i modify the pgie_config file to implement it ?
gst-nvinfer supports " * TAO Encoded Model and Key" Gst-nvinfer — DeepStream 6.1.1 Release documentation
There are lots of samples of deploy TAO “etlt” models with DeepStream NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream (github.com)
You need to be familiar with the model. What is the input layer(s) of the model? Do you know how to handle the output layers of the model?
There is already a DeepStream sample for TAO EmotionNet deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com)
Thank you, i’ve already seen the NVIDIA-AI-IOT but i wanted to use the deepstream emotion app with a usb camera wbut i cannot complete that so i tried to use python script that use deepstream with a usb camera and to modify the model used with an etlt file that i already trained with tao toolkit.
currently i don’t know why but when i launch the deepstream-emotion-app from NVIDIA AI IOT i have this error appearing :
./deepstream-emotion-app: error while loading shared libraries: libnvcv_faciallandmarks.so: cannot open shared object file: No such file or directory
but i can’t find why…
i new to TOA / Deepstream and AI in general so i’m kinda struggling since the begginig of my experiments
Please follow the steps in deepstream_tao_apps/apps/tao_others at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com).
Okey, thank you. Everything should be setup up correctly now. I’ve modified some files to fit with my component and model but I still have an error that comes from the files i’ve modified i guess.
So, I use this command to start my app :
deepstream-app -c deepstream_usbcam_emotion.txt
When it compile i got this error :
WARNING: [TRT]: onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
ERROR: [TRT]: 4: [network.cpp::validate::3004] Error Code 4: Internal Error (input_landmarks:0: for dimension number 1 in profile 0 does not match network definition (got min=3, opt=3, max=3), expected min=opt=max=1).)
ERROR: Build engine failed from config file
Segmentation fault (core dumped)
I also have one other warning but i don’t think it is relevant right now.
I tried to look on other topics of the forum about this error and i saw that i had to modify some dimension to set it same values as the input but i cant find those values.
here is the file that, i think, are used in the process.
deepstream_usbcam_emotion.txt (2.6 KB)
emtionnet_pretrained_etlt_config.txt (842 Bytes)
emotionnet_onnx.etlt (4.3 MB)
label.txt (55 Bytes)
Is there anythink wrong on those file or am i missing somethin else where ?
Maybe it come from this line:
infer-dims=3;544;960
in emtionnet_pretrained_etlt_config.txt. I will try to find the correct values to put here.
How did you generate your " emotionnet_onnx.etlt"? Why don’t you know your model’s input dimensions?
If you are using the EmotionNet model from Emotion Classification — TAO Toolkit 3.22.05 documentation (nvidia.com), it can not be applied to deepstream-app sample.
the emotionnet_onnx.etlt as been generated with the Jupyter Notebook provided by Nvidia with this command :
!mkdir -p $LOCAL_EXPERIMENT_DIR/experiment_dir_final
# Removing a pre-existing copy of the etlt if there has been any.
import os
output_file=os.path.join(os.environ['LOCAL_EXPERIMENT_DIR'],
"experiment_dir_final/emotionnet_onnx.etlt")
if os.path.exists(output_file):
os.system("rm {}".format(output_file))
!tao emotionnet export -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt \
-o $USER_EXPERIMENT_DIR/experiment_dir_final/emotionnet_onnx.etlt \
-t tfonnx \
-k $KEY
I’m sorry i’m new to IA and as i said before i’m struggling with it and i don’t know how i should know it and where i should find it…
Yes my model is based on the emotion Classification… Should i convert it to another format with tao toolikit ?
if it’s not possible, how am i supposed to get a model to use it with deepstream ?
I don’t understand because on the Nvidia documentation :https://docs.nvidia.com/tao/tao-toolkit/text/overview.html#export
They say it’s possible to use etlt model with deepstream.
Is that because it’s classification instead of object detection ?
Surely, the etlt model is supported in deepstream. If you have read our sample deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com), it is using etlt model.
If you have read the DeepStream document and samples, you should know the nvinfer plugin Gst-nvinfer — DeepStream 6.1.1 Release documentation only supports the model which only have one input layer and the layer should be a processed image. The model in Emotion Classification — TAO Toolkit 3.22.05 documentation (nvidia.com) has one input layer which is facial landmarks but not images. So that is why deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com) is provided by us.
Please make sure you know the input layers and output layers of your model.
i’ve seen the sample but your last answer made me doubt about it, sorry.
I’m not really fluent in english but in the introduction nvinfer plugin it says
The plugin also supports the interface for custom functions for parsing outputs of object detectors and initialization of non-image input layers in cases where there is more than one input layer
anyway i wasn’t even aware of the problem of EmotionNet output being facial landmarks instead of images so thank you !
The first and only models i used before this one were Yolov5 and Yolov7 with Pytorch they look so beginner friendly.
What I say is input layer but not output layers. Do you know the input layer(s) of your model “emotionnet_onnx.etlt”?
What will the yolov5 and yolov7 detect?
oh i finally understood, as you said it take human facial landmarks (1 x 136 x 1) from 68 points (coordinates “X,Y”).
so as you say even with the right dimension it still doesn’t work because the entry isn’t an image :
WARN nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1876> [UID = 1]: Could not find output layer 'output_cov/Sigmoid' in engine
ERROR: [TRT]: 3: Cannot find binding of given name: output_bbox/BiasAdd
0:00:12.400054054 14538 0xaaaaed40b800 WARN nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1876> [UID = 1]: Could not find output layer 'output_bbox/BiasAdd' in engine
0:00:12.449271994 14538 0xaaaaed40b800 ERROR nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::preparePreprocess() <nvdsinfer_context_impl.cpp:971> [UID = 1]: RGB/BGR input format specified but network input channels is not 3
ERROR: Infer Context prepare preprocessing resource failed., nvinfer error:NVDSINFER_CONFIG_FAILED
0:00:12.450229128 14538 0xaaaaed40b800 WARN nvinfer gstnvinfer.cpp:846:gst_nvinfer_start:<primary_gie> error: Failed to create NvDsInferContext instance
0:00:12.450330317 14538 0xaaaaed40b800 WARN nvinfer gstnvinfer.cpp:846:gst_nvinfer_start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-6.1/samples/configs/tao_pretrained_models/emtionnet_pretrained_etlt_config.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
I don’t know if it would work with Yolo on this dataset (with the problem of input not being an image probably not) but I don’t know I feel like the documentation and the Yolo tool seems less sprawling than the TAO and Deepstream app, it seems better centralized and with less special cases.
I’m sorry i know that is isn’t the initial subject of this topic but I just looked at deepstream_tao_apps from NVIDIA AI IOT and i already tried to use it but i want to use my camera instead of the URI of an image, my camera device is located at /dev/video0 but i haven’t found how to use that.
There is DeepStream Yolo samples: NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream (github.com) and NVIDIA-AI-IOT/yolo_deepstream: yolo model qat and deploy with deepstream&tensorrt (github.com)
And also some 3rd party yolov-x deploying with DeepStream samples: DeepStream SDK FAQ - #24 by mchi
You’ve already got the sample for usbcam deepstream_python_apps/apps/deepstream-test1-usbcam at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub. The corresponding c/c++ version can use the same “v4l2src->capsfilter->videoconvert->nvvideoconvert->” to input to nvstreammux. v4l2src (gstreamer.freedesktop.org)
This is exactly the initial reason of this topic, i used this sample, it correctly use the webcam but recognize cars and person and i wanted it to recognize emotion. That’s why i tried to ad the etlt custom model.
The correct way is to port the "“v4l2src->capsfilter->videoconvert->nvvideoconvert->” code to deepstream_tao_apps/apps/tao_others/deepstream-emotion-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com).
What on earth is blocking you? The two samples are completely open source, you can port as you like.
Finally i only had to use this command : ./deepstream-emotion-app 3 ../../../configs/facial_tao/sample_faciallandmarks_config.txt v4l2:///dev/video0 ./landmarks
the ‘3’ make a pipeline with display sink and ‘v4l2’ replace URI for streaming.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.