Error when running Emotion Classification Sample in TAO Computer Vision Inference Pipeline

Hi everybody,

I am running the Emotion Classification Sample in TAO Computer Vision Inference Pipeline in a Docker-Container as described here: https://docs.nvidia.com/tao/tao-toolkit/text/tao_cv_inf_pipeline/running_samples.html#running-the-emotion-classification-sample
My hardware is a Jetson AGX Xavier with Jetpack 4.5.
The demo program is running, but occasionaly crashes with the following error:

[Jarvis] [E] faciallandmarks.postprocessor/postprocessor: Mismatch between landmarks batch (7) and face detections (6)terminate called after throwing an instance of ‘std::runtime_error*’
#01 /workspace/tlt_cv-pkg/private/api/libjrt_tlt_cv_api.so(+0xa1290) [0x7fa4c0e290]
#02 google_breakpad::ExceptionHandler::GenerateDump(google_breakpad::ExceptionHandler::CrashContext*) /workspace/tlt_cv-pkg/private/api/…/…/_solib_k8/_U_S_Sprivate_Sapi_C_Ujrt_Ushared_Ulink___Uprivate_Sapi/libjrt.so(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3e8) [0x7fa2344e10]
#03 google_breakpad::ExceptionHandler::SignalHandler(int, siginfo_t*, void*) /workspace/tlt_cv-pkg/private/api/…/…/_solib_k8/_U_S_Sprivate_Sapi_C_Ujrt_Ushared_Ulink___Uprivate_Sapi/libjrt.so(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xe0) [0x7fa2345118]
#04 linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7fa53c66b0]
#05 /lib/aarch64-linux-gnu/libc.so.6(raise+0xb0) [0x7fa488e4f8]

The error occurs especially when there are multiple faces present in an input image. I do not get the error when testing on an x64-machine.
Maybe somebody can help me with this issue. Thanks!

P.S.: The same issue occured when running the emotion demo in TLT CV Inference Pipeline v0.2-ga.

To narrow down, can you try to run again with the official released model from the NVIDIA NGC ?

I am already running the officially released model (this is pulled inside the Docker-container and provided via Triton Inference server). These are the steps I performed in more detail:

./samples/tao_cv/demo_emotion/emotion samples/tao_cv/demo_emotion/demo.conf

→ This runs on a camera stream and shows emotions for each detected face. But occasionally the program cashes and the above described error occurs.

In addition, the face detection sample application is running without problems. To me, there seems to be a problem when working with this defintion in the code:

const auto pipelineType = nja::PipelineType::EMOTION;
nja::TAOCVAPI cvAPI(pipelineType);

In summary, I am only trying the released scripts/containers from NGC without modification to the code.

Thanks for the info. If possible, could you help run on a video file instead of a camera stream ?

The same problem occurs with a video file (independent of resolution and frame rate). An example video is attached, which results in the error:

Thanks for the info. We will check further. More, did you ever try less faces present in an input image?

Yes, I also tried with less faces, but it takes longer for the program to crash. The more faces, the easier to provoke the error in the program. Also, when running my posted example video, the error does NOT always occur at the same time. So the error message

Mismatch between landmarks batch (7) and face detections (6)terminate called after
throwing an instance of ‘std::runtime_error*’

differs in the number of detected faces and landmarks batch. But they all have in common that the numbers do not match.
As said before, the error does not occur when defining a pipeline only for face detection with

const auto pipelineType = nja::PipelineType::FACE_DETECT;

Please use below way to run inference instead.
See deepstream_tao_apps/apps/tao_others at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub

You can apply for access for DS 6.0 EA from https://developer.nvidia.com/deepstream-sdk-6.0-members-page
Then, run this DS 6.0 EA emotion app for L4T Jetpack 4.5.1

In container: nvcr.io/nvdeepstream/deepstream6_ea/deepstream-l4t:6.0-ea-21.06-samples

apt-get install gstreamer1.0-tools libglib2.0-dev
libgstrtspserver-1.0-0 libjansson4 libjson-glib-1.0-0
glib2.0 gstreamer1.0-libav gstreamer1.0-plugins.*
libgstreamer1.0.* libgstreamer-plugins-.1.0.
libglu1-mesa-dev librdkafka-dev

./deepstream-emotion-app 1 file:///tmp/test3.mp4 ./emotion

Thanks for the reply. So, is it currently not possible to use with the TAO Toolkit CV Inference Pipeline? Can you reproduce my error?

It is possible to use with the TAO Toolkit CV Inference Pipeline. For this random issue of emotion sample, we can reproduce but do not find a quick workaround. Also, the inference pipeline is planned to be deprecated in next release. So, internal team just mentioned above deepstream inference.

Ok, thank you for your feedback!