Trouble Integrating Emotion Detection Model with DeepStream 7.1 and Clara Guardian – Engine Build and Pipeline Errors

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Dear NVIDIA Support Team,

I am trying to integrate an emotion detection ONNX model into DeepStream 7.1, intended for use with the Clara Guardian platform. However, I am facing multiple critical issues during this process:

  1. TensorRT Engine Generation Errors:
  • During engine generation using trtexec, I encountered dimension mismatch errors (profile has min=3, opt=3, max=3 but tensor has 160) even after adjusting input shapes manually.
  • Required disabling --explicitBatch and correcting input dimensions to successfully build the .engine file.
  1. DeepStream Pipeline Failure:
  • After setting up the DeepStream configuration (config_infer_primary_emotionnet.txt), running deepstream-app fails with the error:
    "Failed to set pipeline to PAUSED"
    "Output width not set" (from src_bin_muxer).
  • It seems to be related to incomplete or incompatible configuration parameters, although standard setup guidelines were followed (setting input-dims, output-blob-names, and linking the generated .engine file).
  1. ONNX Model Format Concerns:
  • The model input (input_5) has a dynamic dimension for batch size (unk__1418) and static dimensions (160, 160, 3). Adjusting the input-dims in DeepStream didn’t fully resolve the integration problems.
  1. Impact:
  • Due to these issues, real-time emotion detection integration with Clara Guardian is blocked, and project timelines are affected.

Request:

  • Need support on properly preparing ONNX models and building TensorRT engines compatible with DeepStream 7.1.
  • Need guidance on correct DeepStream configuration for custom emotion detection models with dynamic batch size ONNX inputs.
  • Any recommendations if Triton Inference Server should be preferred over DeepStream for emotion detection with Clara Guardian integration.

I can provide model files, configuration files, logs, and detailed reproduction steps if needed.

Thank you for your assistance.

3

Model conversion using trtexec is usually unnecessary. nvinfer can directly convert onnx to engine file. In addition, since deepstream uses a memory pool with fixed width/height, it does not support dynamic shape.

Missing width/height in nvstreammux configuration group

[streammux]
## Set muxer output width and height
width=1920
height=1080

Please check network-input-order (Integer 0:NCHW 1:NHWC) in the nvinfer configuration file.

As I described above, due to memory usage considerations, deepstream does not support dynamic shapes, but can support dynamic batch-size.
Please provide pipeline/configuration file/onnx model to analyze the problem.

4

Hi NVIDIA Team,

Thank you for the initial guidance. I wanted to follow up with more details on what I’m trying to build, what I’ve achieved so far, and where I still need help.

My Goal

I’m setting up a real-time Emotion Detection pipeline using DeepStream 7.1 as part of a Clara Guardian E-Sitter monitoring system. The aim is to process live or recorded video feeds in a hospital setting and classify patient emotions (e.g., happy, sad, neutral) for behavioral monitoring and safety.

What I’ve Achieved So Far

I successfully launched the DeepStream app using the following command:

deepstream-app -c deepstream_app_config_test.txt

** WARN: <parse_app:328>: Unknown key ‘osd-enable’ for group [application]
** WARN: <parse_app:328>: Unknown key ‘display’ for group [application]
** INFO: <bus_callback:291>: Pipeline ready
** INFO: <bus_callback:277>: Pipeline running


So the pipeline runs without crashing and loads the EmotionNet engine file correctly.

---

My Current Config File (`deepstream_app_config_test.txt`)**

[application]
enable-perf-measurement=0

[source0]
type=1
uri=file:/opt/nvidia/deepstream/deepstream-7.1/deepstream_tao_apps_copy/configs/app/falling.mp4
num-sources=1

[streammux]
batch-size=1
width=160
height=160
enable-padding=0

[primary-gie]
model-engine-file=/opt/nvidia/deepstream/deepstream-7.1/deepstream_tao_apps_copy/models/emotionnet/engine_file/emotionnet.etlt_b1_gpu0_int8.engine
batch-size=1
infer-precision=INT8
gpu-id=0
network-mode=2
use-triton=0

[tracker]
enable=0


---

Clarifications

* The warnings about `osd-enable` and `display` came from keys placed incorrectly under `[application]`. I’ve now removed them, and the warning no longer appears.
* My EmotionNet model is INT8, converted from `.etlt` to engine format.
* I’m not using a tracker currently, and inference runs fine.

---

What I Need Help With

1. **Face Cropping → Emotion Model**:
How can I feed YOLO-detected face crops into the emotion classifier within DeepStream (chained inference setup)?
2. **Headless Logging**:
I want to **log emotion predictions using Python bindings (`pyds`)** without rendering/displaying the video. Any recommended sample or boilerplate to follow?
3. **Model Input Format**:
My model has input shape (1, 3, 224, 224). Does this mean I should set `network-input-order=0` (NCHW)? Are there other config flags required?
4. **Integration with Clara Guardian**:
Are there any special considerations or best practices for plugging this pipeline into the Clara Guardian framework?

---

Thanks again for your help. I'm happy to share the ONNX model, engine, or full logs if that assists in debugging or improvement suggestions.

Best,
**Naga Durga Sai Addala**
5

From your description, it seems that you should build the following pipeline.
The emotionnet configuration should be wrong. If emotionnet is a classifier, this will not work at all.

If emotionnet cannot directly process the output object（such as face) of pgie, you can add nvdspreprocess upstream of sgie

nvstreamux --> pgie(yolo detector output face object) --> sgie(emotionnet classifier input face object) -->....

test2.py is an example of a detector + two classifiers, which may be more suitable for your scenario.

Add a probe function to the sgie src pad, and then access the object meta to get the output of the classifier.
For test2.py, It is osd_sink_pad_buffer_probe

This only deals with deepstream related topics. I don’t know much about Clara Guardian. Please ask the sales staff how to get help with Clara Guardian.

6

Issue with nvvidconv Element in DeepStream 7.1 Container and Clara Guardian Inquiry

Dear junshengy,

I am reaching out regarding two matters:

1. Issue with nvvidconv in DeepStream 7.1

I am currently working with the DeepStream 7.1 container (nvcr.io/nvidia/deepstream:7.1-gc-triton-devel) and have set up my pipeline using Python and the DeepStream Python APIs. Here’s the summary:

Container: nvcr.io/nvidia/deepstream:7.1-gc-triton-devel
Pipeline Built using deepstream_new.py and config file deepstream_app_config_yolo_new.txt for YOLOv5 + emotion detection.

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5

[source0]
enable=1
type=3
uri=file:///opt/nvidia/deepstream/deepstream-7.1/deepstream_tao_apps_copy/configs/app/falling.mp4
num-sources=1
gpu-id=0
cudadec-memtype=0

[streammux]
batch-size=1
width=640
height=480
enable-padding=1
live-source=0

[primary-gie]
enable=1
unique-id=1
gpu-id=0
batch-size=1
network-type=0
config-file=/opt/nvidia/deepstream/deepstream-7.1/deepstream_tao_apps_copy/yolov5-face/config_infer_primary_yolo.txt
model-engine-file=/opt/nvidia/deepstream/deepstream-7.1/deepstream_tao_apps_copy/yolov5-face/weights/yolov5s.trt
infer-raw-output-dir=outputs

[preprocess0]
enable=1
config-file=emotion_preprocess_config.txt

[secondary-gie0]
enable=1
unique-id=2
gpu-id=0
network-type=1
batch-size=1
model-engine-file=/opt/nvidia/deepstream/deepstream-7.1/deepstream_tao_apps_copy/models/emotionnet/engine_file/emotionnet.etlt_b1_gpu0_int8.engine
config-file=/opt/nvidia/deepstream/deepstream-7.1/deepstream_tao_apps_copy/configs/app/emotionnet_config.txt
operate-on-gie-id=1
operate-on-class-ids=0

[sink0]
type=2
sync=0

“deepstream_app_config_yolo_new.txt”

[property]
gpu-id=0
net-scale-factor=0.007843
offsets=127.5;127.5;127.5
model-engine-file=/opt/nvidia/deepstream/deepstream-7.1/deepstream_tao_apps_copy/models/emotionnet/engine_file/emotionnet.etlt_b1_gpu0_int8.engine
labelfile-path=labels.txt
batch-size=1
network-type=1
num-detected-classes=7
input-object-min-width=20
input-object-min-height=20
gie-unique-id=2

[classification]
threshold=0.01

“emotionnet_config.txt”

import sys
import gi
gi.require_version(‘Gst’, ‘1.0’)
from gi.repository import Gst

Initialize GStreamer

Gst.init(None)

def main(config_path):
try:
# Load the configuration file
with open(config_path, ‘r’) as f:
config_file = f.read()

    # Parse the pipeline from the configuration
    pipeline = Gst.parse_launch(
        "filesrc location=falling.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! "
        "nvstreammux name=mux batch-size=1 width=640 height=480 ! "
        "nvinfer config-file-path=yolo-face-config.txt name=pgie batch-size=1 ! "
        "nvinfer config-file-path=emotionnet_config.txt name=sgie batch-size=1 ! "
        "nvvidconv ! video/x-raw,format=I420 ! avimux ! filesink location=output.avi"
    )

    # Get the pipeline bus to listen for messages
    bus = pipeline.get_bus()
    bus.add_signal_watch()

    # Start the pipeline
    pipeline.set_state(Gst.State.PLAYING)

    # Wait for the pipeline to finish processing
    loop = Gst.MainLoop()
    loop.run()

except Exception as e:
    print(f"Failed to load config file: {e}")
    sys.exit(1)

if name == ‘main’:
if len(sys.argv) < 2:
print(“Usage: deepstream_new.py <path_to_config_file>”)
sys.exit(1)

config_path = sys.argv[1]
main(config_path)

~
~
“deepstream_new.py”

python3 deepstream_new.py deepstream_app_config_yolo_new.txt

When I run the pipeline, I get the following error:

Failed to load config file: No such file or directory
** ERROR: <gst_nvinfer_parse_config_file:1392>: failed
Failed to load config file: gst_parse_error: no element "nvvidconv" (1)

  • Troubleshooting: I tried running gst-inspect-1.0 nvvidconv, but it returns:

    No such element or plugin 'nvvidconv'

It appears the plugin is missing or not loading, despite using the official NVIDIA container. Could you please assist in resolving this?

2. Clara Guardian Inquiry

Additionally, I would like to get more information regarding NVIDIA Clara Guardian:

  • Is Clara Guardian available as an open-source framework for integration?
  • If not open-source, what are the licensing or commercial terms?
  • Could you please provide contact details or the right channel to engage with the Clara Guardian team for technical and commercial discussions?

Looking forward to your support.

Best regards,
Sai A.

7

Are you running on dGPU ? nvvidconv is part of Jetpack but not part of DeepStream. nvvidconv only runs on Jetson platforms. Please use nvvideoconvert which is part of DeepStream and supported on both dGPU and Jetson.

It looks like Clara Guardian is a collection of nvidia’s SDK/platform, DeepStream and TAO are part of it and are not fully open source, but can be used for integration. Build applications using DeepStream and TAO and discuss them in the forum

8

Let’s narrow the question down, what does your current pipeline look like, what problems are you encountering, and did you refer to the reference pipeline I provided above?

I suggest that you consider drawing the bbox of the face through nvdsosd first, then continue with the next step of emotion classification.