Retail Object Detection - Training Help

pjpretorius.work · March 17, 2025, 10:05pm

Hi all,

I hope you are well. I am running a SeedStudio reComputer J3011 (Jetson Orin Nano 8GB) with Jetpack 6.2 Flashed. I have Deepstream 7.1 Installed and I am trying to test the retail_detector_100.onnx model, through my RTSP stream.

It seems that my model is loaded, but it just flashed random bounding boxes around the border of my RTSP stream. My camera is quite far away from some of the products. What would be the best way to train my model on my products from this distance starting from a single shelf?

Would it be necessary to add cameras on each shelf or maybe move this current one closer?

Any advice would be appreciated.

Kind Regards,

PJ Pretorius.

junshengy · March 18, 2025, 3:27am

For retail object detection, you can refer to this sample.

github.com/NVIDIA-AI-IOT/deepstream_tao_apps

apps/tao_others/deepstream-mdx-perception-app/README.md

master

## Description
The MDX perception sample application drives two Deepstream pipelines, i.e. retail 
item recognition and people ReID. Retail item recognition pipeline detects retail 
items from a video and extracts the embedding vector out of every detection bounding box. 
The embedding vector can form a query to a database of embedding vectors and find
the closest match.
People ReID detects people from a video and extracts the embedding vector out of 
every detection bounding box. The pipelines have a primary GIE module detecting
the objects of interest from a video frame. The secondary GIE module extracts an
embedding vector from the primary GIE result.

The TAO 4.0 pretrained models used in this sample application:

* [Retail Object Detection Binary](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/retail_object_detection)
* [Retail Object Recognition](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/retail_object_recognition)
* [ReIdentificationNet Model](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/reidentificationnet)
* [PeopleNet Transformer Model](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/peoplenet_transformer)

## Prerequisition

This file has been truncated. show original

This sample integrate retail_object_detection_binary_dino/retail_object_recognition model. you can find

TAO model has some fine tune guidelines here.

pjpretorius.work · March 18, 2025, 7:45am

Thanks, as you can see:

My RTSP Stream has a big bounding box around the stream, so it’s not picking up objects. I see the engine was built with 416x416 MAX dims, so do you think these products are too small for the model to pick up?

Is there anyone I can chat with or call who might be able to give me better advice on how to train my model or position my cameras?

Thanks,

PJ.

pjpretorius.work · March 18, 2025, 8:42am

I downloaded the MDX perception application. It seems like a good fit for what I am trying to do.

Is there a way that I can use this with RTSP? For example give it a URI, and then have it output an RTSP stream as well?

I see the current source is:

source:
csv-file-path: sources_retail_object.csv

Any help?

Thanks.

junshengy · March 18, 2025, 9:03am

Maybe it’s because the products are too small, or it may be because of the accuracy of the model.

If you need fine tuning, please use the TAO retail_object_detection_binary_dino/retail_object_recognition model mentioned above. We don’t know how to fine tune your model.

Yes, refer to /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.yml, set the sink type value to 4.

Rtsp stream as input is also supported.

pjpretorius.work · March 18, 2025, 10:14am

Thanks!

I managed to get it up and running, it seems it keeps falling back to my SW encoder rather than the HW encoder.

I am averaging 3FPS, and my RTSP stream is really laggy.

Would you have any suggestions on how I can increase my performance?

junshengy · March 18, 2025, 10:16am

There is no hardware encoder in Orin Nano .

pjpretorius.work · March 18, 2025, 10:33am

I see, thanks for letting me know.

How can I boost my performance? The thing is, I have not even trained this model on any of my products and it’s already suffering to get more than 10fps.

What would I need to do/get in order to run it at a stable performance level?

Thanks,

PJ.

junshengy · March 18, 2025, 10:57am

Set the device to maxn mode, adjust interval property value of nvinfer element. use int8 quantized model, or other methods to optimize the model and so son. You can ask related questions in TAO forum
If the performance still cannot match your requirements, you may need a more powerful device

pjpretorius.work · March 18, 2025, 11:59am

Would it be better if instead of using the live RTSP feed, I rather send myself JSON updates of detections through a dashboard?

junshengy · March 19, 2025, 5:51am

This can only reduce the CPU usage for encoding, but cannot reduce the time spent on inference, because nvstreammux/nvinfer/nvvideoconvert use the GPU

pjpretorius.work · March 27, 2025, 11:01am

I am trying to use transfer learning to add some of my products to the retail_object_detection_binary_dino/retail_object_recognition on NGC.

The problem is that NGC only offers a ‘.pth’ file as a trainable binary. How would I then be able to use TAO to train/transfer-learn the model if the only model available is a PyTorch model?

Thanks.

junshengy · March 27, 2025, 7:13pm

github.com/NVIDIA/tao_tutorials

notebooks/tao_launcher_starter_kit/retail_object_detection/retail_object_detection.ipynb

main

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retail Object Detection\n",
    "\n",
    "Transfer learning is the process of transferring learned features from one application to another. It is a commonly used training technique where you use a model trained on one task and re-train to use it on a different task. \n",
    "\n",
    "Train Adapt Optimize (TAO) Toolkit  is a simple and easy-to-use Python based AI toolkit for taking purpose-built AI models and customizing them with users' own data.\n",
    "\n",
    "<img align=\"center\" src=\"https://d29g4g2dyqv443.cloudfront.net/sites/default/files/akamai/TAO/tlt-tao-toolkit-bring-your-own-model-diagram.png\" width=\"1080\">\n",
    "\n",
    "### Sample prediction of FAN-Small+ DINO model\n",
    "Sample images are smudged\n",
    "\n",
    "<img align=\"left\" src=\"sample.jpg\" width=\"400\">\n",
    "<img align=\"right\" src=\"sample2.jpg\" width=\"600\">"

This file has been truncated. show original

You can get more help in TAO forums. I don’t know much about it.

yingliu · April 29, 2025, 2:42am

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

system · May 13, 2025, 2:43am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Advice on AI Retail Solution Computer Vision & Image Processing jetson , deepstream	8	29	March 13, 2025
Fine Tuning Retail Object Detection Models provided in NGC TAO Toolkit ngc	17	132	February 7, 2025
How to detect small object with deepstream? DeepStream SDK deepstream	14	39	February 26, 2025
FaceDetect Pre-Trained model implementation using DS DeepStream SDK	26	915	July 30, 2023
Custom Object Detection Model for Jetson nano using Deepstream DeepStream SDK cuda , tensorflow , ssd , yolo , nano	7	3543	November 23, 2021
Recommendation of an existing application for object detection and tracking with jetson Nano, YOLO V3 Tiny and Tensorflow Jetson Nano tensorflow	17	1834	October 12, 2021
DINO Retail Object Detection - Distillation TAO Toolkit tao , retail-object-detection	13	56	February 12, 2025
Moving past the tutorials: advice for custom object detection Jetson Nano	4	653	October 14, 2021
Face embedding for face identification in Python DeepStream SDK deepstream	3	92	October 15, 2024
Run Gaze Estimation model on Nvidia Jetson Nano on own data TAO Toolkit tensorrt	16	1390	March 12, 2022

Retail Object Detection - Training Help

Related topics