Object mishandling detection - How to start?

AI & Data Science Computer Vision & Image Processing
,
1

Hello!

It is my first time working on a computer vision project. The goal is to detect people handling objects and classify the mishandling cases.

The project is not using real time streaming videos, it processes batches of videos every hour. There are many cameras that capture different angles and there are people and objects at different distances of the camera.

The team tried using some open source libraries but sometimes, objects that are not being manipulated in the video are classified as being manipulated because people appear in the background of the video, far from the camera, but with their hands close to the object in the image.

In other cases, objects moving on a conveyor belt in the factory are classified as mishandled due to high acceleration, but in reality the object is moving at a constant speed.

I am thinking about starting the project from scratch, and I became very interested in some things I saw on NVIDIA’s computer vision page, especially Metropolis and Jetson. However, there are so many tools that I do not know which ones to use or where to start.

Could you tell me which tools are the most suitable and which tutorials I should start with? How are object manipulation classifications usually done? Are there specific libraries for this, or is there any NVIDIA tool that already incorporates this kind of functionality?

3

Hi @leandrorcamargo,

You can use DeepStream with recorded video files. It supports file inputs and multiple sources, so the videos do not need to be live streams.

The NVIDIA components have different roles:

  • DeepStream builds the video processing pipeline.
  • TAO Toolkit trains or fine-tunes the models.
  • TensorRT accelerates inference.
  • Jetson is suitable when processing happens beside each camera. A discrete GPU may be easier if all hourly video batches are processed centrally.

There is no generic model that already understands “mishandling,” because that definition depends on your objects, factory process, and safety rules. I would start with this pipeline:

  1. Detect people and the relevant object categories.
  2. Track each person and object across frames.
  3. Associate each person with nearby objects.
  4. Classify a short sequence around each person-object interaction.

A tracker such as NvDCF or NvDeepSORT can maintain object identities, but it cannot determine whether a hand is touching an object. Two items being close in a 2D image does not establish physical contact.

For the background-person false positives, define camera-specific work zones and include those false positives as negative training examples. If accurate physical contact matters, use camera calibration, synchronized views, depth information, or a temporal model trained on the full interaction rather than relying only on 2D distance.

For the conveyor belt, avoid using raw pixel acceleration. Perspective and tracker noise can make constant physical motion look non-uniform in image coordinates. Calibrate each camera and compare the object’s motion with the expected conveyor direction and speed. DeepStream’s nvdsanalytics plugin can help with regions, directions, and line-crossing rules, but the final mishandling decision will still require your domain logic or trained classifier.

For a temporal classifier, start with TAO ActionRecognitionNet. The DeepStream 3D Action Recognition sample demonstrates file inputs, temporal batching, and action inference. If the action can be identified mainly from body movement, PoseClassificationNet is another option.

I recommend beginning with one camera and a small labeled dataset containing both real mishandling events and the failure cases you described. Once that baseline works, test it across the other camera distances and angles before selecting the final hardware.

For hardware sizing, how many hours of video, at what resolution and frame rate, must be processed during each one-hour batch window?

Thanks,
Atharva