Unreliable alert generation for weapon-like object detection in uploaded video

Accelerated Computing Intelligent Video Analytics Visual AI Agent
,
1

Setup:

Steps to Reproduce:

  1. Open the VSS UI and upload a video showing a person swinging a knife or intentionally damaging items (e.g., boxes).

  2. Modify the summarization prompt to reflect expected events in the video.

  3. Create an alert for the event you want to detect (e.g., “a person is seen handling an item resembling a weapon”).

  4. Click Summarize.

  5. Observe the Alerts section.

Actual Outcome:

  • In many cases, no alert is generated, even when the video clearly shows a person handling a knife.

  • Occasionally, the alert is generated correctly using the same prompt and event configuration.

  • The behavior is inconsistent and unreliable.

Expected Outcome:
Alerts should be reliably generated for videos showing a person swinging a knife or handling weapon-like objects.

Additional Notes:

  • The issue appears to be intermittent and not tied to specific prompt wording or video format.

  • This inconsistency may impact the reliability of VSS in safety-critical or surveillance scenarios.

3

This seems to be a problem with the accuracy of the VLM model. There are two possible solutions:

  1. Switch to a larger VLM model，such as remote GPT-compat VLM model.
  2. Fine-tune the current Cosmos-Reason model.
    In fact, only when the model accuracy is high enough can the VSS framework achieve greater performance.
4

Thank you for your reply.

Regarding point 2 (Fine-tune the current Cosmos-Reason model):

2.1) Do you have any datasets to share with us for fine-tuning the model to improve the detection and understanding of the following scenarios:

  • Weapon-holding (e.g., guns or knives)
  • Vandalism acts

Alternatively, can you perhaps point us to some open-source datasets we could use?

2.2) We understand that you have ready-to-use recipes for fine-tuning. Do you have any specific recommendations for this process? For example:

  • Catastrophic Forgetting: Should we mix a dedicated new dataset with the original one to help avoid issues like catastrophic forgetting, and if so, what is the recommended ratio?

  • Sample Size: What is the recommended minimum number of samples per each scenario (e.g., weapon-holding and vandalism) needed to achieve reliable performance metrics?

  • Hyperparameters: How should we set up the hyperparameters (e.g., learning rate, batch size) to best reflect the size and diversity of our specialized datasets?

We would appreciate detailed information on both the recommended model alternatives and the fine-tuning specifications for Cosmos-Reason

5

Sorry, I don’t know much about this

I only understand the VSS-related parts and am not very familiar with the details of the training parameters. You can refer to the following command to finetune. For algorithm details, you can raise an issue on GitHub or ask sales for more professional help.

Inference sample with your finetuned model:

./scripts/inference.py \
  --model ./outputs/sft/20250516061336/safetensors/final \  # Path to your fine-tuned model
  --prompt prompts/caption.yaml \\
  --videos assets/sample.mp4 \
  -v
6

Some other suggestions: Optimize the prompt; for example, the weapon is too broad, so you can specify a specific weapon category, such as gun/knife, etc.

Nvidia TAO toolkit provides support for Cosmos finetune; you can view the documentation here and post your questions here.

7

Some other suggestions: Optimize the prompt; for example, the weapon is too broad, so you can specify a specific weapon category, such as gun/knife, etc.

Thank you, did try that but with no major improvement.

Will try to look into the finetune at a later time.

8

Hey Karol,

Here’s a Cosmos ‘Cookbook’ with examples to help ramp up on fine-tuning Cosmos Reason. Zero shot capability can improve with system and user prompt engineering, but SFT or RL really shows a dramatic increase in accuracy.

9

Please note that this might be a more serious issue as alert generation seems to be rather unreliable even on the examples provided by NVIDIA.

I used video-search-and-summarization/examples/cv-event-detector and video-search-and-summarization/deploy/docker/event_reviewer.

Events for examples with “cardboard boxes” and “ladder without a hardhat and safety vest” provided by default are not working properly. They are not reliable and more often than not they aren’t properly detected.

I tested them with Cosmos-Reason1-7B and even with different prompts or “Enable Descriptions” and “Enable Alert Reasoning”, but results still vary, as sometimes events are detected, but more often they are not.

Below I am attaching video with cardboard boxes example, with default settings/prompts.

10

  1. For cv-event-detector mode, nvdsanalytics plugin is used to detect events and slice the video, then VLM (Cosmos-Reason) is used for verification.

  2. This means that if nvdsanalytics is inaccurate, VLM will also not perform well.
    In the cardboard boxes scenario, CV pipeline/VLM failed to detect the events, so LLM gave the same result in the summary, resulting in consistent performance. However, this is likely due to slicing in the CV pipeline.
    In the warehouse scenario, the CV pipeline sliced ​​the correct segments, so both VLM and LLM gave the correct results.
    I tested it multiple times, and the results were consistent.This mode is used to ensure compatibility with the user’s production environment’s CV pipeline.

11

Thanks for the answer @junshengy

It seems not to be working so well for us.
Please provide the full set of prompts and config (incl. docker compose, .env) that you are using for both the warehouse and conveyor belt examples. Screenshots will be very welcome!

12

I used the default deployment, and my results were consistent with those in your video.

What I mean is, for the cardboard boxes scenario, the cv pipeline output was consistent every time; the VLM output an alert result of false, and the LLM also output false.

Therefore, this doesn’t seem to be an error, but rather a precision issue with the cv pipeline’s event detection.

This cv pipeline performs reasonably well in the warehouse scenario.

Please create a new topic for the cv-event-detector mode, we’ve discussed too many issues under one topic.

13

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks.