How to send encoded frames to http service on detections only

I have a use-case where I need to send encoded frames to a http service when there are detections.

Ie. read from rtsp stream - decode - inference - encode - send 30s clips of frames if there has been a detection.

Without being able to do this easily it limits the platforms use a an edge device as we don’t want to be full time streaming. We only want to stream when interesting things are happening. I’m surprised there is no standard capability for this??

I’m wondering if anyone has advice on he best way to do this?

I’m thinking that I could modify the deepstream-app and change the encoded file sink bin code to replace the final file sink element with my own element based on the nvmsgbroker element. On my version of the msg broker I could detect if detections were made and send to my external server.

Does this sound like a viable solution or would there be better ways?
It does rely on the detections meta being available all the way to the end of the pipeline after the frames are re-encoded.

Hi,
Please check

deepstream_sdk_v4.0.1_jetson\sources\apps\sample_apps\deepstream-test4
deepstream_sdk_v4.0.1_jetson\sources\apps\sample_apps\deepstream-test5

It demonstrates the usecases of uploading data to kafka server.

I’ve seen these but from my understanding they only send text data. I need the frames sent. We want to send out video but only when there is something interesting to watch like when an object is detected.

Without this capability your edge devices need big internet connections so they can full-time stream all video feeds such that it can be viewed in the cloud in combination with th kafka messages highlighting events etc.

I don’t see how the message broker sending just text information has any benefit other than for counting detections or something like that.

Hi,
Please check 6 in FAQ
You can drop the frames in prob callback if no objects are detected.

We have ‘Encode + RTSP streaming’ in sink group:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream_Development_Guide%2Fdeepstream_app_config.3.2.html%23wwpID0ENHA
Drop the frames before encoding should be close to your usecase.

Encode + rtsp streaming doesn’t really fit as that requires some other system to initiate a connection into our private network to “read” the stream.

The closest thing is the msg broker plugin but it only sends metadata and not the actual frames.

So there’s a couple of things here that would make deepstream extremely useful for edge device scenarios:

  1. ability to save encoded video clip files when detections are made. Ie. security scenario - when a person is detected start recording a file for 30 seconds then stop till next detection.

  2. same as above but with the msgbroker which gives the ability to send metadata about detections as well as frames or video clips

  3. an example code using a probe to access encoded frames. This way I can write my own code to check the meta on an element downstream of the encoder and then use an api to push the encoded frames to another service such as amazon kinesis.

The reason I’m looking at these options is that on edge device scenarios you do not want to be full time transmitting video from multiple cameras as it uses huge bandwidth. You only want to send the video when there’s something interesting. Simply dropping frames with an upstream probe is not good enough.

Hi,
YUV420 frames are with larger size( widthxheightx1.5 bytes ), so we would suggest compress the frames into h264/h265 stream and send out in rtsp protocol. Using msgbroker might not be a good choice.

I need to access the compressed frames after h264/265 encoding. I will create a new sink element as a copy of the msgbroker element and change the code to send each h264/265 encoded frame to amazon kinesis via their C api where you ‘put’ one frame at a time.
Is there any example code of how I would access each individual frame after its encoded?

Hi,
We don’t have sample same as this usecase. You may customize deepstream-test4. After nvdsosd, you can link as

... ! nvdsosd ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! nvv4l2h264enc ! appsink

And get each encoded h264 frame in appsink.