We are building an edge vision product on Jetson Orin NX and are putting together the capability plan for a small team. I would value a sanity check on what we have identified, and guidance on which DLI courses map to it.
The pipeline: 4 ONVIF network cameras, H.265 at 2560×1440, decoded in hardware through DeepStream, into an object detection model, with events matched against an external transaction record within a defined time window. Deployed to sites in India and Tanzania with unreliable connectivity, no remote restart, and high ambient temperature.
Clock synchronisation and timestamp handling across devices
Camera placement and lighting for detection reliability in real environments
False-positive management and threshold setting
Two questions:
1. What are we missing? We are particularly unsure whether we are underestimating anything on the deployment and maintenance side — model drift, model update over the air, or diagnosing a site remotely.
2. Which DLI courses would you recommend for our junior team, and in what order? We have a computer science graduate starting on this work with no prior computer vision experience, and two assistants supporting on dataset and benchmarking work.
We have identified Getting Started with AI on Jetson Nano, Getting Started with DeepStream for Video Analytics, and Optimization and Deployment of Models with TensorRT. Is there a recommended sequence for someone starting from zero, and roughly how much time should we allow between each?
We would also like to know whether the Jetson AI Specialist or Ambassador certification is worth targeting, and what it requires.
Model training has nothing to do with DeepStream, Jetson device is not suitable for model training. You can do the model training with a server. The training method depends on the model itself, you may contact the model owner for training details.
You need to check what is missing according to your detailed requirements.
From DeepStream point of view, we support on-the-fly model update . But as to the model transferring to the remote site, seems it has nothing to do with Jetson itself, you can use any method you are familiar with.
Since you requirements are related to several different areas and SDKs, they are related but relatively independent from the engineer’s point. The only suggestion is to study DeepStream last. The user manuals are more clear and suitable for engineers than DLI. And the study from zero to cover so many areas may take quite a long time.
It may depends on your own requirement and your final purpose.
Thank you, Fiona — that is helpful, and the point about splitting these across the right forums is taken. I will open separate threads for the TensorRT and Jetson items.
On your question about clock synchronisation: our detection events have to be matched against records produced by a separate system on the same site, within a defined time window. So the requirement is a defensible time relationship between two sources on one LAN, rather than absolute accuracy. We are looking at PTP across the local switch and at never stepping the system clock. I will take that to the Jetson forum as you suggest.
That leaves three questions that I think sit squarely with DeepStream, and I would be grateful for guidance on these:
Source failure handling. With four ONVIF/RTSP sources at 2560x1440 H.265, what is the recommended way to detect that an individual source has stopped delivering frames while the pipeline itself remains running, and to re-add that source automatically without restarting the pipeline? Sites are unattended with no possibility of a remote restart, so silent source death is our main operational risk.
On-the-fly model update in production. Thank you for the link — that answers the mechanism. What we are unsure about is the engine file itself. Since a TensorRT engine is built against a specific device, TensorRT version and JetPack version, is the expectation that a deployed fleet builds the engine on device from ONNX at update time, or that engines are built centrally per exact device and JetPack combination? If it is the former, is there guidance on the failure modes when an on-device build runs out of memory or is interrupted?
Decode verification. Is there a recommended way to confirm at runtime, not just at bring-up, that all four streams are being decoded on NVDEC rather than silently falling back to software, and to expose that as a health signal?
On the certification question — I understand from other threads that the Jetson AI Specialist and Ambassador programmes have been discontinued, so please disregard that part.
DeepStream app is just a RTSP client in your use case. If the camera stops delivering frames, the client will not know this. You need to contact your camera vendor for how to get the camera status and errors through the camera driver or SDK.
For your question about generating the TensorRT engine file from the onnx model, you need to decide according to your actual use case and situation. The engine generated in one Jetson Orin NX device can be directly used in another Jetson Orin NX device with exactly the same JetPack version as the generation Orin NX.
From DeepStream point, you can only use gst-nvv4l2decoder plugin to decode compressed video with NVDEC. Once you construct pipeline with this plugin, the NVDEC works, there is no mechanism to fallback to software decoder inside the plugin. I don’t understand your point.
Thank you, Fiona. Two of those are now settled for us.
On the engine file — that is exactly what we needed to know. If an engine built on one Orin NX runs on another Orin NX at the identical JetPack version, we can build centrally and ship the engine rather than building on each device. We will treat a uniform JetPack version across the fleet as a hard requirement and rebuild engines whenever we move JetPack forward.
On decode — understood, and my question was poorly framed. If the pipeline is constructed with nvv4l2decoder then there is no software fallback to detect, so there is nothing to monitor. That resolves it.
On source failure, may I ask once more, because this is our largest operational risk and I think I described it badly.
Our concern is not that the camera hardware fails. It is that the RTSP stream stops arriving for any reason — the camera reboots, the PoE switch glitches, the network path drops, or the RTSP session dies — while the DeepStream pipeline stays in PLAYING and continues to appear healthy. At an unattended site with no remote access, that state can persist for days. A camera vendor SDK would not help us here, since in most of these cases the camera itself is perfectly healthy.
What we would like to confirm is whether the intended DeepStream approach is:
the rtsp-reconnect-interval-sec and rtsp-reconnect-attempts settings in the source configuration of deepstream-app, and whether these cover a stream that goes silent without the TCP connection closing;
the runtime source add and delete mechanism shown in the deepstream-runtime-source-add-delete sample, driven by our own per-source frame-arrival watchdog, so that a dead source is removed and re-added without restarting the pipeline;
or something else you would recommend for unattended deployments.
If option 2 is the expected pattern, is there guidance on how long to wait on no frames before removing a source, and any caveats around nvstreammux batching when a source is removed and re-added while the pipeline is running?
As what I have post previously, the DeepStream app is only a RTSP client, what the client can do is very limited.
The rtsp-reconnect-interval-sec and rtsp-reconnect-attempts can work when the RTSP session is active and the RTSP packets can reach to the client within rtsp-reconnect-interval-sec x rtsp-reconnect-attempts time-slot. If the reconnection failed after rtsp-reconnect-interval-sec x rtsp-reconnect-attempts timeout, what the DeepStream interfaces can do is just to report the failure, but DeepStream never know the root cause of the failure as a client. If the app want to know the root cause and take the correct action to resolve the issue, it should not rely on DeepStream but need to check with other methods.
The runtime source add and delete mechanism only works when you already know the RTSP server works correctly, it will not tell you the reason of the failure when the source adding fails.
I think you can use the rtsp-reconnect-interval-sec and rtsp-reconnect-attemptsas the basic reconnection mechanism and when the reconnection failed after timeout, you may try to remove/add the failed source to check whether the RTSP source is available. If remove/add source failed either, you may need to try other methods to check the RTSP server status but not rely on DeepStream(which is just a RTSP client).
For your ONVIF sources, it may be much more complicated rtsp onvif client, you need to refer to GStreamer open source resources for help.
GStreamer works in asynchronized mode, there is no exactly time for “remove/re-add source” since the time will be related to a lot of things such as frame interval, downstream element processing speed, RTSP protocol stack handling, ethernet transferring speed, key-frame interval in the stream,…