Hi,
I have Jetson Xavier NX and I am trying to create a simple GStreamer pipeline like this: GIE → Tracker → SIE. The secondary inference is an age classification which operates on object detections from primary inference and object tracker. I need to read raw tensor data from the age classification but when I enable the parameter “output-tensor-meta=1”, the pipeline freezes. It seems that the secondary inference exports tensor metadata six times and then maybe a buffer is overfilled, which stops whole GStreamer pipeline. My pipeline is based on deepstream-test2 example app, which freezes as well. However, when I use a different model or when I set “output-tensor-meta=0”, everything works fine. Unfortunately, I need the raw data from the age classification and I haven’t been able to find a solution to this problem for a couple of days. Have you any suggestion how to solve this problem, please?
Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) • DeepStream Version • JetPack Version (valid for Jetson only) • TensorRT Version • NVIDIA GPU Driver Version (valid for GPU only) • Issue Type( questions, new requirements, bugs) • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) • Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
If you only add "output-tensor-meta=1” to sample codes deepstream-test2, it will not cause such problem. Either there is some problem of the age classification model or there is something wrong with your processing with the tensor meta. The information is not enough to know what is wrong.
Hi,
thank you for fast response. It seems strange to me too. I have read the documentation and I am still not able to find the problem. Even if I don’t read metada from a probe, pipeline freezes. If I disable output-tensor-meta, it works fine. Maybe, I will do some tests with deepstream-app application instead of our pipeline.
Hi,
I have finally found a solution for this strange problem. I’ll try to explain briefly why the pipeline freezes and what my solution is. As I mentioned above, when I set “output-tensor-meta=1", the pipeline freezes after a couple of detections. The reason is simple but hard to find in gst-nvinfer plugin code.
When I enable exporting of tensor metadata, the NvDsInferTensorMeta keeps reference to tensor_out_object (GstNvInferTensorOutputObject). It causes that all resources allocated by batch are released only if the whole batch is processed. But it also uses nvinfer->pool, which is of limited size, and each object from the batch gets one resource from the pool. In the situation when there is more objects in the batch than the pool size is, deadlock happens. The reason is that the next object in the batch waits for the resource from the pool but the pool is not freed until the whole batch is processed. In other words, the code in function gst_nvinfer_process_objects should check the number of objects according to the pool size. I have found this problem just because my cluster settings weren’t perfect and the primary inference sent too many objects to secondary inference but it should have worked anyway.
I have fixed the code myself but I hope this short description allows you to improve the gst-nvinfer plugin in some of the next releases.