Nvinfer opperating in secondary mode not always giving tensor output

Since the tensor output is the modle output, so you need to check whether there is model output from your classifier model when the issue happens. Even the output of the detector is correct, it does not mean the classifier model can classify the object correctly. It is decided by model but not deepstream.