My model structure is like CNN-RNN. The first simple CNN model extract feature vector from frame. and the second model use the feature vector and generated state vector as the input. The state vector is generated by itself.
Now I want to implement these two models in deepstream. my idea is to build this pipeline
like nvpreprocess->nvinfer1->modified_nvpreprocess->nvinfer2. Setting nvinfer1 output_tensor=1 and I will process the added tensor in my modified nvpreprocess plugin. Did it work? or there may be some more simple method?
Yep, actually the second model is RWKV, the model includes two inputs: embedding_vector and the previous state vector. two outputs: classify result and the current state vector. In my application, embedding vector is actually the extracted feature by CNN. and as you can see, the state vector will be passed through from the previous output to the model next input.
you can use pipeline nvinfer->nvpreprocess->nvinferserver.
nvinferserver supports feeding output as inputs to the model. User can derive interface IInferCustomProcessor, then implement extraInputProcess() and inferenceDone() for loop process. pleae refer to doc loop and sample opt\nvidia\deepstream\deepstream-6.2\sources\objectDetector_FasterRCNN\nvdsinfer_custom_impl_fasterRCNN\nvdsinferserver_custom_process.cpp.