Deepstream: IOptions lost after genericPostProcess_ in nvinferserver

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hardware Platform: NVIDIA GeForce RTX 3070

DeepStream Version: 7.0 and 9.0

NVIDIA GPU Driver Version: 580.159.03

Issue Type: Bug / question

Module: nvinferserver / nvdsinferserver

Issue

I found that IOptions attached to a BaseBatchArray is lost after the generic postprocess path.

In my case, I attach custom metadata before the inference callback:


auto options = std::make_shared<BufOptions>();

options->setValue("x-model-id", model_id);

outputsArr->setIOptions(options);

However, later in GstNvInferServerImpl::InferenceDone(), calling:


bufArray->getOptions()

returns nullptr.

Finding

After debugging, I found that genericPostProcess_() creates a new outBuf from inBuf and only copies bufId:


outBuf->setBufId(inBuf->bufId());

It does not propagate IOptions.

Adding the following line fixes the issue for me:


outBuf->setIOptions(inBuf->getSafeOptions());

So the patch is:


outBuf->setBufId(inBuf->bufId());

outBuf->setIOptions(inBuf->getSafeOptions());

Question

Could NVIDIA confirm whether IOptions should be propagated when genericPostProcess_() creates a new output batch array?

From my understanding, since BaseBatchArray already provides getSafeOptions() and setIOptions(), the options should be preserved across postprocess, similar to bufId.

4

Where did you add this code? in nvinferserver plugin? You can use IInferCustomProcessor to pass custom value. For example, you can add the following value in extraInputProcess()

        auto* mutableOptions = const_cast<dsis::IOptions*>(options);
        auto* bufOptions = dynamic_cast<dsis::BufOptions*>(mutableOptions);
        INFER_ASSERT(bufOptions);
        bufOptions->setValue(std::string("x-model-id"), std::string("yolov3-onnx"));

Then use the following code to get the added value in inferenceDone.

        if (inOptions && inOptions->hasValue("x-model-id")) {
            std::string modelId;
            if (inOptions->getValue<std::string>("x-model-id", modelId) == NVDSINFER_SUCCESS) {
            }
        }

Please refer to IInferCustomProcessor sample /opt/nvidia/deepstream/deepstream/sources/TritonOnnxYolo.

5

Thanks for the reply and the suggestion.

Let me clarify my use case.

In my pipeline, x-model-id is not a static value and it is not known before inference. The value is generated on the inference model side and returned through the gRPC response metadata.

The high-level architecture is:

mermaid-diagram-2026-06-03T06-17-12
mermaid-diagram-2026-06-03T06-17-121920×1226 144 KB

In our Kubernetes environment, the inference workload runs behind Istio. We use an EnvoyFilter on the inference workload sidecar to inject x-model-id into the gRPC response header. The value comes from the inference pod metadata, such as ISTIO_META_MODEL_ID.

So the value is only available after the remote inference response is returned. It is not available before inference starts.

That is why IInferCustomProcessor::extraInputProcess() does not fit this specific case. My understanding is that extraInputProcess() is useful when the custom value is already known before the request is sent to the backend. In my case, the metadata is generated after backend inference, so I need to propagate backend-response metadata to GstNvInferServerImpl::InferenceDone().

In DeepStream 7.0, I solved a similar requirement by adding a custom model_id field directly to IBatchArray / BaseBatchArray. However, for DeepStream 9.0, I would like to avoid changing the common batch-array data structure if possible. Since BaseBatchArray already has IOptions support, I tried to use setIOptions() / getOptions() to carry this metadata instead.

This is why I attached the value after receiving the Triton gRPC result:

auto options = std::make_shared<BufOptions>();
options->setValue("x-model-id", model_id);
outputsArr->setIOptions(options);

Then I read it later in GstNvInferServerImpl::InferenceDone():

const IOptions* options = bufArray->getOptions();

if (options && options->hasValue("x-model-id")) {
    std::string model_id;
    if (options->getValue<std::string>("x-model-id", model_id) == NVDSINFER_SUCCESS) {
        // use model_id
    }
}

The reason I looked into genericPostProcess_() is that the metadata was correctly attached before postprocess, but it was lost before reaching InferenceDone().

After debugging, I found that genericPostProcess_() creates a new outBuf from inBuf and preserves only the buffer id:

outBuf->setBufId(inBuf->bufId());

If I also propagate the existing options:

outBuf->setIOptions(inBuf->getSafeOptions());

then the metadata is preserved correctly and InferenceDone() can read x-model-id.

So my intention is not to add a new custom field to the common batch-array structure again. I am trying to use the existing IOptions mechanism to carry backend-response metadata through the postprocess path.

6

Thanks for the sharing! did you add the above code in InferGrpcClient::InferComplete of nvinfersever low-level library after getting ‘x-model-id’ from gRPC response header?
Since nvinferserver plugin and low-level library are open-sourced, you can modify the code to customize. In your case, ‘outBuf->setIOptions(inBuf->getSafeOptions());’ is needed, as you have discovered. Is there still any DeepStream issue to support? Thanks!

7

No, there is no further issue for this post.

Thanks for confirming. I will keep this customization in our internal build.

I have another issue related to building nvdsinferserver, but I will organize the details and create a separate post for it.

Thanks again for your support.

8

OK. Closing this topic.