Jetson Xavier - Inference multiple images

Hi,everyone.I want to make inference multiple images on the Jetson Xavier, but the tutorial of the official net of Nvidia only give the example of one images inference.
Does anyone have the same demands? If you know how to infer multiple images on the Jetson Xavier.I will be very appreciated to your response.

Sophia

Hi Sophia, you can check the TensorRT samples (found under /usr/src/tensorrt/samples) which support different batch sizes.

Essentially what you do for processing multiple images (i.e. batch size > 1) is:

  1. When building the TensorRT engine, call IBuilder::setMaxBatchSize()
  2. At runtime, call IExecutionContext::enqueue() or execute() with the desired batch size as the first parameter
  3. The input/output buffer passed to enqueue() or execute() is typically in NCHW format, where N is the batch size.

So basically instead of allocating input buffer to store one image for example, you would allocate the buffer to store N images (where N=batch size)

Thank you very much,dusty. I am trying your suggestions.

hi,dusty.I tried the sample of end_to_end_tensorflow_mnist under /usr/src/tensorrt/samples. It used the python API.
Here are the codes, I only added two lines in the sample.py file. setMaxBatchSize(10) and enqueue(10).
(1)for it showed invalid syntax, I removed the IBuilder::and IExecutionContext::.
(2)then it also showed “name ‘setMaxBatchSize’ is not defined”.

So,I want to verify your suggestion of setting batches with 'IBuilder::'and ‘IExecutionContext::’ is for C++ API? Not for python API?

def build_engine(model_file):    #构建引擎时,调用 IBuilder::setMaxBatchSize()!!!!!
    # For more information on TRT basics, refer to the introductory samples.
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
        builder.max_workspace_size = common.GiB(1)   ####  GiB(1) 函数的功能???? 最大的工作空间:max_workspace_size
###################################zhaoyu:按照dusty建议 ###############################
#        IBuilder::setMaxBatchSize(10)
        setMaxBatchSize(10)
        # Parse the Uff Network
        parser.register_input(ModelData.INPUT_NAME, ModelData.INPUT_SHAPE)
        parser.register_output(ModelData.OUTPUT_NAME)
        parser.parse(model_file, network)
        # Build and return an engine.
        return builder.build_cuda_engine(network)
def main():
    data_path, _ = common.find_sample_data(description="Runs an MNIST network using a UFF model file", subfolder="mnist")
    model_path = os.environ.get("MODEL_PATH") or os.path.join(os.path.dirname(__file__), "models")
    model_file = os.path.join(model_path, ModelData.MODEL_FILE)

    with build_engine(model_file) as engine:
        # Build an engine, allocate buffers and create a stream.  ##建立一个引擎,分配缓存并创建一个流
        # For more information on buffer allocation, refer to the introductory samples. (buffer allocation)??????
        inputs, outputs, bindings, stream = common.allocate_buffers(engine)
        with engine.create_execution_context() as context:
############################################zhaoyu:按照dusty建议 ###########################
#            IExecutionContext::enqueue(10)
            enqueue(10)	
            case_num = load_normalized_test_case(data_path, pagelocked_buffer=inputs[0].host)
            # For more information on performing inference, refer to the introductory samples.    !!!!!!!!
            # The common.do_inference function will return a list of outputs - we only have one in this case.   !!!!!!!!
            [output] = common.do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)
            pred = np.argmax(output)
            print("Test Case: " + str(case_num))
            print("Prediction: " + str(pred))

Could you please give me instructions on how to make multiple images inference on the exact end_to_end_tensorflow_mnist sample?
Hope to receive your response.

Hi, yes these are for the C++ API, not Python. For Python, see the tensorrt.Builder API here and the max_batch_size argument: https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/infer/Core/Builder.html#tensorrt.Builder

tensorrt.IExecutionContext.execute() accepts batch_size argument: https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/infer/Core/ExecutionContext.html#tensorrt.IExecutionContext.execute

I recommend that you check one of the TensorRT Python samples that already support multiple batch sizes, such as:

  • network_api_pytorch_mnist
  • int8_caffe_mnist
  • uff_ssd
  • yolov3_onnx

Note that to find these, I just did a grep for batch_size under /usr/src/tensorrt/samples/python

Thanks a lot for your nice response. I encountered a new problem. I am following the sample “Object Detection And Instance Segmentations With A TensorFlow MasK R-CNN Network”.

Prerequisites
Install the dependent Python packages.

pip install -r $TRT_SOURCE/samples/opensource/sampleUffMaskRCNN/converted/requirements.txt

When I installed packages of the requirements.txt file on the Jetson Xavier, it failed. So I confused
where do I make the implementations,on the Jetson,am I right? Or in the host PC?

Hi Sophia, are you trying that sample from https://github.com/nvidia/tensorrt? I believe MaskRCNN is new sample with TensorRT 6.0 (the master branch of that repo requires TensorRT 6.0). Whereas JetPack 4.2.x is using TensorRT 5.1. See the JetPack 4.3 Developer Preview for sneak-peek of TensorRT 6.0 for Xavier: https://developer.nvidia.com/jetpack-4_3_DP

After you reflash with it, it includes the TensorRT samples at /usr/src/tensorrt/samples. If you still have trouble with the MaskRCNN sample after that, you may want to post a new topic about it to the TensorRT forum or Xavier forum.