ResNet50 classifier model as pgie

Thanks for the answer! I read the materials above and slightly changed the configuration file.

[property]
process-mode=1 # Process full frames
network-type=1 # Classifier

gpu-id=0
net-scale-factor=1.0
model-color-format=1
onnx-file=resnet50.onnx
model-engine-file=resnet50_trt.engine
batch-size=1
force-implicit-batch-dim=1
interval=0
network-mode=0
labelfile-path=labels.txt
output-blob-names=495
classifier-threshold=0.51
gie-unique-id=1
cluster-mode=2

[class-attrs-all]
pre-cluster-threshold=0.2
topk=20
nms-iou-threshold=0.5

I also checked the functionality of the engine file using bare trt following the example from here: https://github.com/NVIDIA/TensorRT/blob/main/samples/python/introductory_parser_samples/onnx_resnet50.py

def load_normalized_test_case(test_image, pagelocked_buffer):

    # Converts the input image to a CHW Numpy array
    def normalize_image(image_candidate):
        # Resize, antialias and transpose the image to CHW.
        c, h, w = 3, 224, 224
        image_arr = (cv2.resize(image_candidate, (w, h), interpolation=cv2.INTER_AREA)
                     .transpose([2, 0, 1])
                     .astype(trt.nptype(trt.float32)))

        # This particular ResNet50 model requires some preprocessing, specifically, mean normalization.
        return (image_arr / 255.0 - 0.45) / 0.225

    # Normalize the image and copy to pagelocked memory.
    np.copyto(pagelocked_buffer, normalize_image(test_image).ravel())
    return test_image


if __name__ == "__main__":
    image = cv2.imread('./images/laptop.png')
    labels = open(LABEL_FILE, "r").read().split("\n")
    runtime = trt.Runtime(TRT_LOGGER)
    with open(ENGINE_PATH, "rb") as engine_file:
        serialized_engine = engine_file.read()
    engine = runtime.deserialize_cuda_engine(serialized_engine)
    inputs, outputs, bindings, stream = allocate_buffers(engine)
    context = engine.create_execution_context()
    test_case = load_normalized_test_case(image, inputs[0].host)
    begin = time.time()
    trt_outputs = do_inference_v2(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)
    pred = labels[np.argmax(trt_outputs[0])]
    end = time.time()
    print(f"Prediction is {pred} in {end - begin} secs")
    free_buffers(inputs, outputs, stream)

The model works correctly, but I still can’t get metadata from Deepstream. ObjMeta is null.

deepstream nvinfer only supports this kind of pre-processing y = net scale factor*(x-mean). please refer to explanation.
nvinfer plugin and low level lib are opensource. you can modify the code to customize, or you can use nvdspreprocess to customize preprocessing. please refer to doc and sample /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-preprocess-test/.

Thanks for the tip! I tried to include the preprocessing plugin in the pipeline and gave it the config from /opt/nvidia/deepstream/deepstream-6.1/sources/apps/sample_apps/deepstream-preprocess-test/test/config_preprocess_classifier_resnet50.txt.
There is now a green bbox in the image, however obj_meta is still null :(

is green bbox correct? if there are normal bboxes,there should be object meta. please refer to deepstream_test_1.py for how to access objectmeta.

I get the metadata according to the official deepstream examples, but they are empty. The green square simply appears in the corner of the screen and does not change in any way, regardless of the class of the object in the image. There may be a problem with the preprocessing configuration?

    def frame_sink_pad_buffer_probe(self, pad, info, u_data):
        gst_buffer = info.get_buffer()
        if not gst_buffer:
            print("Unable to get GstBuffer ")
            return

        batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
        l_frame = batch_meta.frame_meta_list
        while l_frame is not None:
            try:
                frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            except StopIteration:
                break
            l_obj = frame_meta.obj_meta_list
            while l_obj is not None: # Value here is null
                try:
                    # Casting l_obj.data to pyds.NvDsObjectMeta
                    obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
                except StopIteration:
                    break
                try:
                    l_obj = l_obj.next
                except StopIteration:
                    break

            try:
                l_frame = l_frame.next
            except StopIteration:
                break

        return Gst.PadProbeReturn.OK
  1. did it appear around the detected objects? can you share a screenshot?
  2. from the code and printing, there is no detected object.

Shure!
Wouldn’t deepstream create an object with class information if a certain class was recognized?

thanks for the update. where did you add this frame_sink_pad_buffer_probe?you should add a src probe function on pgie.

Shouldn’t PadProbe come after inference? Shouldn’t it be added to sink? I tried different things, src and sink for inference. Both src and sink for nvosd. The result did not change.

the probe function should be added after inference.

  1. nvinfer plugin is opensource. you can add log in attach_metadata_classifier of DeepStream to check if there is object. especially you need to rebuild and replace /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_infer.so.
  2. or you can provide the onnx model ,code and simplified code. let me have a try.

Sure! Here is the simplified code to get the model and run the pipeline. I would like to try to solve the problems using the correct settings instead of code. If it is possible of course.
resnet50_issue.zip (90.5 MB)

  1. the green rectangle(0,0,100,100) is ROI area, not the object bboxes. I have run the code with add log in C code attach_metadata_classifier. the model give null result.

  2. deepstream nvinfer supports pre-processing y = net scale factor*(x-mean), please refer to config_infer_secondary_carcolor.txt in DeepStream SDK for how to set this value. net-scale-factor=1.0 is wrong.

  3. from the python code “np.argmax”, you need to do argmax after inference. please refer to sample deepstream-infer-tensor-meta-test in DeepStream SDK, if network-type is 100, the inference results will be attached in meta, then you can process inference results in probe function.

If your model also returned null. Apparently the problem is still in preprocessing, since this engine runs on pure tensorrt.

Sorry for the late reply, Is this still an DeepStream issue to support? Thanks!

Hello!, yes, unfortunately, it is still DeepStream issue. I’m trying to set up correct preprocessing, or use another resnet classifier

we suggest using yolo detection model, which can give objects and classifications. please refer to this deepstream FAQ.

But i need to classify full frames, not objects.

Hi, I’m trying to get metadata using the method described in deepstream-infer-tensor-meta-test. However, the buffer still contains null. I am using the following code and config.

    def frame_sink_pad_buffer_probe(self, pad, info, u_data):
        gst_buffer = info.get_buffer()
        if not gst_buffer:
            print("Unable to get GstBuffer ")
            return

        # Retrieve batch metadata from the gst_buffer
        # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
        # C address of gst_buffer as input, which is obtained with hash(gst_buffer)
        batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
        l_frame = batch_meta.frame_meta_list
        while l_frame is not None:
            try:
                # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
                # The casting is done by pyds.NvDsFrameMeta.cast()
                # The casting also keeps ownership of the underlying memory
                # in the C code, so the Python garbage collector will leave
                # it alone.
                frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            except StopIteration:
                break
            l_user = frame_meta.frame_user_meta_list
            while l_user is not None:
                try:
                    # Casting l_obj.data to pyds.NvDsObjectMeta
                    user_meta = pyds.NvDsUserMeta.cast(l_user.data)
                    print(f"obj_meta: {user_meta}")
                    tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
                    info = pyds.NvDsInferLayerInfo.cast(tensor_meta.output_layers_info(0))
                    print(f"Output layer info")
                    print(f"name: {info.layerName}")
                    print(f"{tensor_meta.out_buf_ptrs_dev}")
                except StopIteration:
                    break
                try:
                    l_user = l_user.next
                except StopIteration:
                    break

            try:
                l_frame = l_frame.next
            except StopIteration:
                break
        self.fps.update_fps()
        return Gst.PadProbeReturn.OK
[property]
process-mode=1 # Process full frames
network-type=100 # Other

gpu-id=0
net-scale-factor=0.01735207
model-color-format=0
onnx-file=/home/user/classify/models/resnet50.onnx
model-engine-file=/home/user/classify/models/resnet50.onnx_b1_gpu0_fp32.engine
batch-size=1
interval=0
network-mode=0
labelfile-path=/home/user/classify/config/labels.txt
output-blob-names=495
output-tensor-meta=1
classifier-threshold=0.51
gie-unique-id=1
cluster-mode=2

[class-attrs-all]
pre-cluster-threshold=0.2
topk=20
nms-iou-threshold=0.5

Looks like i need custom preprocessing in nvinfer plugin and custom parse lib.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

dstest_appsrc_config.txt (3.3 KB)
zebra.zip (73.4 KB)
using this configuration file, after testing a zebra picture, I can get the index 340 in 1000 classifications. here is the test logs after adding printing in nvdsinfer_context_impl_output_parsing.cpp in /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer.
log-1116.txt (55.5 KB)

especially plugin did not gave “zebra” label string. it is because application failed to parse the label strings in labels.txt. nvinfer plugin and low level are opensource. you can modify InferPostprocessor::parseLabelsFile to customize.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.