DeepStream Test3 Running pipeline on Different GPU

[ljay189]

Hi,

When running deepstream-test3, I want to override the gpu-id configs for pgie
and all gstreamer elements that utilize the GPU. On a multi-gpu server, after setting the gpu-id to 1, I noticed that there was still a bit of utilization on gpu-id=0.

Please correct me if I am wrong, but I am assuming that this is occurring because input frame buffer is attached to gpu-id=0. I noticed that this slows down the inference pipeline by 14 seconds when running inference on four 5306 frame videos. Is there a way to make it so that every single operation performed on the GPU placed on the same device (that is not gpu-id=0)?

FYI the use case is as follows:

# --gpu 1 (override gstreamer gpu-id settings from config 
# such as pgie / sgie and replace with this value)
python3 deepstream_test_4.py --gpu 1 

Gstreamer element gpu-id property is being overriden as follows:

element.set_property("gpu-id", 1) # Override existing gpu-id

What was done

Set gstreamer elements gpu-id to 1.
Result: there still exists some gpu utilization in gpu-id = 0

Goal

Ensure that all gpu resource utilization is container to a single gpu card.

DeepStream Environment Specifications

• Hardware Platform (Jetson / GPU) : dGPU
• DeepStream Version : 5.1
• TensorRT Version : 7.2.2.1
• NVIDIA GPU Driver Version (valid for GPU only) : 470.42.01

Thank you.

[Amycao]

Can you share the sample and config files you used to repro the issue you mentioned?

[ljay189]

Can you share the sample and config files you used to repro the issue you mentioned?

I have attached the code needed to reproduce the issue. gpu_reproduction.py (15.3 KB). Please place the python script in the same location as deepstream_test_3.py.

FYI: The config file is identical to the one used in deepstream_test_3.py: dstest3_pgie_config.txt

I ran the script with the following configuration:

python3 gpu_reproduction.py -i file:///opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_qHD.h264 -g 1

I get the following utilization output from nvidia-smi:

[Amycao]

You can not set gpu-id directly for source_bin. please refer to function decodebin_child_added in sources/apps/apps-common/src/deepstream_source_bin.c for how to set gpu-id for dGPU.

[ljay189]

You can not set gpu-id directly for source_bin. please refer to function decodebin_child_added in sources/apps/apps-common/src/deepstream_source_bin.c for how to set gpu-id for dGPU.

Thank you for the helpful response. I took a look at the function decodebin_child_added and managed to get the problem resolved.

For those who may come across similar use case, I will leave my basic approach below. Feel free to modify to suit your specific design / use case.

def decodebin_child_added(child_proxy,Object,name,user_data):
    print("Decodebin child added:", name, "\n")
    try: 
        # feel free to receive gpu id as input variable elsewhere 
        gpu_id = 1
        # set gpu-id if it exists
        Object.set_property('gpu-id', gpu_id) 
    except: 
        # current Gstreamer element does not have property "gpu-id"
        # Feel free to do some logging for debugging purposes 
        pass

    if(name.find("decodebin") != -1):
        Object.connect("child-added",decodebin_child_added,user_data)