Suppress TensorRT warning and logging messages in Terminal

AI & Data Science Deep Learning (Training & Inference) TensorRT
, , ,
1

Description

I’m running TensorRT models to inference images captured from a camera. However, I’m seeing a ton of terminal output which I suspect is slowing down the inference framerate. Sample terminal output is shown below - can I disable all TensorRT terminal logging and how?

Environment

TensorRT Version: 8.4.1
GPU Type: Jetson Xavier NX 16GB production
CUDA Version: 11.4.14
CUDNN Version: 8.4.1
Operating System + Version: Jetson Linux 35.1
Python Version (if applicable): 3.8

Sample Output Log to Suppress:

[03/07/2023-15:30:23] [TRT] [I] The logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
[03/07/2023-15:30:23] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 1198, GPU 6153 (MiB)
[03/07/2023-15:30:23] [TRT] [I] Loaded engine size: 21 MiB
[03/07/2023-15:30:23] [TRT] [W] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[03/07/2023-15:30:23] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +1, GPU +1, now: CPU 1221, GPU 6154 (MiB)
[03/07/2023-15:30:23] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +22, now: CPU 0, GPU 227 (MiB)
[03/07/2023-15:30:23] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +0, now: CPU 1199, GPU 6154 (MiB)
[03/07/2023-15:30:23] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +3, now: CPU 0, GPU 230 (MiB)

2

Hi @zhifeis ,
Can you please share the detailed logs with us?
For the TRT part code, youcan change the log level and can try setting the log level to kWARNING

Thanks

5

The code responsible for the terminal output above is shown here:

def load_trt_model(model_path, device):
    Binding = namedtuple('Binding', ('name', 'dtype', 'shape', 'data', 'ptr'))
    logger = trt.Logger(trt.Logger.INFO)
    trt.init_libnvinfer_plugins(logger, namespace="")
    with open(model_path, 'rb') as f, trt.Runtime(logger) as runtime:
        model = runtime.deserialize_cuda_engine(f.read())
    bindings = OrderedDict()
    for index in range(model.num_bindings):
        name = model.get_binding_name(index)
        dtype = trt.nptype(model.get_binding_dtype(index))
        shape = tuple(model.get_binding_shape(index))
        data = torch.from_numpy(np.empty(shape, dtype=np.dtype(dtype))).to(device)
        bindings[name] = Binding(name, dtype, shape, data, int(data.data_ptr()))
    binding_addrs = OrderedDict((n, d.ptr) for n, d in bindings.items())
    context = model.create_execution_context()

    # warmup for 10 times
    for _ in range(10):
        tmp = torch.ones(1,3,288,640).half().to(device)
        binding_addrs['input.1'] = int(tmp.data_ptr())
        context.execute_v2(list(binding_addrs.values()))
    return [binding_addrs, context, bindings]

How would I set the log level? Where in the code would I do that?