Why inference in jetson nano with fp16 is slower than fp32

when i use fp16 to inference with jetson nano,i found that is slower than fp32,. and i found that jetson nano is support fp16,so why is slower?

Hi,

How do you apply the inference on Nano?

Do you use TensorRT?
If yes, it can automatically choose an optimal algorithm (either fp32 or fp16) if the best mode is used.

$ /usr/src/tensorrt/bin/trtexec --best ...

Thanks.

@AastaLLL yes , i use TensorRT, you mean tensorRT can optimal choose to use fp32 or fp16?
i have model.onnx(fp32),now i want to convert onnx to .trt, and i have convert successful! but is slower than fp16

Hi,

Could you share the comment and corresponding benchmark data with us first?
Thanks.

F_Let_fp16.trt (156.1 KB)
F-Let.trt (267.5 KB)
F_Let.pth (276.3 KB)
convert_to_onnx.py (505 Bytes)
inference-Let-10.py (5.7 KB)
thank you very much

@AastaLLL

Hi,

We try to gnereate the ONNX format from .pth file with convert_to_onnx.py.
However, it shows the below dependency error:

$ python3 convert_to_onnx.py
1.8.0
Traceback (most recent call last):
  File "convert_to_onnx.py", line 11, in <module>
    model = torch.load('F_Let.pth', map_location="cuda:0")
  File "/home/nvidia/.local/lib/python3.6/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/nvidia/.local/lib/python3.6/site-packages/torch/serialization.py", line 772, in _legacy_load
    result = unpickler.load()
ModuleNotFoundError: No module named 'lenet'

Could you share the required LeNet class with us?
Or the converted ONNX file with us?

Thanks.

F-LeNet-0.pth (258.4 KB)
Uploading: F-LeNet-0.pth…

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi,

Thanks for updating the new .pth with us.
Unfortunately, it still request for a custom code when converting the .pth into ONNX format.

$ python3 convert_to_onnx.py
1.8.0
Traceback (most recent call last):
  File "convert_to_onnx.py", line 11, in <module>
    model = torch.load('F_Let.pth', map_location="cuda:0")
  File "/home/nvidia/.local/lib/python3.6/site-packages/torch/serialization.py", line 592, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/nvidia/.local/lib/python3.6/site-packages/torch/serialization.py", line 851, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'models'

Do you mind to share the model definition with us?
Or could you attach a converted ONNX file for us checking?

Thanks.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.