Quantization Support for PyTorch on Jetson AGX Orin: Issues and Solutions?

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hello,

I am currently working on a project using the Jetson AGX Orin development board, which runs on an ARM64 architecture. I am using a specific version of PyTorch: 1.12.0a0+2c916ef.nv22.3, provided by NVIDIA.

I encountered an issue with PyTorch quantization support on this platform. When attempting to check available quantization engines using:

print(torch.backends.quantized.supported_engines)

the output is ['None'], indicating no available quantization engines. Additionally, attempting to set the quantization backend to qnnpack resulted in the following error:

RuntimeError: quantized engine QNNPACK is not supported

Given that fbgemm is typically not supported on ARM64, I am wondering:

  1. Does this specific PyTorch version for Jetson AGX Orin support any quantization backend?
  2. Are there any available quantization engines for this platform that I might have missed?
  3. What are the recommended approaches for performing quantization on the Jetson AGX Orin with PyTorch?

I would appreciate any insights or solutions, especially in terms of how to enable or use quantization on the Jetson AGX Orin.

Thank you for your assistance!

1 Like
4

Hi,

1. It looks like quantization is not supported in this release.
But since the package is released for a while, it’s recommended to upgrade to a newer version.

2. TensorRT can do the PTQ for precision int8.

3. Usually, we recommend running the DNN model with TensorRT and applying PTQ for acceleration.

Thanks.

5

Thanks you for your help.

Since quantization is not supportted in the release 1.12.0a0+2c916ef.nv22.3, can you tell me which release of PyTorch should I fetch for my Jetson AGX Orin？
I need to make sure that the custom distro supports quantization operations.

Best Regards.

1 Like
7

Hi,

Just double-check our AGX Orin with JetPack 6.0 GA.

>>> torch.__version__
'2.4.0a0+07cecf4168.nv24.05'
>>> print(torch.backends.quantized.supported_engines)
['qnnpack', 'none']

qnnpack is listed in the PyTorch v2.4. Please give it a check.

Thanks.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.