I have a mae model, and I want to make to run faster with similar accuracy.
I have tried to quantize it by following the guide (PyTorch Quantization — Model Optimizer 0.15.0), and I can get the quantized model, and runs the quantized model on original data, the accuracy get a bit worse which is acceptable, however it runs much slower (around 2 times).
I cannot find where is the problem, hope some one can give me some advice. Thank you
Hi @relaxtheo ,
Can you help us with the model, performance matrix/logs, and repro scripts?
thanks
May I provide the model and test code privately?
Debugging run in vs.code, I can see such a message “Unable to load extension modelopt_cuda_ext and falling back to CPU version.”, but no such message running straightly in system by python command. Not sure if this is a potential reason.
How to fix this problem?
modelopt version is 0.15.1
I can reproduce this problem on resnet152, original model requires less than 2s, and quantized model requires almost 9s.
It seems loading extension modelopt_cuda_ext requires long time for quantized model, but run the job 2 times seems and there no big speed difference.
t2q.py.txt is the code to quantized the model, and t2t.py.txt is the test code for quantized model and original model.
t2q.py.txt (3.3 KB)
t2t.py.txt (4.1 KB)
Below is the log:
(pruning) user@server/data/WS/PruningWS/code$ python t2at.py
/data/WS/anaconda3/envs/pruning/lib/python3.11/site-packages/modelopt/torch/quantization/tensor_quant.py:92: FutureWarning: torch.library.impl_abstract was renamed to torch.library.register_fake. Please use that instead; we will remove torch.library.impl_abstract in a future version of PyTorch.
scaled_e4m3_abstract = torch.library.impl_abstract(“trt::quantize_fp8”)(
BEGINING 1…
BEGINING 2…
ORIGINAL: 1.84156, 1.51373
/data/WS/anaconda3/envs/pruning/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory ‘/tmp/tmpjfip2iuh’>
_warnings.warn(warn_message, ResourceWarning)
(pruning) user@server/data/WS/PruningWS/code$ python t2at.py
/data/WS/anaconda3/envs/pruning/lib/python3.11/site-packages/modelopt/torch/quantization/tensor_quant.py:92: FutureWarning: torch.library.impl_abstract was renamed to torch.library.register_fake. Please use that instead; we will remove torch.library.impl_abstract in a future version of PyTorch.
scaled_e4m3_abstract = torch.library.impl_abstract(“trt::quantize_fp8”)(
/data/WS/PruningWS/code/t2at.py:86: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See pytorch/SECURITY.md at main · pytorch/pytorch · GitHub for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don’t have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model_ft = mto.restore_from_modelopt_state(model_ft, torch.load(quantized_opt_file))
Inserted 782 quantizers
/data/WS/anaconda3/envs/pruning/lib/python3.11/site-packages/modelopt/torch/quantization/nn/modules/tensor_quantizer.py:837: UserWarning: Could not initialize the quantizer states for maxpool.input_quantizer. The quantizer states after load_state_dict could be in the wrong device. Please move the modules to the correct device after loading the state dict.
warnings.warn(warning_msg, UserWarning)
/data/WS/anaconda3/envs/pruning/lib/python3.11/site-packages/modelopt/torch/quantization/nn/modules/tensor_quantizer.py:837: UserWarning: Could not initialize the quantizer states for avgpool.input_quantizer. The quantizer states after load_state_dict could be in the wrong device. Please move the modules to the correct device after loading the state dict.
warnings.warn(warning_msg, UserWarning)
/data/WS/anaconda3/envs/pruning/lib/python3.11/site-packages/modelopt/torch/quantization/nn/modules/tensor_quantizer.py:740: UserWarning: maxpool.input_quantizer: No ‘_amax’ buffer to load amax into. '_amax` will be created as WAR for now. This behavior will change in future.
warnings.warn(
/data/WS/anaconda3/envs/pruning/lib/python3.11/site-packages/modelopt/torch/quantization/nn/modules/tensor_quantizer.py:740: UserWarning: avgpool.input_quantizer: No ‘_amax’ buffer to load amax into. '_amax` will be created as WAR for now. This behavior will change in future.
warnings.warn(
BEGINING 1…
Loading extension modelopt_cuda_ext…
BEGINING 2…
QUANTIZE: 9.27202, 8.82052
/data/WS/anaconda3/envs/pruning/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory ‘/tmp/tmprm3d_3jm’>
_warnings.warn(warn_message, ResourceWarning)