How am I able to make TensorRT work and quantize the AI model to FP4?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I tried looking on pages and related subjects for these but nothing of help came up, also if anyone could link me datasets for coding, programming and stuff like that from huggingface or others, it would be greatly appreciated!

4

TensorRT-Model-Optimizer/examples/README.md at main · NVIDIA/TensorRT-Model-Optimizer · GitHub does have examples to quantize to FP4.

Model Optimizer - a nvidia Collection HF space also has several FP4 quantized checkpoints you can choose from

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.