Originally published at: https://developer.nvidia.com/blog/boosting-llama-3-1-405b-performance-by-up-to-44-with-nvidia-tensorrt-model-optimizer-on-nvidia-h200-gpus/
The Llama 3.1 405B large language model (LLM), developed by Meta, is an open-source community model that delivers state-of-the-art performance and supports a variety of use cases. With 405 billion parameters and support for context lengths of up to 128K tokens, Llama 3.1 405B is also one of the most demanding LLMs to run. To…
accessibility:
Speed is not the only thing that matters for the LLMs. The other thing that i hope to see from nvidia is how to make this technology more accessible, specially with it requiring lots of ram and currently some of the best laptops by Lenovo or HP with a 4090 can only have maximum of 96gb of ram. maybe M.2 Modules that would act like ram with the pcie gen5 than would be 256 or 512gb and can fit in the second m.2 slot.
the other thing that i want to see the prebuilt Kubernetes models. so if you want to train a model and don’t care about time and you are just learning. you can have multiple devices on the network to work as one.
for if someone has multiple machine like multiple gaming computers or etc