Originally published at: https://developer.nvidia.com/blog/accelerating-hebrew-llm-performance-with-nvidia-tensorrt-llm/
Developing a high-performing Hebrew large language model (LLM) presents distinct challenges stemming from the rich and complex nature of the Hebrew language itself. The intricate structure of Hebrew, with words formed through root and pattern combinations, demands sophisticated modeling approaches. Moreover, the lack of capitalization and the frequent absence of punctuation like periods and commas…
To send requests and receive the results in streaming mode, you first have to apply the following patch to the TensorRT-LLM backend: Fixed Whitespace Error in Streaming mode by enochlev · Pull Request #423 · triton-inference-server/tensorrtllm_backend · GitHub
Then request streaming using the same curl call in the blog and set “stream”:true