NIM_MAX_MODEL_LEN environment variable does nothing

[mark355]

Hey,

The documentation here:

Configuring a NIM - NVIDIA Docs

States clearly that the NIM_MAX_MODEL_LEN environment variable can control the context window length.
However, setting it does nothing and the vLLM engine under the hood is launched with the default value of None.

In the following question the top answer by the NVIDIA team states that said parameter does not exist:

Mistral v.03 NIM container on AWS Sagemaker: cannot set max_model_len!

Hello, When launching a Mistral Instruct v0.3 NIM container on a ml.g5.4xlarge AWS Sagemaker instance, we get the following error: ValueError: The model’s max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (16912). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine. How can we specify a value for max_model_len when launching the SageMaker endpoint? We tried setting “MAX_MODEL_LEN” and “OPTION_MAX_MODEL_LE…

Please either fix the docs or introduce the variable.

[calexiuk]

Thanks for this feedback, Mark! I’ll pass it along!