Context:
Error Details: The error indicates that the prompt length (1514 tokens) exceeds the maximum input length (255 tokens). It also suggests checking the TRTGptModel logs for more details about how the maximum input length is set.
Error during inference of request chat-6abeade2e0bc48f6bd02486825f35ed1 – Encountered an error when fetching new request: Prompt length (1514) exceeds maximum input length (255). Set log level to info and check TRTGptModel logs for how maximum input length is set (/home/jenkins/agent/workspace/LLM/release-0.12/L0_PostMerge/llm/cpp/include/tensorrt_llm/batch_manager/llmRequest.h:249)
Attempted Fixes: I’ve tried the following without success.
sudo docker run -it --runtime=nvidia --gpus all --shm-size=16GB --memory=“40g” --memory-swap=“50g” --cpus=8 -e PROFILE_ID=“193649a2eb95e821309d6023a2cabb31489d3b690a9973c7ab5d1ff58b0aa7eb” -e NGC_API_KEY=“$NGC_API_KEY” -v “$LOCAL_NIM_CACHE:/opt/nim/.cache” -p 8000:8000 nvcr.io/nim/mistralai/mistral-7b-instruct-v0.3:latest
sudo docker run -it --runtime=nvidia --gpus all --shm-size=16GB --memory=“40g” --memory-swap=“50g” --cpus=8 -e MAX_PROMPT_LENGTH=4096 -e NGC_API_KEY=“$NGC_API_KEY” -v “$LOCAL_NIM_CACHE:/opt/nim/.cache” -p 8000:8000 nvcr.io/nim/mistralai/mistral-7b-instruct-v0.3:latest
sudo docker run -it --runtime=nvidia --gpus all --shm-size=16GB --memory=“40g” --memory-swap=“50g” --cpus=8 -e NIM_MAX_MODEL_LEN=4096 -e MAX_INPUT_LENGTH=4096 -e NGC_API_KEY=“$NGC_API_KEY” -v “$LOCAL_NIM_CACHE:/opt/nim/.cache” -p 8000:8000 nvcr.io/nim/mistralai/mistral-7b-instruct-v0.3:latest
I would greatly appreciate assistance in resolving this issue.
Thank you,