2. I tried to jetson-containers build sglang:0.5.3, but errors occur when it proceeds to build flashinfer, as current cuda is 12.6 while flashinfer yields 12.8
3. I tried to pip install sglang==0.5.3 in existing sglang docker, but there occurs error information no kernel image is available for execution on the device as Orin is SM87
Is there any suggestion to deal with these problems? I’d greatly appreciate your time and help. Thank you in advance for your support!
Additionally, I tried to search existing whl packages at https://pypi.jetson-ai-lab.io/jp6/cu129/+simple/sgl-kernel/ and other cuda versions, and these packages not that sufficient, as my python version is 3.12.
*** Please note that this reply is generated by LLM automatically ***
I’d be happy to help you with your issue!
It appears that you’re trying to update Jetson AGX Orin’s Sglang to version 054, but you’re encountering errors with different approaches. Let’s break down each of your attempts and provide suggestions:
Attempt 1: Searching for existing Docker images
You’ve searched for the latest Docker image on Docker Hub, but the newest image available is version 047, which is outdated. This approach won’t work since the image is not up-to-date.
Attempt 2: Building from source
You’ve tried to build Sglang from source, but you’re encountering errors due to CUDA version incompatibility. The current CUDA version is 11.6, while the build process requires CUDA 12.8. This approach won’t work due to the version mismatch.
Attempt 3: Updating in existing container
You’ve tried to update Sglang in an existing container, but you’re encountering an error related to the Orin architecture. This approach won’t work since the update process is not compatible with the Orin architecture.
Suggestions
Wait for an updated Docker image: You can keep an eye on the Docker Hub repository for an updated image that matches your required version (054). This might take some time, but it’s the easiest way to get the latest version.
Build from source with compatible CUDA version: You can try building Sglang from source, but you’ll need to ensure that you have the correct CUDA version (12.8) installed on your system. This might require updating your CUDA installation or using a different build environment.
Use a different update method: If updating in an existing container is not possible, you can try using a different update method, such as using apt-get or pip to update Sglang. However, this might require additional dependencies or configuration.
Contact the Sglang maintainers: If none of the above approaches work, you can try reaching out to the Sglang maintainers or the NVIDIA developer community for assistance. They might be able to provide guidance or a temporary workaround.
In summary, the best approach would be to wait for an updated Docker image or build from source with the correct CUDA version. If these options are not feasible, you can try alternative update methods or seek assistance from the Sglang maintainers or the NVIDIA developer community.
*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***
Thanks for your reply. I’ve tried dustynv/sglang:r36.4-cu128-24.04, whose publish time close to the image you mentioned. I’ll try dustynv/sglang:r36.4-cu129-24.04 and report the result.
hi, I’ve pulled dustynv/sglang:r36.4-cu129-24.04 and the sglang version inside is 0.4.9, falling behind the current up-to-date version 0.5.3. Is there any further suggestion?
Follow the suggestion of D-G-Dimitrov in the following link https://github.com/dusty-nv/jetson-containers/pull/1470, I’ve pulled the corresponding docker image and solved this problem, thank you, AastaLLL and D-G-Dimitrov!