Nvidia Cosmos running on Jetson

[johnnynunez]

🚀 𝗧𝗵𝗿𝗶𝗹𝗹𝗲𝗱 𝘁𝗼 𝗮𝗻𝗻𝗼𝘂𝗻𝗰𝗲 𝗮 𝗺𝗮𝗷𝗼𝗿 𝗺𝗶𝗹𝗲𝘀𝘁𝗼𝗻𝗲 𝗶𝗻 𝗺𝘆 𝗷𝗼𝘂𝗿𝗻𝗲𝘆 𝘄𝗶𝘁𝗵 𝗡𝗩𝗜𝗗𝗜𝗔 𝗖𝗼𝘀𝗺𝗼𝘀™! 🌌⁣

⁣

I successfully ported the revolutionary NVIDIA 𝗖𝗼𝘀𝗺𝗼𝘀™ 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 to the 𝗝𝗲𝘁𝘀𝗼𝗻 𝗔𝗚𝗫 𝗢𝗿𝗶𝗻, along with the Transformer Engine, making both fully containerized with Docker for a true plug-and-play experience. 𝗖𝗼𝘀𝗺𝗼𝘀 is a groundbreaking platform of generative world foundation models (W𝗙𝗠𝘀), advanced tokenizers, and an accelerated data processing pipeline, purpose-built to advance Physical AI in autonomous vehicles and robotics.⁣

⁣

This work was recently showcased at CES 2025, where Cosmos took center stage as a transformative technology for developers and industries worldwide. With the port to Jetson AGX Orin, we’re unlocking the power of Cosmos and the 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿 𝗘𝗻𝗴𝗶𝗻𝗲 for edge applications, allowing developers to leverage its physics-based synthetic data generation, model fine-tuning capabilities, and highly efficient inference on compact, efficient systems.⁣

⁣

𝗣𝗼𝗿𝘁𝗶𝗻𝗴 𝗖𝗼𝘀𝗺𝗼𝘀 𝗮𝗻𝗱 𝘁𝗵𝗲 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿 𝗘𝗻𝗴𝗶𝗻𝗲 wasn’t just about integration—it’s about empowering developers to harness the future of AI-driven robotics and autonomous systems. With its modular, scalable design, Cosmos is a key enabler for innovation, helping the industry address challenges like data scarcity and variability through synthetic environments that are both photoreal and physics-based.⁣

⁣

Let’s shape the future of Physical AI together! Feel free to connect, collaborate, and share your insights on this exciting journey. 🚀⁣

⁣

#NVIDIA cosmos #NvidiaCosmos #TransformerEngine #PhysicalAI #GenerativeAI #JetsonAGXOrin robotics edgecomputing #AutonomousVehicles ai #Innovation #CES2025 docker #SyntheticData⁣

[kalustian]

Hello Johnny

Can you please share step by step how you install it on the AGX orin ( I have the 32Gb version on Jetpack 6.1 ) ?

NOTE: When following and running from below page., the process got “killed”

NVIDIA/Cosmos/blob/main/INSTALL.md

Thanks

[johnnynunez]

Using jetson containers and my docker:

Cosmos

https://hub.docker.com/r/johnnync/r36.4.0-cu126-cp310-cosmos

[kalustian]

@johnnynunez

Cosmos by johnnynunez · Pull Request #774 · dusty-nv/jetson-containers · GitHub

I am downloading the Cosmos from the Docker page now for testing. But I can not find it on the Dusty github portal. How I should pull it from the Dusty container page ?

Thanks

[johnnynunez]

because, he still is working in other promising things @dusty_nv

[kalustian]

ok , no problem. I finished downloading it from the Docker page. I am running it as sudo docker run johnnync/r36.4.0-cu126-cp310-cosmos, but it does not start the Cosmos container. During installation no errors found. Any suggestion to how properly run it ?

[johnnynunez]

jetson-containers run -it -v $(pwd):/workspace johnnync/r36.4.0-cu126-cp310-cosmos (my docker)

in pwd it is pointing to Cosmos clone, because if you download model inside docker, if it break, you lost the models… and the process to download them is very large.

also @shahizat replicates my process. Can you help him?

[shahizat]

Hello @kalustian

You can use this command also. I confirm that @johnnynunez’s container image works.

docker run --runtime nvidia -it --rm -v ./cosmos:/models --network=host johnnync/r36.4.0-cu126-cp310-cosmos:latest

[kalustian]

checking…

[kalustian]

Hi shahizat

I run Johnny’s and your command and I could be able to get into the container. Once in there I typed the following commands:

PROMPT=“A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves.
The robot’s metallic body gleams under the bright, even lighting, highlighting its futuristic design and intricate joints.
A glowing blue light emanates from its chest, adding a touch of advanced technology. The background is dominated by rows of boxes,
suggesting a highly organized storage system. The floor is lined with wooden pallets, enhancing the industrial setting.
The camera remains static, capturing the robot’s poised stance amidst the orderly environment, with a shallow depth of
field that keeps the focus on the robot while subtly blurring the background for a cinematic effect.”

PYTHONPATH=$(pwd) python cosmos1/models/diffusion/inference/text2world.py
–checkpoint_dir checkpoints
–diffusion_transformer_dir Cosmos-1.0-Diffusion-7B-Text2World
–prompt “$PROMPT”
–offload_prompt_upsampler
–video_save_name Cosmos-1.0-Diffusion-7B-Text2World

and I got this :

“Python not found”

[shahizat]

@kalustian please use this commands below, mount docker using -v option to download the models there:

Firstly, download the model:

PYTHONPATH=/opt/Cosmos python3 /opt/Cosmos/cosmos1/scripts/download_diffusion.py --model_sizes 7B 14B --model_types Text2World Video2World

Then run

PROMPT="The video is a dynamic and immersive driving experience captured from the perspective of a car's dashboard camera, likely mounted on the windshield. The setting is a narrow, two-lane road surrounded by lush greenery, suggesting a scenic route through a forested area. The road is marked with a single yellow line in the center, indicating a one-way traffic direction. The camera remains mostly static, providing a consistent view of the road ahead, while the car moves swiftly around a sharp curve to the right. The surroundings are dense with tall trees, and the road is flanked by a guardrail on the left side, which adds to the sense of speed and adventure. The weather appears overcast, with a misty atmosphere that enhances the feeling of being enveloped in nature. The car's speed is evident from the blurred background and the consistent motion of the road's edge. The video captures the thrill of driving through a picturesque landscape, emphasizing the connection between the driver and the natural environment. The camera's perspective remains focused on the road, with no visible pedestrians or other vehicles, creating an uninterrupted driving experience."

and finally run:

PYTHONPATH=$(pwd) python3 cosmos1/models/diffusion/inference/text2world.py  \
    --checkpoint_dir /models/checkpoints \
    --diffusion_transformer_dir Cosmos-1.0-Diffusion-7B-Text2World \
    --prompt "$PROMPT" \
    --video_save_name /models/New_Cosmos-1.0-Diffusion-7B-Text2World_memory_efficient \
    --offload_prompt_upsampler \
    --offload_tokenizer \
    --offload_diffusion_transformer

You can also join our discord channel: Jetson AI Lab Research Group Community

[kalustian]

Please allow me 10-15 min to test drive …will provide feedback soon

[johnnynunez]

change python to python3.
Also use all offload models.
With jetson thor will can execute every model on memory

[kalustian]

Environment:

• JetPack 6.1
• AGX orin 32Gb RAM
• SWP increased from 15Gb to 55Gb
• CPU/GPU clocked setup at max speed: 2.2Ghz / 1.3Ghz

Here are the steps I have taken (thanks to Johnny and Shahizat)

1)Run the Docker:
$sudo docker run --runtime nvidia -it --rm -v ./cosmos:/models --network=host johnnync/r36.4.0-cu126-cp310-cosmos:latest

2) Install and login in HuuginFace:
pip install -U “huggingface_hub[cli]”

3) Download the model:
PYTHONPATH=/opt/Cosmos python3 /opt/Cosmos/cosmos1/scripts/download_diffusion.py --model_sizes 7B --model_types Text2World Video2World

4) Add a Prompt

PROMPT=“A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves.
The robot’s metallic body gleams under the bright, even lighting, highlighting its futuristic design and intricate joints.
A glowing blue light emanates from its chest, adding a touch of advanced technology. The background is dominated by rows of boxes,
suggesting a highly organized storage system. The floor is lined with wooden pallets, enhancing the industrial setting.
The camera remains static, capturing the robot’s poised stance amidst the orderly environment, with a shallow depth of
field that keeps the focus on the robot while subtly blurring the background for a cinematic effect.”

or

PROMPT=“The video is a dynamic and immersive driving experience captured from the perspective of a car’s dashboard camera, likely mounted on the windshield. The setting is a narrow, two-lane road surrounded by lush greenery, suggesting a scenic route through a forested area. The road is marked with a single yellow line in the center, indicating a one-way traffic direction. The camera remains mostly static, providing a consistent view of the road ahead, while the car moves swiftly around a sharp curve to the right. The surroundings are dense with tall trees, and the road is flanked by a guardrail on the left side, which adds to the sense of speed and adventure. The weather appears overcast, with a misty atmosphere that enhances the feeling of being enveloped in nature. The car’s speed is evident from the blurred background and the consistent motion of the road’s edge. The video captures the thrill of driving through a picturesque landscape, emphasizing the connection between the driver and the natural environment. The camera’s perspective remains focused on the road, with no visible pedestrians or other vehicles, creating an uninterrupted driving experience.”

5) Run it:
PYTHONPATH=$(pwd) python3 cosmos1/models/diffusion/inference/text2world.py \
–checkpoint_dir checkpoints \
–diffusion_transformer_dir Cosmos-1.0-Diffusion-7B-Text2World
–prompt “$PROMPT” \
–video_save_name Cosmos-1.0-Diffusion-7B-Text2World_memory_efficient
–offload_tokenizer \
–offload_diffusion_transformer \
–offload_text_encoder_model \
–offload_prompt_upsampler \
–offload_guardrail_models

5) Success !!
After almost 3h @60 watts a 5 sec. video have been created. Special thanks to Johnny and Shahizat