This is a forked Windows Installation Tutorial and the main codes will not be updated This forked GitHub project is intented for folks who want to install and generate videos with Cosmos in a Windows native machine. You can also follow the guide and skip the WSL references to run it natively on Linux. As stated above, at the time you are using this forked project, the original Cosmos project may have been updated. Refer to the original project page if parts of this guide are not working.
The section below is from the original GitHub page. Jump down to Installation to get started.
NVIDIA Cosmos is a developer-first world foundation model platform designed to help Physical AI developers build their Physical AI systems better and faster. Cosmos contains
- pre-trained models, available via Hugging Face under the NVIDIA Open Model License that allows commercial use of the models for free
- training scripts under the Apache 2 License, offered through NVIDIA Nemo Framework for post-training the models for various downstream Physical AI applications
Details of the platform is described in the Cosmos paper. Preview access is avaiable at build.nvidia.com.
- Pre-trained Diffusion-based world foundation models for Text2World and Video2World generation where a user can generate visual simulation based on text prompts and video prompts.
- Pre-trained Autoregressive-based world foundation models for Video2World generation where a user can generate visual simulation based on video prompts and optional text prompts.
- Video tokenizers for tokenizing videos into continuous tokens (latent vectors) and discrete tokens (integers) efficiently and effectively.
- Video curation pipeline for building your own video dataset. [Coming soon]
- Post-training scripts via NeMo Framework to post-train the pre-trained world foundation models for various Physical AI setup.
- Pre-training scripts via NeMo Framework for building your own world foundation model. [Diffusion] [Autoregressive] [Tokenizer].
| Model name | Description | Try it out |
|---|---|---|
| Cosmos-1.0-Diffusion-7B-Text2World | Text to visual world generation | Inference |
| Cosmos-1.0-Diffusion-14B-Text2World | Text to visual world generation | Inference |
| Cosmos-1.0-Diffusion-7B-Video2World | Video + Text based future visual world generation | Inference |
| Cosmos-1.0-Diffusion-14B-Video2World | Video + Text based future visual world generation | Inference |
| Cosmos-1.0-Autoregressive-4B | Future visual world generation | Inference |
| Cosmos-1.0-Autoregressive-12B | Future visual world generation | Inference |
| Cosmos-1.0-Autoregressive-5B-Video2World | Video + Text based future visual world generation | Inference |
| Cosmos-1.0-Autoregressive-13B-Video2World | Video + Text based future visual world generation | Inference |
| Cosmos-1.0-Guardrail | Guardrail contains pre-Guard and post-Guard for safe use | Embedded in model inference scripts |
The original Cosmos project was intended to be run locally or deployed on a server via a Linux operation system. This guide walks you through the installation of Cosmos on Windows via WSL2.
Cosmos requires a recent GPU with at minimum 24 GB of VRAM. To run the 16B models, a minumum of 48 GB of VRAM is needed.
Supported Hardware Microarchitecture Compatibility:
- NVIDIA Blackwell
- NVIDIA Hopper
- NVIDIA Ampere
Minimum hardware configuration for 7B: NVIDIA RTX 3090. Recommended 4090 or better
Minimum hardware configuration for 14B: NVIDIA RTX A6000. Recommend RTX 6000 Ada or better
Follow the Cosmos Installation Guide for Windows via WSL2 to setup the docker.
For inference with the pretrained models, please refer to Cosmos Diffusion Inference and Cosmos Autoregressive Inference.
The code snippet below provides a gist of the inference usage.
PROMPT="A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves. \
The robot's metallic body gleams under the bright, even lighting, highlighting its futuristic design and intricate joints. \
A glowing blue light emanates from its chest, adding a touch of advanced technology. The background is dominated by rows of boxes, \
suggesting a highly organized storage system. The floor is lined with wooden pallets, enhancing the industrial setting. \
The camera remains static, capturing the robot's poised stance amidst the orderly environment, with a shallow depth of \
field that keeps the focus on the robot while subtly blurring the background for a cinematic effect."
# Example using 7B model
PYTHONPATH=$(pwd) python cosmos1/models/diffusion/inference/text2world.py \
--checkpoint_dir checkpoints \
--diffusion_transformer_dir Cosmos-1.0-Diffusion-7B-Text2World \
--prompt "$PROMPT" \
--offload_prompt_upsampler \
--video_save_name Cosmos-1.0-Diffusion-7B-Text2Worldtext2world_example.mp4
We also offer multi-GPU inference support for Diffusion Text2World WFM models through NeMo Framework.
NeMo Framework provides GPU accelerated post-training with general post-training for both diffusion and autoregressive models, with other types of post-training coming soon.
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
NVIDIA Cosmos source code is released under the Apache 2 License.
NVIDIA Cosmos models are released under the NVIDIA Open Model License. For a custom license, please contact [email protected].