How to get clearly visible rain without repainting the scene with Cosmos?

Omniverse Synthetic Data Generation (SDG)
1

Cosmos Transfer 2.5: Keeping Isaac Sim materials vs adding visible rain (vis fixes drift, but rain becomes invisible)

Hi all — I’m using Cosmos Transfer 2.5 (Control2World / World Transfer) to create outdoor weather variations (light→visible rain) from an Isaac Sim scene for robotics training. I ran into two linked problems:

  1. Without appearance anchoring, the model “repaints” the world (water becomes asphalt, textures disappear, terrain changes).

  2. After fixing that drift by adding vis control, the scene identity is preserved… but now rain is basically invisible, even when I push the prompt.

I’d appreciate guidance on the right way to get clearly visible rain while preserving Isaac Sim materials.

Setup

  • Windows 11 + WSL2 (Ubuntu) + Docker Desktop

  • NVIDIA GPU: Blackwell Pro 6000

  • Cosmos Transfer2.5 running in Docker with /workspace mounted to repo and /data mounted to ~/cosmos_data

  • Inputs: Isaac Sim exported:

    • rgb.mp4

    • edges.mp4

    • depth.mp4

Important: my model expects a 93-frame chunk (state_t=24 → pixel_frames=(24−1)*4+1=93), so I run:

  • max_frames: 93

  • num_video_frames_per_chunk: 93

What I see (please refer to attached videos)

(Video 1) Baseline Isaac Sim render (no Cosmos)

➡️ Correct materials/textures: sea looks like sea, dock concrete texture correct, container materials/logos correct.

(Video 2) First Cosmos rain attempt (edge+depth only)
➡️ Rain appears, but major semantic/material drift:

  • Sea/water region turns into asphalt/ground

  • Carefully authored textures/logos get washed out / repainted

  • Background asphalt/ground turns into green grass / terrain change
    Basically the scene identity is lost.

(Video 3) Cosmos with vis control added (edge+depth+vis)

➡️ Scene identity is preserved (sea stays water, textures mostly preserved), but:

  • Rain is so subtle it’s effectively invisible

  • Minor issue: container text/logos get smeared, but the bigger issue is no visible raindrops

(Video 4) “More rain” attempt (higher guidance/steps, rain-heavy prompt)
➡️ Still no visible raindrops. Looks like the original scene with slightly different lighting at most.

Working configs (shortened)

A) Drift case (rain visible, but scene repainted)

{
  "prompt": "Realistic rainy weather... wet surfaces...",
  "negative_prompt": "...",
  "guidance": 3.0,
  "num_steps": 35,
  "max_frames": 93,
  "num_video_frames_per_chunk": 93,
  "edge":  { "control_path": ".../edges.mp4", "control_weight": 1.0 },
  "depth": { "control_path": ".../depth.mp4", "control_weight": 1.0 }
}

B) Identity preserved (but rain invisible)

{
  "prompt": "Apply a light rain overlay to the existing video. Preserve original textures/materials. Keep sea water as water. No terrain changes.",
  "negative_prompt": "grass, vegetation, terrain change, material change, repainting, water turning into ground, texture removal, ...",
  "guidance": 3.0,
  "num_steps": 30,
  "max_frames": 93,
  "num_video_frames_per_chunk": 93,
  "edge":  { "control_path": ".../edges.mp4", "control_weight": 0.9 },
  "depth": { "control_path": ".../depth.mp4", "control_weight": 0.6 },
  "vis":   { "control_weight": 0.7 }
}

C) “More rain” attempt (still invisible)

{
  "prompt": "Apply a clearly visible rain overlay... Add visible rain streaks/raindrops, splashes, ripples on water...",
  "negative_prompt": "terrain/material change, repainting, grass, ...",
  "guidance": 5.0,
  "num_steps": 45,
  "max_frames": 93,
  "num_video_frames_per_chunk": 93,
  "edge":  { "control_path": ".../edges.mp4", "control_weight": 0.9 },
  "depth": { "control_path": ".../depth.mp4", "control_weight": 0.6 },
  "vis":   { "control_weight": 0.6 }
}

My question to the community

What is the recommended way in Cosmos Transfer 2.5 to get clearly visible rain without repainting the scene?

Specifically:

  1. Control weight strategy:
    Is there a best-practice range for vis vs edge/depth when you want appearance preservation but also strong weather effects?

    • Should vis be lower (e.g. 0.3–0.5) and compensate with guidance/steps?

    • Does Cosmos normalize weights internally when sum > 1.0, and does that affect how strong the edit can be?

  2. Prompting strategy:
    Is there a known prompt pattern that produces visible raindrops (streaks, splashes) rather than only “wet look”?
    Should the prompt focus on “overlay” only, or are there recommended tokens / phrasing Cosmos responds to?

  3. Do I need segmentation/masks for water & ground?
    If rain requires more freedom (lower vis), is the right approach to use segmentation masks (freeze water region / freeze textures) so the model can add rain but not change materials?

  4. Is there a limitation here?
    Is Cosmos Transfer more tuned for lighting/wetness changes than explicit raindrop particles?
    If so, do you recommend adding visible rain as a post-process (RTX renderer / compositing) while using Cosmos only for lighting/wetness?

I can share full logs/configs if needed — but the key symptoms are:

  • Edge+depth only → rain appears, but world repaints (water→asphalt, textures→gone, grass appears)

  • Add vis → identity preserved, but rain becomes nearly invisible, even with stronger prompt/steps

Thanks in advance!

1 Like
2

Don’t take this as an authoritative answer, but I think this specific visual transformation is going to be very difficult to get while also keeping the scene identity. Cosmos Transfer 2.5 to the best of my knowledge isn’t a video editing model, meaning that the prompts and control modalities can help transform a scene to another look, and the control modalities maintain the scene structure. However, in this specific case you’re looking to make a very refined edit to a scene that maintains it visual identity.

I’ll ask around if this is a correct take.

3

thank you!

This is something to think about, for sure.
See, I got an answer from GitHub for the same question I posted here:

So as we can see, the solution I got worked out!

I also added a question about the overall correct usage of Cosmos World Transfer for this kinda case..