Terrible banding issues AO map conversion PNG to DDS

I know normally the questions here are of a different level but after another day of trying all kinds of combinations I thought to ask those that know their stuff.

I’m trying to convert a stacked RBG (AO/roughness/metallic) from PNG to DDS BC1 (no alpha involved). But as you see in the image I get terrible banding in the red channel. Which is, the Ambient Occlussion. The source image is created using Substance Painter which I normally use without any issues.

I tried the following:

  • Switching plugin, using Intel Texture Works
  • Used a script which uses ImageMagick
  • Switching computer
  • Changing from linear to sRGB
  • Using a Tiff as original format
  • Downgraded nVidia drivers back to older version

Sometimes it gets a bit less but still enough to notice in game while it’s ‘just’ an AO channel. I’m just without any clue at the moment, getting frustrating.

For those who like to try, the original PNG’s and the DDS which got generated using a script (these are FlightSim textures) are uploaded to WeTransfer at: Original Textures

Hope someone can kick me in the right direction for cause + solution.

Hi fizion! The short answer is that the BC1 compression format isn’t good at compressing textures where the channels are uncorrelated - in this case, the AO and roughness maps in the red and green channels (more details below). Does the flight sim support loading BC7 DDS textures? If so, encoding as BC7 should give much better quality (since BC7 handles uncorrelated channels better). If BC7 is unsupported, if there’s a way to provide the AO, roughness, and metallic maps as separate BC1 textures, that would improve quality. Worst case, the engine should hopefully support the uncompressed R8G8B8A8 or B8G8R8A8 formats, which will be large but have no banding.

What’s going on here is that the BC1 format is trying to compress the AO and roughness maps (and metallic, but that’s only present in one corner of the image) at the same time into a total of 4 bits per pixel. When the channels contain different sorts of content - in this case, the red channel has smooth gradients while the green channel has a rough crack pattern - the quality of each channel has to suffer a bit. (BC1 does well in cases where the channels contain similar content, though!)

Here’s a visual example - if I compress just the red/AO channel of the texture using BC1, the quality is relatively OK. There’s a bit of banding here because of BC1’s endpoint quantization, but it’s better than what we’ll see next.

Here’s what the texture looks like compressed with our highest-quality BC1 settings. It’s hard to see banding here, though, because the green channel hides the subtle differences there.

Here’s just the red channel of that - now you can see the banding from the screenshots:

In other words, BC1 does okay at compressing just the AO, but when it has to compress both AO and the crackly roughness texture, it divides its budget of 4 bits/pixel and introduces banding into the AO texture.

Here’s what I get if I compress the texture using BC7 (using the Normal quality setting) and read it back in:

And then just the red channel of that - the banding looks much better!

Finally, here’s another way of visualizing what’s going on. BC1 encodes a texture by dividing an image into 4x4 pixel blocks. Within each block, it plots the 16 pixel colors within an RGB color cube, and then tries to find a line that goes as close to those 16 colors as possible. (Image from Nathan Reed’s Understanding BCn Texture Compression Formats – Nathan Reed’s coding blog)

(There’s some additional details with 5.6.5 quantization on the endpoints and how the palette values for each block are interpolated, but they aren’t quite as relevant; check out the tooltip for the BC1 format in the Texture Tools Exporter for more on that though!.)

In most natural textures, the red, green, and blue channels within a block are correlated: even though they might have different gains and lifts, the 16 RGB colors often fall pretty close to some line. But when the red, green, and blue channels have texturally varying content, the 16 RGB colors might spread out over a plane or even the whole volume! In this case, there’s no longer a line that fits very well, and quality suffers.

BC7 has a whole host of tricks to improve quality when the channels are uncorrelated (and it can choose the tricks it uses per 4x4 pixel block)! In this case, it might decide to compress the R+B channels and the G channel separately - so that it’s able to express both smooth gradients and the roughness texture at the same time.

Hope this helps!

1 Like

Hi Neil!

Wow, thanks! That’s really an awesome explanation! I’ve sometimes a hard time understanding the more technical background but your story is very clear and that cube made it a lot easier to visualize it in my mind. I did read a lot about the 4bpp and DDS takes chunks of 4x4 but I never realized that it’s over all 3 channels. Makes a lot of sense why it occures.

So with your info I looked for the strange .FLAGS next to the original liveries and searching in the SDK of Microsoft Flight Simulator I came across this page:

That was exactly my problem but I was sure it was some lineair/converting thing I did wrong. Since that AO I used is actually the original liveries RED channel I was wondering how they got it into the file looking good. So I used ImageMagick to check the format and you were absolutely right, it’s a BC7 compressed file.

Happy it’s quite easy to fix, shame I was doubting my file exports/color profiles etc for 2 days. I wonder though why there is no ‘no-alpha’ version since it feels a bit like a waste of space. Guess there’s a reason and will Google tomorrow.

Thanks again, really appreciate the answer and you learned me something today, that’s what I like to do.

Kind regards,

Oscar Rottink

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.