HEVC with alpha layer

I’m able to create an HEVC bitstream with alpha layer using the Video Codec SDK. What I cannot figure out is how to mux this bitstream into an MP4 file or HLS stream. FFmpeg, mp4box, and mp4mux all fail in different ways due to the alpha layer. This is fairly new territory and as far as I can tell only Apple encoders have end-to-end encode support for HEVC with alpha to MP4.

My question is this: How do I mux this HEVC bitstream - with alpha layer - into an MP4 that Quicktime can play?

Related question: How did the NVIDIA devs test their encoder?

Video Codec SDK reference: NVENC Video Encoder API Programming Guide :: NVIDIA Video Codec SDK Documentation
Feature request with FFmpeg: #9088 (HEVC bitstream with alpha layer to mp4) – FFmpeg
Code to create the bitstream: GitHub - johnhe4/nvenc_h265_transparency: Encodes a sequence of YUV frames into HEVC (h.265) with transparency, using Nvidia's nvEncodeApi

1 Like

There is now Apple encoder available.

mp4box, and mp4mux all fail

How? Can you paste their problem here.

this HEVC bitstream - with alpha layer - into an MP4

This is called Annex B. Frankly speacking you will have to write you own code, IMHO, according to Apple spec. See how it was done above.

Thanks for the update, it prompted me to update the FFmpeg ticket with more details.

I don’t think this will help because I need muxing-only support in FFmpeg, and I will be encoding on Linux, but it is good to see activity on this.

I don’t remember the exact issues with mp4box and mp4mux but in both cases they didn’t correctly handle the presence of the alpha layer when muxing to an MP4. You can see the commands I used at the end of the github readme which is linked above in the original ticket.

Hey John, did you get anywhere on this?

I need Hevc + Alpha in my game as well, but don’t want to have to buy or borrow an Apple device just to create MP4s with alpha embedded properly, and I don’t have the time / energy to implement this myself. It’s a bit unfortunate to still have to use separate video streams just for alpha (encoded as the luminance channel for full res even via 420, to conserve bandwidth).

Also, curious what media players, if any, currently support playing back HEVC + Alpha fully, either on Linux or Windows.

Nothing changed since then.

I was told by an Nvidia engineer on github that they would be working on supporting HEVC + alpha decoding via Vulkan Video, but no ETA was given:

It would seem weird to me to only support this format on playback and not encoding, to “close the loop” and be able to actually generate content for it without using an Apple device. But I will if I have to (once Vk Video supports yuva420p HEVC pixel format). As a fallback, I guess I’ll have to simply use multi-layer HEVC streams with separate layers for alpha (using normal yuv420p and feeding it grayscale RGB inputs so the chroma value is always neutral grey).

alpha is such a useful thing to have in gaming, it’s really too bad to have to use a separate hardware decoding unit just for that. oh well.