I’m able to create an HEVC bitstream with alpha layer using the Video Codec SDK. What I cannot figure out is how to mux this bitstream into an MP4 file or HLS stream. FFmpeg, mp4box, and mp4mux all fail in different ways due to the alpha layer. This is fairly new territory and as far as I can tell only Apple encoders have end-to-end encode support for HEVC with alpha to MP4.
My question is this: How do I mux this HEVC bitstream - with alpha layer - into an MP4 that Quicktime can play?
Related question: How did the NVIDIA devs test their encoder?
Video Codec SDK reference: NVENC Video Encoder API Programming Guide :: NVIDIA Video Codec SDK Documentation
Feature request with FFmpeg: #9088 (HEVC bitstream with alpha layer to mp4) – FFmpeg
Code to create the bitstream: GitHub - johnhe4/nvenc_h265_transparency: Encodes a sequence of YUV frames into HEVC (h.265) with transparency, using Nvidia's nvEncodeApi