We noticed something that does not happen with H264 encoding, but just with AV1, given everything else stays identical in our streaming setup to Chrome Browser.
Basically stream starts with PowerEfficientDecider=true and runs flawless, but on the first PLI, when the encoder is creating and sending a forceKeyFrame, the Chrome browser switches to PowerEfficientDecider=false, affecting the streaming performance. We want to understand why, as both encoders inherit the same code and we use insert-seq-hdr=true which is supposed to do exactly that, make a complete keyframe when needed.
Yes, I have applied that patch, this is how the force key frames are generated when PLI happens. Could this be the reason? Those key frames are not “complete”? Maybe this is why the HW decoder cannot resume and the SW one is used as a fallback?
Hi,
Please share more detail about the issue. By default we don’t enable hardware decoding in Chromium. Do you enable it manually? And do you use latest Jetpack 6.2.1?
I don’t think it needs to be enabled in Chrome, it just works by default if the system has a HW decoder for AV1, like all nvidia RTX 3xxx series. We used pt=47 for this and it works as the connection starts, problem is when a force Key Frame happens, chrome downgrades to the software AV1 decoder.
We are also testing with the following python code, with the pipeline above
encoder1 = pipe.get_by_name("encoder1")
encoder1.emit("force-IDR")
print("force-IDR event sent to nvidia encoder")
which should not depend on the automatic key frame generation in the linked topic.
When insert-seq-hdr=true is set, all keyframes generated are “complete” in the sense that the decoder should have all necessary data to restart the stream, or it might still miss some header, that is only sent by the encoder at the beginning of the streaming?
Can we compare the keyFrames in Wireshark to understand what is different and why the HW decoder fails on them?
We went further with our investigation and here are our findings up until this point:
the behaviour can be replicated on Windows with both Nvidia and ATI cards supporting AV1 HW decoder, both downgrade from D3D11 to software when forced keyframe is received.
the behaviour does NOT happen on Mac M3, stream stays on VideoToolBox all the time.
We also tried with the patch from Nvv4l2h264enc NACK / PLI / FIR event handling - #14 by DavidDDD . We noticed that the first forced Key Frame (that we manually send) breaks the HW encoder on Chrome and it fallbacks to the SW decoder and also generates a PLI (so a second keyFrame is generated automatically). Subsequent manual force-IDR signals are fine, as the SW decoder seems to be fine with them.
We found key differences in OBU Headers coming from forced IDR frames vs the initial frame and the auto frames being sent by property idrinterval=60, see diff image below.
Also, we noticed the Frame 0 has 3 OBU packets, while later forced keyframe 1 has only 2 OBU packets, missing the sequence header, which might be exactly what the HW decoder needs to restart the stream. Can we somehow enable this one too in the encoder?
Hi,
Please run $ sudo tegrastats and check whether NVDEC is on. We don’t enable hardware codec in Chromium on Jetson platforms. It is uncertain why you see it be enabled.
Sorry for the confusion, but we do NOT decode on jetson, we only ENCODE on jetson, stream AV1 over network via WebRTC and DECODE on another WIndows PC that runs Windows + has an NVIDIA RTX 3060 with HW AV1 decoder.
We are glad you were able to reproduce the issue. How can we follow this issue? Your bug tracker is only internal? We would love to get a test file and report back if the fix works. Thank you!
Can we expect a fix in the next months? I know it’s hard to give exact timeline, but a rough estimation would help us, since we plan to launch our product in January and depend on this critical fix for us. Thank you in advance!
We understand this time of year can get extremely busy with teams closing out tickets before the holidays. We also appreciate that NVIDIA is actively working on debugging this issue, especially since it’s affecting not only one of our clients but also other teams building AV1-based products on Jetson, such as @ademsah13.
Are there any updates on the status of the task?
If you’re able to share any progress, test results, or insights from the NVIDIA team, we can run parallel tests on our side and potentially contribute to narrowing down the root cause.
best regards,
Andrew
Embedded Software Engineer at ProventusNova
We can confirm the fix is working great! Will this be part of the next JetPack? Can you also share the technical details of the fix, what was wrong in the code?
Hi,
The fix is under review internally and shall be included in next Jetpack 6 release. The issue is in low-level software stack that when forceIDR is triggered, certain must-reset parameters are not reset, generating invalid IDR frames.
We managed to get just once (out of dozens of PLIs) back into the same situation with downgrade to software decoder.
Just wanted to make sure, during your review, maybe there are other parts of the code that can trigger the same invalid frame? Would appreciate you checking that internally.
