Video Codec: Calling cuvidParseVideoData not triggering callbacks for a specific RTSP source

I have an application that works on files and rtsp based on the sample provided.

I recently came across an RTSP stream where calling cuvidParseVideoData does not trigger callbacks.
What can I do to find out what is happening? I tried ulErrorThreshold = 100. No luck.

VLC works on this source. It is H264.

Turns out I have to manually inject the “extradata” from AvCodecContext with