Decode problem encoded H264 CUDA stream, jumping frame

On the server side, we encode the video using NVENC into an rtsp stream. When we try to decode this stream into vlc, we get jumping frames.
Sample initialization code:

NV_ENC_INITIALIZE_PARAMS initializeParams = { NV_ENC_INITIALIZE_PARAMS_VER };
NV_ENC_CONFIG encodeConfig = { NV_ENC_CONFIG_VER };
initializeParams.encodeConfig = &encodeConfig;
encoder()->CreateDefaultEncoderParams( &initializeParams, NV_ENC_CODEC_H264_GUID, NV_ENC_PRESET_P1_GUID, NV_ENC_TUNING_INFO_ULTRA_LOW_LATENCY );
initializeParams.frameRateNum = 10;
initializeParams.frameRateDen = 1; initializeParams.encodeConfig->encodeCodecConfig.h264Config.hierarchicalBFrames = 0;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.useBFramesAsRef = NV_ENC_BFRAME_REF_MODE_DISABLED;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.bdirectMode = NV_ENC_H264_BDIRECT_MODE_DISABLE;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.disableSVCPrefixNalu = 1;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.enableTemporalSVC = 0;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.disableSVCPrefixNalu = 1;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.level = NV_ENC_LEVEL_AUTOSELECT;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.idrPeriod = 30; initializeParams.encodeConfig->encodeCodecConfig.h264Config.chromaFormatIDC = 1;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.disableDeblockingFilterIDC = 0;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.disableSPSPPS = 0;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.sliceMode = 3;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.sliceModeData = 1;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.maxNumRefFrames = 1;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.bdirectMode = NV_ENC_H264_BDIRECT_MODE_AUTOSELECT;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.adaptiveTransformMode = NV_ENC_H264_ADAPTIVE_TRANSFORM_AUTOSELECT;
initializeParams.encodeConfig->encodeCodecConfig.h264Config.entropyCodingMode = NV_ENC_H264_ENTROPY_CODING_MODE_CABAC; initializeParams.enablePTD = 1;
initializeParams.enableSubFrameWrite = 0;
initializeParams.enableEncodeAsync = 0;
initializeParams.encodeConfig->rcParams.rateControlMode = NV_ENC_PARAMS_RC_MODE::NV_ENC_PARAMS_RC_CBR;

Sample video recorded by ffmpeg:

Tell me what could be the problem?