Deepstream-app stops receiving frames after longer streaming time

Hi,

I’ve recently noticed some strange behavior of deepstream-app while streaming from IP camera using RTSP. It seems that after a longer period of time it stops receiving frames and FPS rate suddenly drops to 0.

Here is my full system configuration:
• Hardware Platform (Jetson / GPU) Jetson Nano
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) 4.4
• TensorRT Version 7.1.3

And the configuration file I’m using to feed deepstream-app:

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://192.168.1.101/live/ch0
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
rtsp-reconnect-interval-sec=60

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0
overlay-id=1

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=1
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tracker]
enable=1
# For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=256
tracker-height=512
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
ll-config-file=human_tracker.yml
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1

[tests]
file-loop=0

This problem occurs after at least 12 hours of streaming. I haven’t noticed any memory leaks or unusual CPU/GPU usage at that time. Those are the logs of deepstream-app:

Jul 30 03:59:11 deepstream-app[5357]: 18:25:38.953437506 #033[336m 5357#033[00m   0x7edc089b70 #033[36mINFO   #033[00m #033[00;01;37;41m    GST_ELEMENT_PADS gstelement.c:920:gst_element_get_static_pad:#033[00m found pad rtpjitterbuffer0:src
Jul 30 03:59:13 deepstream-app[5357]: **PERF: 25.46 (24.93)
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.531549224 #033[336m 5357#033[00m   0x7edc089de0 #033[36mINFO   #033[00m #033[00m           rtpsource rtpsource.c:1155:update_receiver_stats:#033[00m duplicate or reordered packet (seqnr 8911, expected 8912)
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.532356724 #033[336m 5357#033[00m   0x555ac89770 #033[36mINFO   #033[00m #033[00m           h264parse gsth264parse.c:1793:gst_h264_parse_update_src_caps:<h264parse_elem0>#033[00m resolution changed 26x16
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.536322036 #033[336m 5357#033[00m   0x555ac89770 #033[36mINFO   #033[00m #033[00;01;34m           GST_EVENT gstevent.c:814:gst_event_new_caps:#033[00m creating caps event video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01dcf002ffe2000b67640028ace80780227e5418706fdcf00231652df8cea80d7d503621929fb61a9e5f690034928214c94dcb78dbd8966d429b043c1fd01e7f6c7273b2fe5397eb44112e66bfc72239c19d8bf7914e58ae6d9612807875ba84aea06aa0bedbc9775bd5323d4b49aef9a3a760b95b3cb53baf3cb8558e370932bb53de38ca898c73432c9244bc7420e5d403eb89e7020a6d88fe7020b44c5047e852ce6d4d8b52a726bcafd601de587f248f3e63e37cf18aa2f3db8aa202c1d17f9eeb269286773d2db8868eea253a83277af756efb2bd71fd4cee3dcddb1ef4cbd78df573aa1ce1f73c62896a35355d53a8f1c7878e83a3a015f69e71a6e07d7f7855fd3c1973c9d257d17bb6d6b4c04fdab6b1be0977ce4461729b06c07eaf354483c351b0a5458b5b4f5ea6a63562c62378ea723c0a0d103556391f15ec06c85bc49628674a30b756f2e9d1e198af3354bd7a1af53d1413dd4263e3d9fb4338b144b0a4eab6df2eb1ab54c750266a75b481ca85357503a47cfe2806eb906e7ff0a2659ac8ec4e86b1ed6d066eb554a554ef813e414f1dc3abbd15e740698bf4c02b252aaaead3b034e835e41c3ee8d75e4d66a77a5b26cfb7a2c17ea52b6831c846e04c02555859f1462481848a189ee7fa09ec01cad53465f0f3bb66296c65d27c5ef576605d81aad7a86276c79c6b3fe67c7b223942aecfbb5522a18a98975e9cb612facfd44700867c7c2ae9774547ef3b397693bcf05f894a1188283ba443388c05eaae27d6b1536a67d2fc69ba8c1838c62544c66d48896c8a4febfd041f47736ebf5b67e9e29cd816ea2f7a71971a4b672df909828d90e50cbc2b3fc9547dba5957884b0fc461bc0fb1315f4ef9d98c5fea7831686a0a471dd9e4eff24825ec51bad09fe433ce8f193506e2a229ee3fd49349035b966949202927f599e0e4b8cff271ee24eb816af7a90588ddaf3e0849c7907149d099b8b7b1c9ee5e56cf38baad4eec8a708793b30f92a92082c3857c8f96b7fb6fe47096994f02c51357fc72183c32a3e2170e2431c30d4a3aa2c7e02a9cbf46f540f00dfbf9f901e028dbc416f2b3e74994261c42bac1842cb3fb0dca0ceeebe0b7b1f9a7f4bf2eb27e7feedba563785cc1c817ec5f816c3cdc2c09201713057d8e97f57869c2f855101d5d0c00e1122dea4cda8729d98a43533affd8075c2fc54cae415f9372a90987ac4ebdb0d03bf6f343ec319e59f1d66b4360effc87e746d5ff427f3cdbd20f14b77d8345909e4e0fdb2ba4f3385f82ea44a60216ac3d57ca3e4468bc49834edaa34301ef22d14a6bea058750f1fb204c33153f0e28c4b354514faeabcedffd7949bed94a980f91ccc6afb5145bdec6717989fd6a9a731d8973bce3a1815973ed1e001ab5f5e6bffa9d935ae665dd9afee2ba76133e4b3a1f5295079494023ae15dde33dfa1c268b926de95cf6356335377a942578584dcc303ef1d2b6282b8d6ec1ea853bcb3b582b4147bd9d15d0400374c733e363f447bf4d5f776dc52d26bce894637dc8c784e494485ce3058654121460543991c0be9cb78c17cbba05b7fedaa5b431db68c68d51983e7631d6c2228fdbdcb05247961f77249617b725363f06080f45176bfdad0ccc0241270f67c990f6c41369ee325c1608fc66a03e0b5ecc655320df1233108edbc967e8ca8043e247b2ebec3ee4011e5d84c6a5f4fb425e63b25a2e52ce7098dd8acd776ced6333ab5321749c0cb3beff9927ea87ee915579fe5edff02a317e8c27268902776e6acf266e40573685fbce8da5ca8ebbd4c3719bc67be02f16993ecc7b8aa61d44a75a8d2432f3ee99903231f46420396456d22ce4c89aee1fa11011f35f65e71b9061a5b17f603237ee9350adeb522e529628693fec765a70806072551cdfd32ae9eb33dcd8297d133c143e7b7473df8ee33237a05fde63711e5fd1726954479946ab7aa7c848dda1c882b18dbc1ab070d2061ec8a3bf62357e4169c2d504a8f1526ba535914be4c2fce4a1c032023156981c35cbf5d93fd6c18e9ca415d6b4502fd9563f70895f5c58e7781dc567b75ef1bfe7b7e9a4abe6a8583fc5cc8a3ce3fbf1d6b5a991cfc40e40ea6d0ccce3cb50f8bd718661ceb0ba3307596c275e0f3e1c0540fd6ae0db759d5ece21f79e78e7452a997a7ae8f459eba43fff47f0601439643c1aab62225fcbcb36e58d74b481a08adec3eb2e611b7d032cec64ffb32a71ac26f45f3971ddc7810178c5fb13c835daef0b291504118374cffc42354e1ce868e368bbef67da2745373493065a382745685cd37e4fce569e327935abfedad609fd110538b7f357c16ff590c34b820ecdeb4d94ae624a5869cae2ddefa4df16c6106055b2b8965551c27cc56093d3387d5f594f5ba5396cd5d4a6e14f88b1a01651af8f88b12b7b122c62ef3d8fc0549ef5b33833bdb819b4c4686f30ac586ba454ce81caf06242c06739e4578de3064425b02864b70a270abfb2300e2e83068c91b7c71f67bd60f730126c992c64cd98f4060b93848d9d8a4318b56259660f7f4570262eda558da2f376aee96c6089663bdab6f882bf733fe4b9cbe947580ea57eea7c25b9171d35b667a7d055d0fc2bbbb4bf99b52ac958dd981d12536d93ad21e10780cd27cf8711c68219e0db68d22d941f4019ec3a2dce1995b4747da8a973e82c445a7b2ee26691a883e7347760cf422e71716f363162f719acae244fbf3a39ac9c83863fbca008cbe36440be3f2c577ca7ff9e53945ec45d8a2722e35999f5de02ee45b1ea99d1d5dbef618780f109dd19d28f011a18389ab4fd97f442ea954866ff67844befecb84c42d7e6ffcb2ed66bcba89b372246a534da1bd9c1ef9301c0535a4a4f0407c44ae88e37d0a451abe2d5de202562f5490f11a123b7fdc17d5a0371f9cb77c1aa5599732827e19051489c419fca5768a8cf1c1ce0470d196677a6aa2936dd719de88423b6c68fe7591e6757716d7e799aea0a0fc2e240e7ca319f405011171d41c660966e3e48a96420c50a3572ba011165d879daa6f3b91cc871dd76c76003cfdbf2ffe25c4642d5291ea50c4cd6259a5688ffc39648957ff923a69428ed13bca6527168e7cdbfac6ba53074e61c495a56530147c2e26ad997d079f394363729ecd41f500146eee9cf23d0741259b8e9b62705ddbcdea7c133e1523e75fd56993ef3fb015e0fac5875217ec2ffbc6f443be9e7f75f6a8c7a724407bf12479d8f4d6b579652a8da7547156be3ec0271ab5999872baaa875d618f5a74587e0f7707f32cbf18a58538cb63da36e1ff3cea3ec84b5e4456a706c6dc4ae2e298172745560564e821b9586db606d4718106d81a023cb46913583adac2a790e2c5d46fa0ba9cb26caf1263f5a66d8eec9ab3b419653bba05f7814895296ac7f63b37b801545f0b78e00333e49d080885cbf8caf9b4fc04993d9db4494afc720956f1e56e85a7547505f8fce889962e4d119d96548e438e94596f7c661cef2abf8a1d43904dd551209adbaac117931f8d19dcd0115ba67b45d064a3d6a3e6e00087313bf63e13422e4a05427befb69244a8c21460784fd32db5dff97850538a9bd1093046665d548888c8f17b73c8c095a3ab6e94197994660384b7f937c6ebe1eb99b4e5787a7c746845326c37736b6233545915f37ca468693bae5e184b595c6d50764ec1a41bcaa32de48f5f8dda19c85cb594d19622eaf799eef5d83396967ac993f6b66921a5059f3c4d4bffaf7f71b195cb471f4558d47200c3cab58b7587fdbfaebf54f891adcb57183cdfff954fafbbd08837496055aab30eee89bccb411d4475a61e5d2806467d30b521c546600027b658677ce5a507a9541f2110180f0d76068ccb233ec6b8036b1b4f068a3b22280112af5549c494c90e312c79d6bab7055b13738360af2491a91c901aed16f54c753115b9349c4ba7f1d99b993a38970583b8f108256828789d50285ff9cf6047f4a644ef63f83f5cfcfb188bda5b8fce0fc07307a9ce2813097fe15d649fb088611936843758a57edf6987490d6bfd5a87dbc068ee330237164f9bae870c214b33349744cc21e7cdfb5f8b4e0c3c06d49b6e89e80f8e3af1b316ab47cd6af6ed0da30588bc8c69ba0ef28843e762190f3d9587232bdec85a518af427afd66c2033c4a083a86aef431e6d04591cbd33a75888375930f616a4f528e6c559a72048501f1eb8639755ebdaabf4885a810844920313362f1249e68d717ba2dbdf8443d04027254a46a3d0a47cb1da44b571a126c39bc600b9bc92b75449ffbd25b7b823ee5d1cdcfc023ab27645a120ea635e5d7a00990c2cbbc4aac8c3744c0dca4d4bd54e9cc723ebec7d037226e9f11604516db66264c9787fca683a743b2e9a5efdce3d1f889bffb6bf974595a3cd0064e3f3cb64e5df2008806fbb877fb77c451fa2ab381ad32c32b5b6fb507cf9c59c0f0b6ea65a329c7efffb4b58ec0e6ef982aebf96d858b8b221fb9a25943119c24e179810c4650c779b665a7d3e58be59bb2abc4745d13f9cb0170b7286c2f46412a90991feee1717bbf400f4e4d7ff9da08d3cf226aca3e49e77818915e6e0815bd159b229c216fa05f068da48a874c8755603a688751ba17ea16f2a99001ee980f99a27420f8ad84677c42af7e37a7f1c4c5ead6219ffadc11085217c69571db11c800f570a01a359d46920acc56c8befc51c1a154fe3b0bed8c6bb660afd1cd90ec2cffa9df619284043999461ab391c576f9cf968c684b59b773d749fef7fb5a81d9c744645762b78dffbcd00a086db07b6118af266ae11a4c84421684fa328576e84d289b78db842405e61d87aaf38d5a1bda1be25898e8962d82b74302be41d5fc1825e75e7eb412762a227d57062657572c4b6ac5537bf0c420447f1f7d819e005c4a8ed63c5da88910ae5ad5c6832cdbac9318fd4beaf88f4276759f021fe1e067f6ddad9e79c8daab95cf294aa09b8e0a3a028844a4a0829f7d1228c69745e3d927186ac26f2366c36bd21c266fcc89cf52d191ff6e73d938db39f22c585cf75a7009909df0c4105addcdebf95f1b74633c14246d9bc2f1d9e741d94ecfc2f46ba6e1ff322bbfc098f0599ee3069a42783103fd2f0ae8566d59ef8f69305f2762d8c0a2b4b8e0aeb28d80328b0316b561b4a70fd81bdd9c023c7a8b1f8202678fd6bb855774125bac18dfc6d1414243aad4157441582f4504383d11072bec2d294ff746f247b16f164d696cda26c67ad7712600745d53870d2db21fa852dee1152050aa37411ea628b4b9c57544c120d87ae591e9d4b81536572c1d4f25185a9604d9e43c0aae6be61bd567d7cc69e69dca63488f848741fe8514a40299dbb8bc5359dd4ce726622a543d6fbfc9279fa5a8f37a3cbf28a0bce7587bb3761296517e1af5d7a738bff029c29abacacad5637fded7038a94f75d8a8fc8ccbecb2fb33a142cb87c404e9bc4ade38dc41effecebd
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.536820161 #033[336m 5357#033[00m   0x555ac36c50 #033[33;01mWARN   #033[00m #033[00m   codecparsers_h264 gsth264parser.c:1904:gst_h264_parse_pps:#033[00m couldn't find associated sequence parameter set with id: 19
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.536868182 #033[336m 5357#033[00m   0x555ac36c50 #033[33;01mWARN   #033[00m #033[00m           h264parse gsth264parse.c:769:gst_h264_parse_process_nal:<h264parse0>#033[00m failed to parse PPS:
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.538423026 #033[336m 5357#033[00m   0x555ac36c50 #033[36mINFO   #033[00m #033[00;01;34m           GST_EVENT gstevent.c:814:gst_event_new_caps:#033[00m creating caps event video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)26, height=(int)16, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.538942505 #033[336m 5357#033[00m   0x555ac36c50 #033[33;01mWARN   #033[00m #033[00m       basetransform gstbasetransform.c:1355:gst_base_transform_setcaps:<capsfilter0>#033[00m transform could not transform video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)26, height=(int)16, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true in anything we support
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.539441151 #033[336m 5357#033[00m   0x555ac36c50 #033[33;01mWARN   #033[00m #033[00m       basetransform gstbasetransform.c:1355:gst_base_transform_setcaps:<capsfilter0>#033[00m transform could not transform video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)26, height=(int)16, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true in anything we support
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.539480578 #033[336m 5357#033[00m   0x555ac36c50 #033[36mINFO   #033[00m #033[00m                task gsttask.c:316:gst_task_func:<dec_que0:src>#033[00m Task going to paused
Jul 30 03:59:18 deepstream-app[5357]: **PERF: 23.96 (24.93)
Jul 30 03:59:18 deepstream-app[5357]: 18:25:46.065147607 #033[336m 5357#033[00m   0x7edc089b70 #033[36mINFO   #033[00m #033[00;01;37;41m    GST_ELEMENT_PADS gstelement.c:920:gst_element_get_static_pad:#033[00m found pad rtpjitterbuffer0:src
Jul 30 03:59:23 deepstream-app[5357]: **PERF: FPS 0 (Avg)
Jul 30 03:59:23 deepstream-app[5357]: **PERF: 0.00 (24.93)
Jul 30 03:59:25 deepstream-app[5357]: 18:25:53.075428334 #033[336m 5357#033[00m   0x7edc089b70 #033[36mINFO   #033[00m #033[00;01;37;41m    GST_ELEMENT_PADS gstelement.c:920:gst_element_get_static_pad:#033[00m found pad rtpjitterbuffer0:src
Jul 30 03:59:28 deepstream-app[5357]: **PERF: 0.00 (24.92)
Jul 30 03:59:32 deepstream-app[5357]: 18:26:00.104334321 #033[336m 5357#033[00m   0x7edc089b70 #033[36mINFO   #033[00m #033[00;01;37;41m    GST_ELEMENT_PADS gstelement.c:920:gst_element_get_static_pad:#033[00m found pad rtpjitterbuffer0:src
Jul 30 03:59:33 deepstream-app[5357]: **PERF: 0.00 (24.92)

I have no clue what might be the core of the problem here. I would appreciate any help.

Regards,
Bartek

Even I am facing this issue!!

• Hardware Platform (Jetson / GPU) Jetson Xavier
• DeepStream Version 5.0

After restarting it starts working again

Yes, exactly, restarting the app does the trick. However, setting rtsp-reconnect-interval-sec parameter doesn’t help at all.

Any ideas? Maybe someone from NVIDIA support?

@bartosz From the log, seems the rtsp server have send a resolution change request to deepstream, but DP5.0 does not support it. So the stream is paused.

Can you try to disable the resolution change request in the RTSP server side?

Another solution may be to update to our latest Deepstream5.0 GA. It may have the resolution change support in client side. It has been published and you can get it here. https://developer.nvidia.com/deepstream-getting-started

I’m not quite sure if disabling resolution change is an option in my scenario as I’m streaming directly from IP camera and I’m not aware of such parameter there.

However, I’ll try updating Deepstream. For the time being I’ve just implemented a workaround, which restarts the source every time the fps rate drops below some threshold. Anyway, thanks for advice, I’ll try it.