Hi,
I’ve recently noticed some strange behavior of deepstream-app while streaming from IP camera using RTSP. It seems that after a longer period of time it stops receiving frames and FPS rate suddenly drops to 0.
Here is my full system configuration:
• Hardware Platform (Jetson / GPU) Jetson Nano
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) 4.4
• TensorRT Version 7.1.3
And the configuration file I’m using to feed deepstream-app:
[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0
[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://192.168.1.101/live/ch0
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device - Memory type Device
# (1): memtype_pinned - Memory type Host Pinned
# (2): memtype_unified - Memory type Unified
cudadec-memtype=0
rtsp-reconnect-interval-sec=60
[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0
overlay-id=1
[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0
[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1
# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=1
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt
[tracker]
enable=1
# For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=256
tracker-height=512
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
ll-config-file=human_tracker.yml
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1
[tests]
file-loop=0
This problem occurs after at least 12 hours of streaming. I haven’t noticed any memory leaks or unusual CPU/GPU usage at that time. Those are the logs of deepstream-app:
Jul 30 03:59:11 deepstream-app[5357]: 18:25:38.953437506 #033[336m 5357#033[00m 0x7edc089b70 #033[36mINFO #033[00m #033[00;01;37;41m GST_ELEMENT_PADS gstelement.c:920:gst_element_get_static_pad:#033[00m found pad rtpjitterbuffer0:src
Jul 30 03:59:13 deepstream-app[5357]: **PERF: 25.46 (24.93)
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.531549224 #033[336m 5357#033[00m 0x7edc089de0 #033[36mINFO #033[00m #033[00m rtpsource rtpsource.c:1155:update_receiver_stats:#033[00m duplicate or reordered packet (seqnr 8911, expected 8912)
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.532356724 #033[336m 5357#033[00m 0x555ac89770 #033[36mINFO #033[00m #033[00m h264parse gsth264parse.c:1793:gst_h264_parse_update_src_caps:<h264parse_elem0>#033[00m resolution changed 26x16
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.536322036 #033[336m 5357#033[00m 0x555ac89770 #033[36mINFO #033[00m #033[00;01;34m GST_EVENT gstevent.c:814:gst_event_new_caps:#033[00m creating caps event video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)01dcf002ffe2000b67640028ace80780227e5418706fdcf00231652df8cea80d7d503621929fb61a9e5f690034928214c94dcb78dbd8966d429b043c1fd01e7f6c7273b2fe5397eb44112e66bfc72239c19d8bf7914e58ae6d9612807875ba84aea06aa0bedbc9775bd5323d4b49aef9a3a760b95b3cb53baf3cb8558e370932bb53de38ca898c73432c9244bc7420e5d403eb89e7020a6d88fe7020b44c5047e852ce6d4d8b52a726bcafd601de587f248f3e63e37cf18aa2f3db8aa202c1d17f9eeb269286773d2db8868eea253a83277af756efb2bd71fd4cee3dcddb1ef4cbd78df573aa1ce1f73c62896a35355d53a8f1c7878e83a3a015f69e71a6e07d7f7855fd3c1973c9d257d17bb6d6b4c04fdab6b1be0977ce4461729b06c07eaf354483c351b0a5458b5b4f5ea6a63562c62378ea723c0a0d103556391f15ec06c85bc49628674a30b756f2e9d1e198af3354bd7a1af53d1413dd4263e3d9fb4338b144b0a4eab6df2eb1ab54c750266a75b481ca85357503a47cfe2806eb906e7ff0a2659ac8ec4e86b1ed6d066eb554a554ef813e414f1dc3abbd15e740698bf4c02b252aaaead3b034e835e41c3ee8d75e4d66a77a5b26cfb7a2c17ea52b6831c846e04c02555859f1462481848a189ee7fa09ec01cad53465f0f3bb66296c65d27c5ef576605d81aad7a86276c79c6b3fe67c7b223942aecfbb5522a18a98975e9cb612facfd44700867c7c2ae9774547ef3b397693bcf05f894a1188283ba443388c05eaae27d6b1536a67d2fc69ba8c1838c62544c66d48896c8a4febfd041f47736ebf5b67e9e29cd816ea2f7a71971a4b672df909828d90e50cbc2b3fc9547dba5957884b0fc461bc0fb1315f4ef9d98c5fea7831686a0a471dd9e4eff24825ec51bad09fe433ce8f193506e2a229ee3fd49349035b966949202927f599e0e4b8cff271ee24eb816af7a90588ddaf3e0849c7907149d099b8b7b1c9ee5e56cf38baad4eec8a708793b30f92a92082c3857c8f96b7fb6fe47096994f02c51357fc72183c32a3e2170e2431c30d4a3aa2c7e02a9cbf46f540f00dfbf9f901e028dbc416f2b3e74994261c42bac1842cb3fb0dca0ceeebe0b7b1f9a7f4bf2eb27e7feedba563785cc1c817ec5f816c3cdc2c09201713057d8e97f57869c2f855101d5d0c00e1122dea4cda8729d98a43533affd8075c2fc54cae415f9372a90987ac4ebdb0d03bf6f343ec319e59f1d66b4360effc87e746d5ff427f3cdbd20f14b77d8345909e4e0fdb2ba4f3385f82ea44a60216ac3d57ca3e4468bc49834edaa34301ef22d14a6bea058750f1fb204c33153f0e28c4b354514faeabcedffd7949bed94a980f91ccc6afb5145bdec6717989fd6a9a731d8973bce3a1815973ed1e001ab5f5e6bffa9d935ae665dd9afee2ba76133e4b3a1f5295079494023ae15dde33dfa1c268b926de95cf6356335377a942578584dcc303ef1d2b6282b8d6ec1ea853bcb3b582b4147bd9d15d0400374c733e363f447bf4d5f776dc52d26bce894637dc8c784e494485ce3058654121460543991c0be9cb78c17cbba05b7fedaa5b431db68c68d51983e7631d6c2228fdbdcb05247961f77249617b725363f06080f45176bfdad0ccc0241270f67c990f6c41369ee325c1608fc66a03e0b5ecc655320df1233108edbc967e8ca8043e247b2ebec3ee4011e5d84c6a5f4fb425e63b25a2e52ce7098dd8acd776ced6333ab5321749c0cb3beff9927ea87ee915579fe5edff02a317e8c27268902776e6acf266e40573685fbce8da5ca8ebbd4c3719bc67be02f16993ecc7b8aa61d44a75a8d2432f3ee99903231f46420396456d22ce4c89aee1fa11011f35f65e71b9061a5b17f603237ee9350adeb522e529628693fec765a70806072551cdfd32ae9eb33dcd8297d133c143e7b7473df8ee33237a05fde63711e5fd1726954479946ab7aa7c848dda1c882b18dbc1ab070d2061ec8a3bf62357e4169c2d504a8f1526ba535914be4c2fce4a1c032023156981c35cbf5d93fd6c18e9ca415d6b4502fd9563f70895f5c58e7781dc567b75ef1bfe7b7e9a4abe6a8583fc5cc8a3ce3fbf1d6b5a991cfc40e40ea6d0ccce3cb50f8bd718661ceb0ba3307596c275e0f3e1c0540fd6ae0db759d5ece21f79e78e7452a997a7ae8f459eba43fff47f0601439643c1aab62225fcbcb36e58d74b481a08adec3eb2e611b7d032cec64ffb32a71ac26f45f3971ddc7810178c5fb13c835daef0b291504118374cffc42354e1ce868e368bbef67da2745373493065a382745685cd37e4fce569e327935abfedad609fd110538b7f357c16ff590c34b820ecdeb4d94ae624a5869cae2ddefa4df16c6106055b2b8965551c27cc56093d3387d5f594f5ba5396cd5d4a6e14f88b1a01651af8f88b12b7b122c62ef3d8fc0549ef5b33833bdb819b4c4686f30ac586ba454ce81caf06242c06739e4578de3064425b02864b70a270abfb2300e2e83068c91b7c71f67bd60f730126c992c64cd98f4060b93848d9d8a4318b56259660f7f4570262eda558da2f376aee96c6089663bdab6f882bf733fe4b9cbe947580ea57eea7c25b9171d35b667a7d055d0fc2bbbb4bf99b52ac958dd981d12536d93ad21e10780cd27cf8711c68219e0db68d22d941f4019ec3a2dce1995b4747da8a973e82c445a7b2ee26691a883e7347760cf422e71716f363162f719acae244fbf3a39ac9c83863fbca008cbe36440be3f2c577ca7ff9e53945ec45d8a2722e35999f5de02ee45b1ea99d1d5dbef618780f109dd19d28f011a18389ab4fd97f442ea954866ff67844befecb84c42d7e6ffcb2ed66bcba89b372246a534da1bd9c1ef9301c0535a4a4f0407c44ae88e37d0a451abe2d5de202562f5490f11a123b7fdc17d5a0371f9cb77c1aa5599732827e19051489c419fca5768a8cf1c1ce0470d196677a6aa2936dd719de88423b6c68fe7591e6757716d7e799aea0a0fc2e240e7ca319f405011171d41c660966e3e48a96420c50a3572ba011165d879daa6f3b91cc871dd76c76003cfdbf2ffe25c4642d5291ea50c4cd6259a5688ffc39648957ff923a69428ed13bca6527168e7cdbfac6ba53074e61c495a56530147c2e26ad997d079f394363729ecd41f500146eee9cf23d0741259b8e9b62705ddbcdea7c133e1523e75fd56993ef3fb015e0fac5875217ec2ffbc6f443be9e7f75f6a8c7a724407bf12479d8f4d6b579652a8da7547156be3ec0271ab5999872baaa875d618f5a74587e0f7707f32cbf18a58538cb63da36e1ff3cea3ec84b5e4456a706c6dc4ae2e298172745560564e821b9586db606d4718106d81a023cb46913583adac2a790e2c5d46fa0ba9cb26caf1263f5a66d8eec9ab3b419653bba05f7814895296ac7f63b37b801545f0b78e00333e49d080885cbf8caf9b4fc04993d9db4494afc720956f1e56e85a7547505f8fce889962e4d119d96548e438e94596f7c661cef2abf8a1d43904dd551209adbaac117931f8d19dcd0115ba67b45d064a3d6a3e6e00087313bf63e13422e4a05427befb69244a8c21460784fd32db5dff97850538a9bd1093046665d548888c8f17b73c8c095a3ab6e94197994660384b7f937c6ebe1eb99b4e5787a7c746845326c37736b6233545915f37ca468693bae5e184b595c6d50764ec1a41bcaa32de48f5f8dda19c85cb594d19622eaf799eef5d83396967ac993f6b66921a5059f3c4d4bffaf7f71b195cb471f4558d47200c3cab58b7587fdbfaebf54f891adcb57183cdfff954fafbbd08837496055aab30eee89bccb411d4475a61e5d2806467d30b521c546600027b658677ce5a507a9541f2110180f0d76068ccb233ec6b8036b1b4f068a3b22280112af5549c494c90e312c79d6bab7055b13738360af2491a91c901aed16f54c753115b9349c4ba7f1d99b993a38970583b8f108256828789d50285ff9cf6047f4a644ef63f83f5cfcfb188bda5b8fce0fc07307a9ce2813097fe15d649fb088611936843758a57edf6987490d6bfd5a87dbc068ee330237164f9bae870c214b33349744cc21e7cdfb5f8b4e0c3c06d49b6e89e80f8e3af1b316ab47cd6af6ed0da30588bc8c69ba0ef28843e762190f3d9587232bdec85a518af427afd66c2033c4a083a86aef431e6d04591cbd33a75888375930f616a4f528e6c559a72048501f1eb8639755ebdaabf4885a810844920313362f1249e68d717ba2dbdf8443d04027254a46a3d0a47cb1da44b571a126c39bc600b9bc92b75449ffbd25b7b823ee5d1cdcfc023ab27645a120ea635e5d7a00990c2cbbc4aac8c3744c0dca4d4bd54e9cc723ebec7d037226e9f11604516db66264c9787fca683a743b2e9a5efdce3d1f889bffb6bf974595a3cd0064e3f3cb64e5df2008806fbb877fb77c451fa2ab381ad32c32b5b6fb507cf9c59c0f0b6ea65a329c7efffb4b58ec0e6ef982aebf96d858b8b221fb9a25943119c24e179810c4650c779b665a7d3e58be59bb2abc4745d13f9cb0170b7286c2f46412a90991feee1717bbf400f4e4d7ff9da08d3cf226aca3e49e77818915e6e0815bd159b229c216fa05f068da48a874c8755603a688751ba17ea16f2a99001ee980f99a27420f8ad84677c42af7e37a7f1c4c5ead6219ffadc11085217c69571db11c800f570a01a359d46920acc56c8befc51c1a154fe3b0bed8c6bb660afd1cd90ec2cffa9df619284043999461ab391c576f9cf968c684b59b773d749fef7fb5a81d9c744645762b78dffbcd00a086db07b6118af266ae11a4c84421684fa328576e84d289b78db842405e61d87aaf38d5a1bda1be25898e8962d82b74302be41d5fc1825e75e7eb412762a227d57062657572c4b6ac5537bf0c420447f1f7d819e005c4a8ed63c5da88910ae5ad5c6832cdbac9318fd4beaf88f4276759f021fe1e067f6ddad9e79c8daab95cf294aa09b8e0a3a028844a4a0829f7d1228c69745e3d927186ac26f2366c36bd21c266fcc89cf52d191ff6e73d938db39f22c585cf75a7009909df0c4105addcdebf95f1b74633c14246d9bc2f1d9e741d94ecfc2f46ba6e1ff322bbfc098f0599ee3069a42783103fd2f0ae8566d59ef8f69305f2762d8c0a2b4b8e0aeb28d80328b0316b561b4a70fd81bdd9c023c7a8b1f8202678fd6bb855774125bac18dfc6d1414243aad4157441582f4504383d11072bec2d294ff746f247b16f164d696cda26c67ad7712600745d53870d2db21fa852dee1152050aa37411ea628b4b9c57544c120d87ae591e9d4b81536572c1d4f25185a9604d9e43c0aae6be61bd567d7cc69e69dca63488f848741fe8514a40299dbb8bc5359dd4ce726622a543d6fbfc9279fa5a8f37a3cbf28a0bce7587bb3761296517e1af5d7a738bff029c29abacacad5637fded7038a94f75d8a8fc8ccbecb2fb33a142cb87c404e9bc4ade38dc41effecebd
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.536820161 #033[336m 5357#033[00m 0x555ac36c50 #033[33;01mWARN #033[00m #033[00m codecparsers_h264 gsth264parser.c:1904:gst_h264_parse_pps:#033[00m couldn't find associated sequence parameter set with id: 19
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.536868182 #033[336m 5357#033[00m 0x555ac36c50 #033[33;01mWARN #033[00m #033[00m h264parse gsth264parse.c:769:gst_h264_parse_process_nal:<h264parse0>#033[00m failed to parse PPS:
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.538423026 #033[336m 5357#033[00m 0x555ac36c50 #033[36mINFO #033[00m #033[00;01;34m GST_EVENT gstevent.c:814:gst_event_new_caps:#033[00m creating caps event video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)26, height=(int)16, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.538942505 #033[336m 5357#033[00m 0x555ac36c50 #033[33;01mWARN #033[00m #033[00m basetransform gstbasetransform.c:1355:gst_base_transform_setcaps:<capsfilter0>#033[00m transform could not transform video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)26, height=(int)16, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true in anything we support
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.539441151 #033[336m 5357#033[00m 0x555ac36c50 #033[33;01mWARN #033[00m #033[00m basetransform gstbasetransform.c:1355:gst_base_transform_setcaps:<capsfilter0>#033[00m transform could not transform video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4, profile=(string)high, width=(int)26, height=(int)16, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true in anything we support
Jul 30 03:59:14 deepstream-app[5357]: 18:25:41.539480578 #033[336m 5357#033[00m 0x555ac36c50 #033[36mINFO #033[00m #033[00m task gsttask.c:316:gst_task_func:<dec_que0:src>#033[00m Task going to paused
Jul 30 03:59:18 deepstream-app[5357]: **PERF: 23.96 (24.93)
Jul 30 03:59:18 deepstream-app[5357]: 18:25:46.065147607 #033[336m 5357#033[00m 0x7edc089b70 #033[36mINFO #033[00m #033[00;01;37;41m GST_ELEMENT_PADS gstelement.c:920:gst_element_get_static_pad:#033[00m found pad rtpjitterbuffer0:src
Jul 30 03:59:23 deepstream-app[5357]: **PERF: FPS 0 (Avg)
Jul 30 03:59:23 deepstream-app[5357]: **PERF: 0.00 (24.93)
Jul 30 03:59:25 deepstream-app[5357]: 18:25:53.075428334 #033[336m 5357#033[00m 0x7edc089b70 #033[36mINFO #033[00m #033[00;01;37;41m GST_ELEMENT_PADS gstelement.c:920:gst_element_get_static_pad:#033[00m found pad rtpjitterbuffer0:src
Jul 30 03:59:28 deepstream-app[5357]: **PERF: 0.00 (24.92)
Jul 30 03:59:32 deepstream-app[5357]: 18:26:00.104334321 #033[336m 5357#033[00m 0x7edc089b70 #033[36mINFO #033[00m #033[00;01;37;41m GST_ELEMENT_PADS gstelement.c:920:gst_element_get_static_pad:#033[00m found pad rtpjitterbuffer0:src
Jul 30 03:59:33 deepstream-app[5357]: **PERF: 0.00 (24.92)
I have no clue what might be the core of the problem here. I would appreciate any help.
Regards,
Bartek