Stack smashing detected

edit_or · July 12, 2024, 3:31am

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU)**rtx3050
• DeepStream Version6.4
I am using deepsteram-app for peoplenet.
I can interface to 17 cctvs successfully.
When increased to one more camera and altogether 18, I have this error.

*** stack smashing detected ***: terminated
Aborted (core dumped)

The whole log is here

Warning: 'input-dims' parameter has been deprecated. Use 'infer-dims' instead.
tst1!
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvTrackerParams::getConfigRoot()] !!![WARNING] File doesn't exist. Will go ahead with default values
[NvTrackerParams::getConfigRoot()] !!![WARNING] File doesn't exist. Will go ahead with default values
[NvMultiObjectTracker] Initialized
0:00:04.503105856 1350039 0x63ed0c14bcc0 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.4/samples/configs/conf-app/../../models/iFocus/resnet34_peoplenet_int8.onnx_b1_gpu0_int8.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1:0       3x544x960       
1   OUTPUT kFLOAT output_cov/Sigmoid:0 3x34x60         
2   OUTPUT kFLOAT output_bbox/BiasAdd:0 12x34x60        

0:00:04.641283671 1350039 0x63ed0c14bcc0 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.4/samples/configs/conf-app/../../models/iFocus/resnet34_peoplenet_int8.onnx_b1_gpu0_int8.engine
0:00:04.645671984 1350039 0x63ed0c14bcc0 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-6.4/samples/configs/conf-app/iFocus_primary.txt sucessfully

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

** INFO: <bus_callback:338>: Pipeline ready

*** stack smashing detected ***: terminated
Aborted (core dumped)

The config file is here.
iFocus_main.txt (9.5 KB)

What could be wrong?
CPU usages reached to 16% only and GPU usages also not exhausted.
This happened immediately right after the program starts.

junshengy · July 12, 2024, 4:01am

1.How much GPU memory is used?
2.This may be a memory problem. You can use valgrind to detect whether there is a memory out of bounds.

edit_or · July 12, 2024, 6:12am

Run valgrind as the following command.

valgrind --leak-check=full \
         --show-leak-kinds=all \
         --track-origins=yes \
         --verbose \
         --log-file=valgrind-out.txt \
         ./deepstream-app -c ../../../../samples/configs/conf-app/iFocus_main.txt

What do I observer?

It is screenshot during test.

edit_or · July 12, 2024, 6:19am

I have segmentation fault at the end.

ser@va:/opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/iFocus$ valgrind --leak-check=full          --show-leak-kinds=all          --track-origins=yes          --verbose          --log-file=valgrind-out.txt          ./deepstream-app -c ../../../../samples/configs/conf-app/iFocus_main.txt
** WARN: <parse_dsexample:908>: Unknown key 'batch-size' for group [ds-example]
Warning: 'input-dims' parameter has been deprecated. Use 'infer-dims' instead.
tst1!
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvTrackerParams::getConfigRoot()] !!![WARNING] File doesn't exist. Will go ahead with default values
[NvTrackerParams::getConfigRoot()] !!![WARNING] File doesn't exist. Will go ahead with default values
[NvMultiObjectTracker] Initialized
0:03:29.176800697 1363565    0x1406c6920 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.4/samples/configs/conf-app/../../models/iFocus/resnet34_peoplenet_int8.onnx_b1_gpu0_int8.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1:0       3x544x960       
1   OUTPUT kFLOAT output_cov/Sigmoid:0 3x34x60         
2   OUTPUT kFLOAT output_bbox/BiasAdd:0 12x34x60        

0:03:39.833621910 1363565    0x1406c6920 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.4/samples/configs/conf-app/../../models/iFocus/resnet34_peoplenet_int8.onnx_b1_gpu0_int8.engine
0:03:40.122489031 1363565    0x1406c6920 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-6.4/samples/configs/conf-app/iFocus_primary.txt sucessfully

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

Segmentation fault (core dumped)

edit_or · July 12, 2024, 6:28am

The followings are GPU memory usage for 17 cctvs. 17 cctvs runs very stable. Only changed to 18, I have issue.
17 cctvs, GPU memory usage is only ~1G. rtx3050 has 8GB.

junshengy · July 12, 2024, 7:48am

1.Run nvidia-smi pmon to check if the dec is too high. The decoder may be fully loaded.

# gpu         pid   type     sm    mem    enc    dec    jpg    ofa    command 
# Idx           #    C/G      %      %      %      %      %      %    name

After the crash, is there any output in the valgrind-out.txt from valgrind?

edit_or · July 12, 2024, 10:41am

The valgrind file is here. Thanks.
valgrind-out.txt - Google Drive

edit_or · July 15, 2024, 5:46am

Can you download the file?

junshengy · July 15, 2024, 6:43am

Yes, I can.I’m trying to analyze it. I guess it’s due to array out of bounds. There’s not much progress so far.

You can refer to the following link

edit_or · July 17, 2024, 2:08am

This is a screenshot for testing with 18 cctvs.
Recorded with nvidia-smi pmon. We can see that dec is only 13%.

junshengy · July 18, 2024, 7:25am

Have you made any changes to the deepstream-app? Especially perf_cb and perf_measurement_callback, there seems to be overflow here

==1363565== 1 errors in context 1 of 777:
==1363565== Invalid write of size 4
==1363565==    at 0x1293B8: perf_cb (in /opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/iFocus/deepstream-app)
==1363565==    by 0x14BE20: perf_measurement_callback (in /opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/iFocus/deepstream-app)
==1363565==    by 0x51552C7: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7200.4)
==1363565==    by 0x5154C43: g_main_context_dispatch (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7200.4)
==1363565==    by 0x51AA2B7: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7200.4)
==1363565==    by 0x51542B2: g_main_loop_run (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7200.4)
==1363565==    by 0x12BD90: main (in /opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/iFocus/deepstream-app)
==1363565==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

edit_or · July 18, 2024, 8:46am

Yeah I did slight modification in perf_cb (gpointer context, NvDsAppPerfStruct * str). I’ll check and thanks.
Stack smashing detected is because of that? Can’t be, right?

junshengy · July 18, 2024, 10:53am

Not necessarily. If there is no problem with the official sample, it may be caused by the modification.

yingliu · August 16, 2024, 8:16am

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

system · August 30, 2024, 8:17am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unsupported operation _MultilevelCropAndResize_TRT (and a few others) DeepStream SDK	10	582	May 31, 2022
Deepstream Occupancy Analytics Maximum Number of Cameras DeepStream SDK	8	902	December 28, 2021
Deepstream occupancy analytics running issues DeepStream SDK	2	600	April 25, 2022
Deepstream error DeepStream SDK	6	324	February 9, 2024
Run BACK-TO-BACK-DETECTORS REFERENCE APP under DeepStream SDK 5.0 DeepStream SDK	16	998	October 12, 2021
Issues with running inference on multiple rtsp streams in deepstream-imagedata-multistream DeepStream SDK jetson-inference	24	671	August 7, 2024
Unable to use custom-trained peoplenet model TAO Toolkit	6	873	April 20, 2022
Internal data stream error while running the deepstream-testsr-app DeepStream SDK	10	1011	March 7, 2023
Deepstream engine files not generated DeepStream SDK	13	4695	October 12, 2021
Windows through SSH Connect to ubuntu DeepStream SDK	20	608	March 22, 2024

Stack smashing detected

Related topics