Hello, I use YoloV8 on my Orin 16G device The converted model automatically shuts down after running for a while

229616726 · July 20, 2023, 2:39am

My project is based on Deepstream-test2 modifications

Pipeline flow:

appsrc ! h264parse ! nvv4l2decoder ! nvstreammux ! nvinfer ! nvtracker ! nvvidconv ! nvdsosd ! nvvideoconvert ! video/x-raw(memory:NVMM),format=I420 ! nvv4l2h264enc ! video/x-h264,stream-format=byte-stream ! h264parse ! flvmux ! rtmpsink

dstest2_config.yml

streammux:
  batch-size: 1
  batched-push-timeout: 40000
  width: 1920
  height: 1440
  live-source: 1

# /root/shared_develop/develop/sqn-iot-dji-spdk/source_object/ai/deep_test2_file/dstest2_config.yml

tracker:
  tracker-width: 640
  tracker-height: 384
  # tracker-width: 1280
  # tracker-height: 736
  gpu-id: 0
  ll-lib-file: ../source_object/ai/deep_test2_file/tracker_file/libnvds_nvmultiobjecttracker.so
  # ll-lib-file: /root/shared_package/ByteTrack/deploy/DeepStream/lib/libByteTracker.so
  # ll-config-file required to set different tracker types
  # ll-config-file: ../ai/deep_test2_file/tracker_file/config_tracker_IOU.yml
  # ll-config-file: ../ai/deep_test2_file/tracker_file/config_tracker_NvSORT.yml
  ll-config-file: ../source_object/ai/deep_test2_file/tracker_file/config_tracker_NvDCF_perf.yml
  # ll-config-file: ../ai/deep_test2_file/tracker_file/config_tracker_NvDCF_accuracy.yml
  # ll-config-file: ../ai/deep_test2_file/tracker_file/config_tracker_NvDeepSORT.yml
  enable-batch-process: 1

# Inference using nvinfer:
primary-gie:
  config-file-path: dstest2_pgie_config.yml

dstest2_pgie_config.yml

# Official model

property:
  gpu-id:  0
  net-scale-factor: 0.0039215697906911373
  model-file: Primary_Detector/resnet10.caffemodel
  proto-file: Primary_Detector/resnet10.prototxt
  model-engine-file: Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
  labelfile-path: Primary_Detector/labels.txt
  int8-calib-file: Primary_Detector/cal_trt.bin
  force-implicit-batch-dim: 1
  batch-size: 1
  network-mode: 1
  process-mode: 1
  model-color-format: 0
  num-detected-classes: 4
  interval: 0
  gie-unique-id: 1
  output-blob-names: conv2d_bbox;conv2d_cov/Sigmoid
  #scaling-filter: 0
  #scaling-compute-hw: 0
  cluster-mode: 2

class-attrs-all:
pre-cluster-threshold: 0.2
topk: 20
nms-iou-threshold: 0.5

# Use the yoloV8s model

# property:
#   gpu-id: 0
#   net-scale-factor: 0.0039215697906911373
#   model-color-format: 0
#   onnx-file: yolov8s.onnx
#   model-engine-file: ../yolov8s.onnx_b1_gpu0_fp32.engine
#   #int8-calib-file: calib.table
#   labelfile-path: ../labels.txt
#   batch-size: 1
#   network-mode: 0
#   num-detected-classes: 80
#   interval: 0
#   gie-unique-id: 1
#   process-mode: 1
#   network-type: 0
#   cluster-mode: 2
#   maintain-aspect-ratio: 1
#   symmetric-padding: 1
#   #force-implicit-batch-dim: 1
#   #workspace-size: 1000
#   parse-bbox-func-name: NvDsInferParseYolo
#   #parse-bbox-func-name: NvDsInferParseYoloCuda
#   custom-lib-path: ../libnvdsinfer_custom_impl_Yolo.so
#   engine-create-func-name: NvDsInferYoloCudaEngineGet

# class-attrs-all:
#   nms-iou-threshold: 0.45
#   pre-cluster-threshold: 0.25
#   topk: 300

Run the flow

When the program runs the official model in dstest2_pgie_config.yml, the device will not break the point during the running process, and when the model of yoloV8s is run, the device will automatically break the point when inference and push for a period of time

Operating environment

Distribution： ubuntu 20.04 focal
Python ： 3.8.10
CUDA：11.4.315
cuDNN ：8.6.0.166
TensorRT ：8.5.2.2
VPI：2.2.7
OpenCV：4.5.4
Model：NVIDIA Orin NX Developer Kit
Module：NVIDIA Jetson Oriin NX(16GB ram)
Jetpack：5.1.1
DeepStream: 6.2

yuweiw · July 20, 2023, 3:17am

If you run the yolo model, please refer to the yolo_deepstream. You need to implement your own post-processing function like: nvdsinfer_custom_impl_Yolo.

229616726 · July 20, 2023, 3:24am

The relevant configuration has been introduced in the configuration file :

  parse-bbox-func-name: NvDsInferParseYolo
  custom-lib-path: ../libnvdsinfer_custom_impl_Yolo.so
  engine-create-func-name: NvDsInferYoloCudaEngineGet

yuweiw · July 20, 2023, 5:27am

OK. Could you help to debug where it got stuck? You can open some nvinfer log by referring https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/33.

229616726 · July 20, 2023, 7:58am

好的，等测完后我尝试一下

229616726 · July 24, 2023, 3:03am

Hello, I have no abnormalities during the operation, but the device is powered off


root@nvidia-desktop:~/DeepStream-Yolo-master# deepstream-app -c deepstream_app_config.txt 
Opening in BLOCKING MODE 
DEBUG: [TRT]: Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::BatchedNMS_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::BatchTilePlugin_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::Clip_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::CoordConvAC version 1
DEBUG: [TRT]: Registered plugin creator - ::CropAndResizeDynamic version 1
DEBUG: [TRT]: Registered plugin creator - ::CropAndResize version 1
DEBUG: [TRT]: Registered plugin creator - ::DecodeBbox3DPlugin version 1
DEBUG: [TRT]: Registered plugin creator - ::DetectionLayer_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::EfficientNMS_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::FlattenConcat_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::GenerateDetection_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::GridAnchor_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::GridAnchorRect_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::GroupNorm version 1
DEBUG: [TRT]: Registered plugin creator - ::InstanceNormalization_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::InstanceNormalization_TRT version 2
DEBUG: [TRT]: Registered plugin creator - ::LayerNorm version 1
DEBUG: [TRT]: Registered plugin creator - ::LReLU_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::MultilevelProposeROI_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::NMSDynamic_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::NMS_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::Normalize_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::PillarScatterPlugin version 1
DEBUG: [TRT]: Registered plugin creator - ::PriorBox_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::ProposalDynamic version 1
DEBUG: [TRT]: Registered plugin creator - ::ProposalLayer_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::Proposal version 1
DEBUG: [TRT]: Registered plugin creator - ::PyramidROIAlign_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::Region_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::Reorg_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::ResizeNearest_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::ROIAlign_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::RPROI_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::ScatterND version 1
DEBUG: [TRT]: Registered plugin creator - ::SeqLen2Spatial version 1
DEBUG: [TRT]: Registered plugin creator - ::SpecialSlice_TRT version 1
DEBUG: [TRT]: Registered plugin creator - ::SplitGeLU version 1
DEBUG: [TRT]: Registered plugin creator - ::Split version 1
DEBUG: [TRT]: Registered plugin creator - ::VoxelGeneratorPlugin version 1
DEBUG: [TRT]: [MemUsageChange] Init CUDA: CPU +215, GPU +0, now: CPU 613, GPU 2964 (MiB)
DEBUG: [TRT]: Trying to load shared library libnvinfer_builder_resource.so.8.5.2
DEBUG: [TRT]: Loaded shared library libnvinfer_builder_resource.so.8.5.2
DEBUG: [TRT]: [MemUsageChange] Init builder kernel library: CPU +302, GPU +429, now: CPU 938, GPU 3429 (MiB)
DEBUG: [TRT]: Loaded engine size: 43 MiB
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
DEBUG: [TRT]: Trying to load shared library libcudnn.so.8
DEBUG: [TRT]: Loaded shared library libcudnn.so.8
DEBUG: [TRT]: Using cuDNN as plugin tactic source
DEBUG: [TRT]: Using cuDNN as core library tactic source
DEBUG: [TRT]: [MemUsageChange] Init cuDNN: CPU +617, GPU +947, now: CPU 1603, GPU 4464 (MiB)
DEBUG: [TRT]: Deserialization required 1860970 microseconds.
DEBUG: [TRT]: [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +42, now: CPU 0, GPU 42 (MiB)
DEBUG: [TRT]: Trying to load shared library libcudnn.so.8
DEBUG: [TRT]: Loaded shared library libcudnn.so.8
DEBUG: [TRT]: Using cuDNN as plugin tactic source
DEBUG: [TRT]: Using cuDNN as core library tactic source
DEBUG: [TRT]: [MemUsageChange] Init cuDNN: CPU +0, GPU +4, now: CPU 1559, GPU 4427 (MiB)
DEBUG: [TRT]: Total per-runner device persistent memory is 0
DEBUG: [TRT]: Total per-runner host persistent memory is 212288
DEBUG: [TRT]: Allocated activation device memory of size 36977152
DEBUG: [TRT]: [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +35, now: CPU 0, GPU 77 (MiB)
0:00:05.508967585  2536 0xaaaaf3cad4a0 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/root/DeepStream-Yolo-master/model_b1_gpu0_fp32.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 4
0   INPUT  kFLOAT input           3x640x640       
1   OUTPUT kFLOAT boxes           8400x4          
2   OUTPUT kFLOAT scores          8400x1          
3   OUTPUT kFLOAT classes         8400x1          

0:00:05.681648608  2536 0xaaaaf3cad4a0 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /root/DeepStream-Yolo-master/model_b1_gpu0_fp32.engine
0:00:05.712426519  2536 0xaaaaf3cad4a0 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/root/DeepStream-Yolo-master/config_infer_primary_yoloV8.txt sucessfully

Runtime commands:
        h: Print this help
        q: Quit

        p: Pause
        r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.


**PERF:  FPS 0 (Avg)
**PERF:  0.00 (0.00)
** INFO: <bus_callback:239>: Pipeline ready

Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:225>: Pipeline running

NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 363466 
NVMEDIA_ENC: bBlitMode is set to TRUE 
**PERF:  64.26 (64.20)
**PERF:  66.11 (65.30)
**PERF:  66.06 (65.64)

yuweiw · July 24, 2023, 5:59am

Could you refer to the link below and get the changes of memory when it’s running?
https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/14

yingliu · July 24, 2023, 9:33am

Can you also collect log from “tail -f /var/log/syslog”? In the meanwhile please open another console to collect the output of “sudo tegerastats” to check the power consumption of the Orin module.
AGX Orin module doesn’t have 16G version, are you using Orin NX? And what’s your Jetpack version?

229616726 · July 24, 2023, 10:07am

terminal-log.txt (42.8 KB)
Here is my memory usage

229616726 · July 24, 2023, 10:19am

syslog.log (6.2 KB)

Distribution： ubuntu 20.04 focal
Python ： 3.8.10
CUDA：11.4.315
cuDNN ：8.6.0.166
TensorRT ：8.5.2.2
VPI：2.2.7
OpenCV：4.5.4
Model：NVIDIA Orin NX Developer Kit
Module：NVIDIA Jetson Oriin NX(16GB ram)
Jetpack：5.1.1
DeepStream: 6.2

yingliu · July 25, 2023, 1:31am

Is this the log before shutdown happened? Please also check the output of “sudo tegrastats” to check the power consumption.

229616726 · July 25, 2023, 1:37am

tegrastats.txt (37.3 KB)
This is the log from tegrastats

yingliu · July 25, 2023, 1:45am

Just to confirm, the Orin NX was shutdown after printing the last line of tegrastats, correct?
Is the power adapter came together with the Orin NX devkit?

229616726 · July 25, 2023, 2:12am

After the power failure, the tegrastats ssh terminal also stopped, and it was impossible to check whether it stopped. I used a 20V4.5A Lenovo power supply

WayneWWW · July 25, 2023, 6:51am

Are we sure it is “shutdown” but not “reboot”? Shutdown indicates thermal or power issue while reboot means it could be software triggered.

229616726 · July 25, 2023, 10:02am

I’m sure it’s a shutdown, not a reboot

Trumany · July 26, 2023, 7:42am

Did you try other power supply like DC supply with sufficient output capability?

229616726 · July 26, 2023, 7:51am

currently being tested