Jetson Nano2g run DS6 inference from a Live Camera RTSP stream with 6 FPS

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc)
jetson nano 2g
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc)
Detectnet_v2
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

I’ve retrained a model from detectnet_v2 (resnet 18) by TAO, and it’s running good on my Jetson NX and Jetson Nano2g board with a testing file source, but see 2 potential issues for live camera RTSP stream (25fps, 1280*960) for JetsonNano 2g board:

NOTE, both testing are running on a customized Deepstream 6 python sample app which based on test4, it now can support read a rtsp source and send detected objects info to a remote kafka server.

  • Low inference performance
    I got 6 FPS in jetson nano 2g board, is it a reasonable result?
    Any recommendation to improve?

  • Too large time lagging
    With the low perf, the real-time inference is impossible, from my testing, the inference osd window show minutes delay from live camera monitor, as I think it’s caused by buffering un-inferenced frames.
    Any Python code sample to follow to disable or lower the input RTSP buffer in my Python app, that means I prefer to drop old frames, and keep the latest frames as priority to do inferencing?

Thanks.

3

The fps depends on hardware, the input-size of your trained model, the classes quantities, the pruned ratio of your model, fp16/int8 mode, batch-size , etc.

Please consider to train a smaller model, prune/retrain as much as possible.

4

thanks Morgan,

I looked the perf chart for DashCamNet which say Jetson Nano (may the model 4G?) can reach 17FPS, as my model is quite similar to it: retrained from detectnet-v2, and detecting classes is 4.

  1. Can I understand the major perf diff is caused by prune rate?
  2. What is the input-size of model?
  3. Is there anyway to drop frames for mostly catch up with live stream in my situation?
5

Can you share your training spec?

After pruning/retraining, the model has less trainable parameters. The pruned model usually has a faster fps.

Yes.

6

I understand that to achieve real-time inference (no matter 24 or 30 FPS) on my Jetson Nano 2g is quite challenge, or may can’t as the hardware’s low spec.

By the fact of very limited training dataset (<=1000 per class) on hand, I don’t want to sacrifice too much on precision by reduce too much on pruning stage though I’ll try to get a good balance later.

So I’ll still stick with that is there anyway to adjust params when constructing the gstreamer pipelines (I’m using Python code) to drop old frames?

My training spec:

random_seed: 42
dataset_config {
data_sources {
tfrecords_path: “/workspace/tao-experiments/data/tfrecords/kitti_trainval/*”
image_directory_path: “/workspace/tao-experiments/data/training”
}
image_extension: “jpg”
target_class_mapping {
key: “door_warning_sign”
value: “door_warning_sign”
}
target_class_mapping {
key: “people”
value: “people”
}
target_class_mapping {
key: “electric_bicycle”
value: “electric_bicycle”
}
validation_fold: 0
}
augmentation_config {
preprocessing {
output_image_width: 800
output_image_height: 608
min_bbox_width: 1.0
min_bbox_height: 1.0
output_image_channel: 3
}
spatial_augmentation {
hflip_probability: 0.5
zoom_min: 1.0
zoom_max: 1.0
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
postprocessing_config {
target_class_config {
key: “door_warning_sign”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.20000000298
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 10
}
}
}
target_class_config {
key: “people”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.15000000596
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “electric_bicycle”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
}
model_config {
pretrained_model_file: “/workspace/tao-experiments/detectnet_v2/pretrained_resnet18/pretrained_detectnet_v2_vresnet18/resnet18.hdf5”
num_layers: 18
use_batch_norm: true
objective_set {
bbox {
scale: 35.0
offset: 0.5
}
cov {
}
}
arch: “resnet”
}
evaluation_config {
validation_period_during_training: 10
first_validation_epoch: 20
minimum_detection_ground_truth_overlap {
key: “door_warning_sign”
value: 0.4
}
minimum_detection_ground_truth_overlap {
key: “people”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “electric_bicycle”
value: 0.5
}
evaluation_box_config {
key: “door_warning_sign”
value {
minimum_height: 10
maximum_height: 9999
minimum_width: 14
maximum_width: 9999
}
}
evaluation_box_config {
key: “people”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “electric_bicycle”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
average_precision_mode: INTEGRATE
}
cost_function_config {
target_classes {
name: “door_warning_sign”
class_weight: 10.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “people”
class_weight: 5.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 1.0
}
}
target_classes {
name: “electric_bicycle”
class_weight: 5.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
enable_autoweighting: true
max_objective_weight: 0.999899983406
min_objective_weight: 9.99999974738e-05
}
training_config {
batch_size_per_gpu: 4
num_epochs: 120
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 5e-06
max_learning_rate: 5e-04
soft_start: 0.10000000149
annealing: 0.699999988079
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
optimizer {
adam {
epsilon: 9.99999993923e-09
beta1: 0.899999976158
beta2: 0.999000012875
}
}
cost_scaling {
initial_exponent: 20.0
increment: 0.005
decrement: 1.0
}
checkpoint_interval: 10
}
bbox_rasterizer_config {
target_class_config {
key: “door_warning_sign”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.40000000596
cov_radius_y: 0.40000000596
bbox_min_radius: 1.0
}
}
target_class_config {
key: “people”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “electric_bicycle”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
deadzone_radius: 0.400000154972
}

7

It is a question for deepstream. Please search or find help in deepstream forum.

Please try to prune/retrain, to check if you can get similar mAP.

1 Like
8

To increase the FPS You could enable the Interval setting in the Deepstream-app.I.e. interval=3. This will produce flickering bounding boxes. To resolve this, use the tracker and enable past frame.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.