System throttled due to Over-Current! - DeepStream 5.1 with just 4 video streams!

We just upgraded our XNX from Jetpack 4.4 to 4.5.1.
Starting our DeepStream based Embedded Vision with just 4 Full HD Video Streams (PeopleNet / Resnet 18) directly ends up in a system throttling error message: “System throttled due to Over-Current!”
This is really an unexpected behavior because the XavierNX is advertised to be able to do an inference on at least 4 video streams of the mentioned size.
Furthermore, everything works perfectly using Jetpack 4.4 without ever reaching critical temperatures after weeks of operations.
What can we do to get rid of that obvious faulty error message?
This is really a critical issue for us!

You may find several topics about this on this forum.
You may inscrease the current limit to 5A with:

sudo sh -c 'echo 5000 > /sys/devices/c250000.i2c/i2c-7/7-0040/iio:device0/crit_current_limit_0'

Thank you for your hint, but that only delays the warning for 20 seconds.
The GPU load checked by jtop (see attached image) peaks above 90% but is in average below 80%.
Although the system warns that it will throttle the GPU it doesn’t do it at all!

So, we’re looking forward to a timely solution from NVIDIA.

Is there any way to get rid of the warning by stopping the associated service until a final solution is available?

Over-Current|690x434

You can temporarily close the notification.
Click the power mode selection bar → Settings → Disable notification

Regards

Please check this thread first.

Yes, we checked the thread already and customized the nvpmodel.conf conservatively to 14.5W, assuming a high GPU load and medium CPU load.

But it does not help (5A patch applied) and still creates the Over-Current.

JTOP (based on Tegra Stats) reports 11W

We are talking about PeopleNet Resnet18/FP16, a batch size of 4; interval 1 (inference on every second frame), KLT-Tracker and reaching a max of 40FPS.

Changing the PeopleNet to Resnet18/int8 helps but the frame rate is far below your benchmark results.

It gets even worse if we try to use PeopleNet Resnet34/int8; batch size 1, interval 1 and a KLT-Tracker which results in a maximum of 40FPS (inference on every second frame!) and a prompt over current.
The Tiler is configured to one tile and we’re working on the full network size of 960x544 using the nveglessink.

Well, I’m really wondering how you achieved 168FPS (PeopleNet Resnet34/int8) with an interval of 0 without directly hitting the over-current warning.

So, we really want to help by supporting appropriate tests to solve the over-current issue, getting the same performance from DeepStream as we got by using JetPack 4.4

That don’t work for many people. And that calculator thing they give for I don’t know what, is a joke.
The problem is with some models specially the ones that use tensorrt
I have a power meter that mesure the whole thing and hou can pass 22W and dont see the alarm, or it can appear with 12W.
I think it could be a bug with tensorrt or jtop.

It could be programmed obsolescence too, it looks like what happens with android.
Just to you buy other thing.
Because is clear that there is this issue and they send you to the calculator saying like, go accepting this, will be like this.

But the fact is that JP4.4 don’t have the issue.

1 Like

hi @sh2222 ,

Could you share your setup steps and your new nvpmodel.conf? We tried similar setup locally but does not hit the oc throttle.

Hi sh2222,

Please try below steps and check your fps, thanks!

$ cd /opt/nvidia/deepstream/deepstream-5.1/samples/models/tlt_pretrained_models/peoplenet
$ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_peoplenet/versions/pruned_v2.0/files/resnet34_peoplenet_pruned.etlt
$ wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_peoplenet/versions/pruned_v2.0/files/resnet34_peoplenet_int8.txt

$ /opt/nvidia/deepstream/deepstream-5.1/samples/configs/tlt_pretrained_models
# Replace deepstream_app_source30_peoplenet.txt and config_infer_primary_peoplenet.txt
$ deepstream-app -c deepstream_app_source30_peoplenet.txt

deepstream_app_source30_peoplenet.txt (3.4 KB) config_infer_primary_peoplenet.txt (2.0 KB)

Hi carolyuu,

Although we’ve implemented our own Embedded Vision (C/C++) using the DeepStream plugins, we’ve invested the effort to analyze the issue by using NVIDIA’s deepstream-app.

Because the configuration you’ve delivered tried to open 30 video streams reaching around 2FPS and a direct over-current we decided to re-write it for a batchsize of 1.
So, the video input stream has a size of 1920x1080 / 30FPS (sample_1080p_h264.mp4) and the tracker is OFF, although that’s not very realistic.
Furthermore, to achieve the maximum performance the nvpmodel is set to 2 (15W / 6 Core).
Here we reach a maximum of around 56FPS and the Over-Current warning comes up and stays active!

56FPS_max

To find out at which frame rate the over-current warning appears and stays active we changed the frame rate of the file (using ffmpeg) and configured the [sink0] to sync=1.

At a frame rate of about 48FPS we hit the over-current warning which does not go away.
Well, this is far below the advertised benchmark of 168FPS.

XNX_Benchmark

Using jtop to do a deeper analysis reveals that the over-current warning reacts very fast to any short power peaks. Our application just spikes at the beginning to 14W and stays at around 11W (30FPS).

As already mentioned JetPack 4.4 worked a lot better.

Please be so kind to provide us with the deepstream-app configuration that achieves 168FPS (advertised benchmark value) without getting the over-current warning.

To make it easily possible for you and the valued forum members to check the test results we’ve attached the adapted config files:

config_infer_primary_peoplenet.txt (2.0 KB) deepstream_app_source30_peoplenet.txt (3.4 KB)

We hope you’re going to invest further efforts to solve this issue as fast as possible.

Thank you very much for your support!

Could you also reply my question?

Hi WayneWWW,

As explained in my answer to carolyuu, we’re using our own implementation of the embedded vision. Our configuration files are therefore not useful for you.
That is the reason why we invested the effort to replicate the issue easily with your (NVIDIA) deepstream-app.
It is in fact very easy (if you follow the steps above and if you use the configuration we’ve shared) to replicate the over-current issue.
I also described that we changed the FPS of the sample file (sample_1080p_h264.mp4 / 1920x1080 / 30FPS) in steps up to 48FPS to find out where the over-current issue appears when the nvpmodel is set to max (15W / 6 Core).

So, it would be really helpful if you send us the configuration for the NVIDIA deepstream-app that achieves 168 FPS (PeopleNet / Resnet34 / Full HD Video Stream / 960 x 544 Network Size) without hitting the over current issue.

Well, we have our doubts whether that frame rate would be ever achievable if the nvpmodel limits the GPU or CPU clocks.

Thanks for your support!

Hi,

So could you share the custom nvpmodel.conf?

Hi WayneWWW,

We’re quite amazed that we don’t get an answer to our questions although we’ve implicitly answered yours.

Nevertheless, we’re eager to help investigating the issue to get a solution for all customers who hit the similar problem as documented by numerous threads on this forum.

To make sure that you and the forum members are able to replicate the problem transparently we decided to go for a very easy (more scientific) approach.
This assures that nobody can make any none verifiable claims.

Setup:

  • Xavier NX (2 different carrier boards from NVIDIA hardware partners)
  • PeopleNet / Resnet34 / int8
  • Input Source 3x 1920 x 1080 (30 FPS) NVIDIA sample file: sample_1080p_h264.mp4
  • Tiler Configuration: 1920 x 360 (1 row, 3 columns)
  • NVIDIA deepstream-app executed with the attached configuration files representing the above mentioned setup:

deepstream_app_source3_resnet34_peoplenet.txt (3.4 KB)
config_infer_primary_peoplenet.txt (2.0 KB)

Expected Results:
1.) The setup delivers 3 video streams in 3 tiles each reaching at least 30 FPS (90FPS in total) which is far below the advertised 168 FPS benchmark.
2.) The system is able to reach and keep the advertised (specification) 21 TOPS at 6 CPUs (1.4GHz) as well as a max. of 1.1GHz for the GPU
3.) The system does not hit any over-current warning when using the standard nvpmodel.conf at mode 2 by demanding the specified video streams, because that demand is far below the official NVIDIA system specification
4.) By using the attached 14.5W nvpmodel.conf the system shall still reach 3x 30FPS without hitting the over-current warning

nvpmodel.conf (6.0 KB)

1. Experiment:

  • Set the standard nvpmodel.conf to mode 2 (6 CPUs at 1.4GHz and GPU at a max. of 1.1GHz)
  • Start the NVIDIA deepstream-app by using the above mentioned and attached configuration files as well as the resnet34 PeopleNet (pruned)

Result:

  • We reach an average frame rate per stream below 25 FPS and therefore by far not even the expected 30 FPS
  • The over-current warning appears imediately and stays active!

2. Experiment:

  • Use the attached 14.5W nvpmodel.conf assuming a high GPU load, medium CPU load and an active h.264 hardware decoder
  • Start the NVIDIA deepstream-app by using the above mentioned and attached configuration files as well as the resnet34 PeopleNet (pruned)

Result:

  • The resulting frame rate is below 23FPS and therefore even lower and far away from the expected 30FPS
  • Despite of the performance limiting 14.5 W nvpmodel.conf the system goes immediately into the over-current warning!

After all the effort to investigate the issue we would be very thankful if you look deeper into it and if you could provide us with the following information:

  • A configuration for the deepstream-app that achieves the 168FPS. At least 120FPS to handle 4 full HD streams at 30FPS by using the Resnet34 PeopleNet (pruned)
  • An nvpmodel.conf that does not limit the 21 TOPS at 6 CPUs (1.4GHz) and GPU at (1.1GHz) to fullfil the NVIDIA Xavier NX specification and not causing any over-current warning

Thank you very much for your support

2 Likes

Hi,
We set OC limit to 5A and run 3-source usecase. The fps looks fine:

**PERF:  44.57 (44.50)	44.57 (44.50)	44.57 (44.50)	
**PERF:  44.61 (44.55)	44.61 (44.55)	44.61 (44.55)	
**PERF:  44.59 (44.56)	44.59 (44.56)	44.59 (44.56)	
**PERF:  44.37 (44.52)	44.37 (44.52)	44.37 (44.52)	
**PERF:  44.53 (44.50)	44.53 (44.50)	44.53 (44.50)	
**PERF:  44.64 (44.55)	44.64 (44.55)	44.64 (44.55)

Although rhere are occasional OC warnings, it is ~44 fps for each source. Somehow you only get ~22fps. Do you run on Xavier NX devkit or custom carrier board?

The benchmark is run without enabling OC function. For better stability we would like to enable OC function. Will check with our teams and update the benchmark.

Hi DaneLLL,

1st of all I want to thank you and your team for the effort you invested into investigating this issue.

We’re using indeed carrier boards from different vendors (including NVIDIA HW partner).
By trying just another carrier board we can confirm your measurements!
So, we already contacted the vendor to find out what is going wrong. The measured clock rates are at the specified limits of the nvpmodel (2) chosen, but the performance is just at 50% with continous over-current.

Do you think that it still will be possible for all future implementations to make use of the full 21 TOPS although the nvpmodel and oc-current protection limit the clock rates?

Hi, sh2222

We are discussing this issue with our internal team.
Will update more information with you later.

Thanks.

Hi,

Thanks for your patience.
After checking with our internal team, you can reach 164fps with setting as following:

$ sudo nvpmodel -m 2
$ sudo jetson_clocks
$ wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/tlt_peoplenet/versions/pruned_quantized_v2.1/zip -O tlt_peoplenet_pruned_quantized_v2.1.zip
$ unzip tlt_peoplenet_pruned_quantized_v2.1.zip
# Put deepstream_app_source5_peoplenet_v2.txt and config_infer_primary_peoplenet_v2.txt in the same folder
$ deepstream-app -c deepstream_app_source5_peoplenet_v2.txt

Thanks.

deepstream_app_source5_peoplenet_v2.txt (3.3 KB)
config_infer_primary_peoplenet_v2.txt (1.9 KB)