Drive AGX Orin Dev Kit: Power Mode Selection based on Frame Drops with IMX728 while using recording tool

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Software Version
DRIVE OS 6.0.10.0
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
2.1.0
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Issue Description
We are currently using a setup with 6 IMX728 camera units, each having 8.3 MP. At the moment, the cameras are connected as follows:

Camera 0 → A,csi-ab,link=0,cphy-mode=1
Camera 1 → A,csi-ab,link=1,cphy-mode=1
Camera 2 → C,csi-ef,link=0,cphy-mode=1
Camera 3 → C,csi-ef,link=1,cphy-mode=1
Camera 4 → C,csi-ef,link=2,cphy-mode=1
Camera 5 → C,csi-ef,link=3,cphy-mode=1

When we use the DriveWorks recording tool, we experience “Frame Drops” (ICP_FRAME_DROP), and for example, the FPS with the recording tool occasionally drops to 15 FPS (the camera units can natively run at 30 FPS).

According to the bandwidth calculation guidelines (https://developer.nvidia.com/docs/drive/drive-os/6.0.10/public/drive-os-linux-sdk/common/topics/nvmedia_use_setup/bandwidth_computation.html), the clock frequency of the ISP, VI 0/1, and NVCSI depends on the Orin power mode (see table).

This is our sample calculation:

Total MP = 6*8.3 MP (IMX728) = 49.8 MP

Total BW: 49.8 MP * 12 BPP * 30 FPS = 17.928 Gb/s

NVCSI:

* Total BW: 17.928 Gb/s < 164.6 Gb/s (in 4x C-Phy mode)  plausible

* Restrictions:

*(i) TEGRA234_CLK_NVCSI is fixed (642.9 MHz) for all available Power Profiles*

=> TEGRA234_CLK_NVCSI: 642.9 MHz

VI:

* Performance: 8 PPC (since 12 BPP ≤ 16 BPP)

* Required: 49.8 MP * 30 FPS / 8 PPC = 186,75 MHz

* Restrictions:

*(i) TEGRA234_CLK_VI ≥ TEGRA234_CLK_NVCSI (642.9 MHz)*

=> TEGRA234_CLK_VI: 642.9 MHz

=> (plausible 186,75 MHz < 642.9 MHz)

ISP:

* Performance: 2 PPC

* Required 49.8 MP * 30 FPS / 2 PPC = 747 MHz

* Additional: +15% for Tiling+ISP PFSD overhead

* Required = 747 MHz *1.15 = 859,05 MHz

=> (Reached limit: 859,05 MHz > 772.8 MHz)

=> TEGRA234_CLK_ISP: 772.8 MHz

Is the sample calculation correct, or is it limited by the NVCSI or VI frequency? How can I adjust the power mode to check whether the frame drops are caused by this (Power Profiles | NVIDIA Docs)? Is there perhaps potential for optimization by redesigning the camera configuration?

Error String
ICP_FRAME_DROP

2

Dear @marius.westendorf ,
I suspect issue could be due to storage BW. Could you check the bandwidth of storage device using dd command? Please check increasing cameras one by one to identify where you notice frame drops. Also use MP4 format for recording.

1 Like
3

First of all thanks for the fast reply.

I will carry out the test and post the results here by tomorrow. We are currently using a hard drive connected via USB-C. What would you recommend for data recording and additional storage space on the Drive AGX Orin Dev Kit in general? Is there any further documentation on this?

4

Yes. We recommend to connect storage device via USB 3.2 port.

Check if the steps at https://developer.nvidia.com/docs/drive/drive-os/6.0.10/public/driveworks-nvsdk/nvsdk_dw_html/dwx_recording_devguide_high_throughput_recording.html increases BW on storage device.

1 Like
6

Dear SivaRamaKrishnaNV

Your assumption was correct. The bandwidth of the USB hard drive is below its potential because it is set up with NTFS.

I therefore made another recording directly on the Orin itself and made the changes described above beforehand. In addition, I distributed the cameras across A, C and D. I am still experiencing frame drops with the recording tool. Can you help me determine whether I should adjust the configuration further? Thank you very much for your help.

Error logs:

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:5 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:3 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:1 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:2 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:4 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:5 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:1 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:3 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:2 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:4 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:5 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:1 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:3 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:2 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:5 of type: NOTIF_WARN_ICP_FRAME_DROP

[29-10-2025 17:49:16] SIPLNotifier: Notification received from pipeline index:4 of type: NOTIF_WARN_ICP_FRAME_DROP

The current rig file:

{

“rig”: {

"sensors": \[

  {

    "name": "camera:front:left:60fov",

    "nominalSensor2Rig_FLU": {

      "roll-pitch-yaw": \[

        0.0,

        -0.0,

        0.0

      \],

      "t": \[

        0.0,

        0.0,

        0.0

      \]

    },

    "parameter": "camera-name=IMX728_RGGB,interface=csi-ab,CPHY-mode=1,link=0,format=mp4,output-format=yuv,warn-per-frame=0,output-timestamp=host,fifo-size=10",

    "properties": {

      "Model": "ftheta",

      "bw-poly": "0.0 0.000545421498827636 -1.6216719633103e-10 -4.64720492990289e-12 2.85224527762934e-16",

      "cx": "960",

      "cy": "540",

      "height": "1080",

      "width": "1920"

    },

    "protocol": "camera.gmsl"

  },

  {

    "name": "camera:front:right:60fov",

    "nominalSensor2Rig_FLU": {

      "roll-pitch-yaw": \[

        0.0,

        -0.0,

        0.0

      \],

      "t": \[

        0.0,

        0.0,

        0.0

      \]

    },

    "parameter": "camera-name=IMX728_RGGB,interface=csi-ab,CPHY-mode=1,link=1,format=mp4,output-format=yuv,warn-per-frame=0,output-timestamp=host,fifo-size=10",

    "properties": {

      "Model": "ftheta",

      "bw-poly": "0.0 0.000545421498827636 -1.6216719633103e-10 -4.64720492990289e-12 2.85224527762934e-16",

      "cx": "960",

      "cy": "540",

      "height": "1080",

      "width": "1920"

    },

    "protocol": "camera.gmsl"

  },

  {

    "name": "camera:front:left:120fov",

    "nominalSensor2Rig_FLU": {

      "roll-pitch-yaw": \[

        0.0,

        -0.0,

        0.0

      \],

      "t": \[

        0.0,

        0.0,

        0.0

      \]

    },

    "parameter": "camera-name=IMX728_RGGB,interface=csi-ef,CPHY-mode=1,link=0,format=mp4,output-format=yuv,warn-per-frame=0,output-timestamp=host,fifo-size=10",

    "properties": {

      "Model": "ftheta",

      "bw-poly": "0.0 0.000545421498827636 -1.6216719633103e-10 -4.64720492990289e-12 2.85224527762934e-16",

      "cx": "960",

      "cy": "540",

      "height": "1080",

      "width": "1920"

    },

    "protocol": "camera.gmsl"

  },

  {

    "name": "camera:front:right:120fov",

    "nominalSensor2Rig_FLU": {

      "roll-pitch-yaw": \[

        0.0,

        -0.0,

        0.0

      \],

      "t": \[

        0.0,

        0.0,

        0.0

      \]

    },

    "parameter": "camera-name=IMX728_RGGB,interface=csi-ef,CPHY-mode=1,link=1,format=mp4,output-format=yuv,warn-per-frame=0,output-timestamp=host,fifo-size=10",

    "properties": {

      "Model": "ftheta",

      "bw-poly": "0.0 0.000545421498827636 -1.6216719633103e-10 -4.64720492990289e-12 2.85224527762934e-16",

      "cx": "960",

      "cy": "540",

      "height": "1080",

      "width": "1920"

    },

    "protocol": "camera.gmsl"

  },

  {

    "name": "camera:rear:left:120fov",

    "nominalSensor2Rig_FLU": {

      "roll-pitch-yaw": \[

        0.0,

        -0.0,

        0.0

      \],

      "t": \[

        0.0,

        0.0,

        0.0

      \]

    },

    "parameter": "camera-name=IMX728_RGGB,interface=csi-gh,CPHY-mode=1,link=0,format=mp4,output-format=yuv,warn-per-frame=0,output-timestamp=host,fifo-size=10",

    "properties": {

      "Model": "ftheta",

      "bw-poly": "0.0 0.000545421498827636 -1.6216719633103e-10 -4.64720492990289e-12 2.85224527762934e-16",

      "cx": "960",

      "cy": "540",

      "height": "1080",

      "width": "1920"

    },

    "protocol": "camera.gmsl"

  },

  {

    "name": "camera:rear:right:120fov",

    "nominalSensor2Rig_FLU": {

      "roll-pitch-yaw": \[

        0.0,

        -0.0,

        0.0

      \],

      "t": \[

        0.0,

        0.0,

        0.0

      \]

    },

    "parameter": "camera-name=IMX728_RGGB,interface=csi-gh,CPHY-mode=1,link=1,format=mp4,output-format=yuv,warn-per-frame=0,output-timestamp=host,fifo-size=10",

    "properties": {

      "Model": "ftheta",

      "bw-poly": "0.0 0.000545421498827636 -1.6216719633103e-10 -4.64720492990289e-12 2.85224527762934e-16",

      "cx": "960",

      "cy": "540",

      "height": "1080",

      "width": "1920"

    },

    "protocol": "camera.gmsl"

  }

\],

"vehicle": {

  "valid": false,

  "value": {

    "aeroHeight": 1.5,

    "aerodynamicDragCoeff": 0.699999988079071,

    "axlebaseFront": 1.5820000171661377,

    "axlebaseRear": 1.5750000476837158,

    "brakeActuatorTimeConstant": 0.5,

    "bumperFront": 0.9120000004768372,

    "bumperRear": 1.1089999675750732,

    "centerOfMassHeight": 0.550000011920929,

    "centerOfMassToFrontAxle": 1.2860000133514404,

    "centerOfMassToRearAxle": 1.5640000104904175,

    "driveByWireTimeConstant": 0.25999999046325684,

    "driveByWireTimeDelay": 0.10999999940395355,

    "effectiveMass": 0,

    "frontCorneringStiffness": 30654,

    "frontSteeringOffset": 0,

    "frontalArea": 1,

    "height": 1.4730000495910645,

    "inertia3D": \[

      750,

      3200,

      3500

    \],

    "length": 4.872000217437744,

    "mass": 1779.4000244140625,

    "maxEnginePower": 134000,

    "maxSteeringWheelAngle": 8.290313720703125,

    "rearCorneringStiffness": 36407,

    "rollingResistanceCoeff": 0.014999999664723873,

    "steeringWheelToSteeringMap": \[

      0,

      0.06764844059944153,

      -0.00002211817081843037,

      0.0001220461999764666,

      6.98658084274939e-7,

      0.0000010106930403708247

    \],

    "throttleActuatorTimeConstant": 1,

    "torqueLUT": {

      "brakePedalInput": "0.000, 0.07, 0.14, 0.21, 0.29, 0.36, 0.43, 0.50, 0.57, 0.64, 0.71, 0.79, 0.86, 0.93, 1.000",

      "brakeTorqueOutput": "0.000, 236.05, 470.44, 708.48, 944.52, 1180.57, 1416.95, 1653.000, 1889.05, 2125.43, 2361.48, 2597.52, 2833.57, 3069.95, 3306.000",

      "throttlePedalInput": "0.000, 0.07, 0.14, 0.21, 0.29, 0.36, 0.43, 0.50, 0.57, 0.64, 0.71, 0.79, 0.86, 0.93, 1.000",

      "throttleSpeedInput": "0.000, 2.000, 4.000, 6.000, 8.000, 10.000, 12.000, 14.000, 16.000, 18.000, 20.000, 22.000, 24.000, 26.000, 28.000",

      "throttleTorqueOutput": \[

        "0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000",

        "103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15, 103.15",

        "205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96, 205.96",

        "359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03, 359.03",

        "874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77, 874.77",

        "1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32, 1412.32",

        "1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32, 1775.32",

        "2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79, 2000.79",

        "2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61, 2103.61",

        "2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41, 2167.41",

        "2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94, 2224.94",

        "2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79, 2282.79",

        "2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62, 2344.62",

        "2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76, 2407.76",

        "2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57, 2470.57"

      \]

    },

    "wheelRadius": \[

      0.33059999346733093,

      0.33059999346733093,

      0.33059999346733093,

      0.33059999346733093

    \],

    "wheelbase": 2.8499999046325684,

    "width": 1.8519999980926514,

    "widthWithMirrors": 2.121000051498413

  }

},

"vehicleio": \[\]

},

“version”: 7

}
7

Dear @marius.westendorf ,
What is the write bandwidth of storage?
Check recoding single camera about 30 secs and compare file size and number frame stored from timestamp. This helps to get per frame size of MP4 encoding. Using this info we can know how much bandwidth is needed to record 6 camera.
Can you check increasing cameras from 1 to 6 and observe when frame drops is noticed.

8

Dear @marius.westendorf ,
Does this issue still need support? Could you please provide any update here?

9

Dear SivaRamaKrishnaNV,

sorry for the late response. Here is my update:

Regarding the bandwith of storage, this is my new result based on ext4 format of storage:
dd if=/dev/zero of=/media/sda1_USB_Linux/02_Data/drive_works/file2 bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 29.0956 s, 369 MB/s

Regarding the proposed testing:
I can confirm that first drops occur when 4 cameras are atttached. I attached statistics for 3 and 4 cameras. The higher max dt between frames in seconds indicates frame drops:

Statistics for 3 cams (max dt between frames 0.033361 s):
Frames per video:
[1175, 1174, 1173] @30fps / 0.033s per frame
Videosizes in bytes:
[39307000, 39303623, 39474348]
Megabytes (MB) per frame:
[0.0335, 0.0335, 0.0337]
Gigabits (Gb) per frame:
[0.0003, 0.0003, 0.0003]

Statistics for 4 cams (max dt between frames 0.099998 s):
Frames per video:
[1340, 1339, 1336, 1334] @30fps / 0.033s per frame
Videosizes in bytes:
[44853269, 44824698, 44867102, 44816908]
Megabytes (MB) per frame:
[0.0335, 0.0335, 0.0336, 0.0336]
Gigabits (Gb) per frame:
[0.0003, 0.0003, 0.0003, 0.0003]

10

Dear @marius.westendorf ,
1175, 1174, 1173] @30fps / 0.033s per frame
Videosizes in bytes:
[39307000, 39303623, 39474348]
Megabytes (MB) per frame:
[0.0335, 0.0335, 0.0337]

That means each frame size is 33KB? Could you cross verify it with timestamp file generated with recorder tool to know number of frames and file size.

11

The results are based on the “.timestamps” file per camera. For example for one of the three cameras in this case:

First timestamp: 0 1761761941502998
Last timestamp: 1174 t 1761761980635840
with a video size of 39,3MB

So yes, you are right.

12

Dear @marius.westendorf ,
Another possibility could be limited by encoder bandwidth.
MP4 uses h264 encoding. Could you check running NvMedia IEP – Encode Processing (nvm_iep_sci) | NVIDIA Docs for h264 encoding with recorded data from your camera to know the achieved FPS.

13

Dear @marius.westendorf ,
Could you provide any update?

14

Sorry for the long delay in updating. Is it possible to get more detailed information about which configuration I need to use for the case described?

15

DId you try this to know the limitation of encoder for your usecase?

16

I located the nvm_iep_sci utility within the DRIVE OS Docker container and transferred it to the Orin platform. Subsequently, I recorded a short video sequence using a single camera in raw format (output-format=raw+yuv). After processing this data with the tool, I obtained the following results with a sample config:

./nvm_iep_sci -cf /home/iff_drive/nv_media_iep/enc_h264_sample.cfg -profile 30

Exiting encode process

============ Perf. Report: Initialization Latency
[NOTE] All time-values are in USEC

min : 4146.368
max : 4146.368
average : 4146.368
99.99-pct : 4146.368
Std. Deviation : 0.000
Num. of Samples : 1

============ Perf. Report: Submission Latency

[NOTE] All time-values are in USEC

min : 108.384
max : 368.704
average : 129.963
99.99-pct : 261.440
Std. Deviation : 14.364
Num. of Samples : 33076

============ Perf. Report: Execution Latency

[NOTE] All time-values are in USEC

min : 19.520
max : 656.800
average : 158.147
99.99-pct : 363.552
Std. Deviation : 8.204
Num. of Samples : 33076

Encode FPS achieved: 6323.2295

Total encoded frames = 33076, avg. bitrate=6726780, maxBitrate=6930200, minBitrate=5945856

17

In my understanding, these results do not align with the original error case.

18

Dear @marius.westendorf ,
Could you please share the resolution of the input frame , input video and configuration file used with IEP sample?

19

Here are the requested details:

  • Resolution: 3840x2160
  • Video: It is 10 GB in size; how can I make it available to you? I also need to clarify whether I am permitted to forward the data.
  • The iep config file is attached.

enc_h264_sample.zip (4.0 KB)

20

Dear @marius.westendorf ,
Could you modify below params and test?

EPEncodeWidth               = 3840   # Specifies the encode width
EPEncodeHeight              = 2160   # Specifies the encode height
FramesToBeEncoded = 100 # modify to decide number of frames to encode instead of whole stream
21

With changed settings, the result seems to be the reason:

*./nvm_iep_sci -cf /enc_h264_sample.cfg -profile 30

============ Perf. Report: Initialization Latency
[NOTE] All time-values are in USEC*

min : 98734.976
max : 98734.976
average : 98734.976
99.99-pct : 98734.976
Std. Deviation : 0.000
Num. of Samples : 1

*============ Perf. Report: Submission Latency

[NOTE] All time-values are in USEC*

min : 186.560
max : 248.640
average : 197.636
99.99-pct : 248.640
Std. Deviation : 9.699
Num. of Samples : 100

*============ Perf. Report: Execution Latency

[NOTE] All time-values are in USEC*

min : 21717.920
max : 28593.504
average : 22216.808
99.99-pct : 28593.504
Std. Deviation : 1102.502
Num. of Samples : 100

Encode FPS achieved: 45.0110

Total encoded frames = 100, avg. bitrate=-2066718004, maxBitrate=-1991250968, minBitrate=80530672

ENCODING PROCESS ENDED SUCCESSFULY
total failures: 0

What would be your recommendation for optimizing the rig file or decoder configuration to minimize latency and prevent frame drops? Regarding the rig file for recordings, I am currently only aware of the setting that allows for capturing every nth frame.