NvMediaIEP FeedFrame issues

jmanning2 · July 23, 2024, 4:17pm

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hi, we’re seeing intermittent issues with calls to NvMediaIEPFeedFrame(). Abridged output from journalctl -b0 is attached below, but roughly the sequence of events appears to be as follows:

A call to NvMediaIEPFeedFrame() fails

[23166552785] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder

The kernel prints debugging information about NvSciSync?
Multiple subsequent NvMediaIEPFeedFrame() calls fail with an error message about host1x job submission failed and the same “FeedFrame failed” message as the first failing call

Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x 13e40000.host1x: cdma_timeout_handler: timeout: 284 (284-camera_driver_n), HW thresh 3715, done 3717
Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x-virtual-engine 13e40000.host1x:nvenc@154c0000: camera_driver_n: job submission failed: host1x job submission failed: -1
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23166988967] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x-virtual-engine 13e40000.host1x:nvenc@154c0000: camera_driver_n: job submission failed: host1x job submission failed: -1
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23167420768] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x-virtual-engine 13e40000.host1x:nvenc@154c0000: camera_driver_n: job submission failed: host1x job submission failed: -1
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23167853362] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x-virtual-engine 13e40000.host1x:nvenc@154c0000: camera_driver_n: job submission failed: host1x job submission failed: -1
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23168071461] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x-virtual-engine 13e40000.host1x:nvenc@154c0000: camera_driver_n: job submission failed: host1x job submission failed: -1
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23168511473] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x-virtual-engine 13e40000.host1x:nvenc@154c0000: camera_driver_n: job submission failed: host1x job submission failed: -1
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23168740817] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x-virtual-engine 13e40000.host1x:nvenc@154c0000: camera_driver_n: job submission failed: host1x job submission failed: -1
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23169189356] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu kernel: tegra-host1x-virtual-engine 13e40000.host1x:nvenc@154c0000: camera_driver_n: job submission failed: host1x job submission failed: -1

All subsequent NvMediaIEPFeedFrame() calls fail because the max number of pre-fences has been added to the NvMediaIEP instance

Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23170396994] Module_id 81 Severity 2 : ValidateTaskStatusArray numInputTaskStatus should be less than NVVIDEO_IEP_MAX_IN_TASKSTATUS
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23170399123] Module_id 81 Severity 2 : NvVideoEncodeFeedFrameForTEGRAV1 ValidateTaskStatusArray failed in IEP FeedFrame
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23170399394] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23173519119] Module_id 81 Severity 2 : ValidateTaskStatusArray numInputTaskStatus should be less than NVVIDEO_IEP_MAX_IN_TASKSTATUS
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23173521237] Module_id 81 Severity 2 : NvVideoEncodeFeedFrameForTEGRAV1 ValidateTaskStatusArray failed in IEP FeedFrame
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23173521548] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23176643021] Module_id 81 Severity 2 : ValidateTaskStatusArray numInputTaskStatus should be less than NVVIDEO_IEP_MAX_IN_TASKSTATUS
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23176645410] Module_id 81 Severity 2 : NvVideoEncodeFeedFrameForTEGRAV1 ValidateTaskStatusArray failed in IEP FeedFrame
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23176645728] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23179770518] Module_id 81 Severity 2 : ValidateTaskStatusArray numInputTaskStatus should be less than NVVIDEO_IEP_MAX_IN_TASKSTATUS
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23179773188] Module_id 81 Severity 2 : NvVideoEncodeFeedFrameForTEGRAV1 ValidateTaskStatusArray failed in IEP FeedFrame
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23179773495] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23182893185] Module_id 81 Severity 2 : ValidateTaskStatusArray numInputTaskStatus should be less than NVVIDEO_IEP_MAX_IN_TASKSTATUS
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23182895596] Module_id 81 Severity 2 : NvVideoEncodeFeedFrameForTEGRAV1 ValidateTaskStatusArray failed in IEP FeedFrame
Jul 23 09:45:34 tegra-ubuntu camera_driver_node[9975]: [23182896019] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23186018192] Module_id 81 Severity 2 : ValidateTaskStatusArray numInputTaskStatus should be less than NVVIDEO_IEP_MAX_IN_TASKSTATUS
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23186020986] Module_id 81 Severity 2 : NvVideoEncodeFeedFrameForTEGRAV1 ValidateTaskStatusArray failed in IEP FeedFrame
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23186021303] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23189143198] Module_id 81 Severity 2 : ValidateTaskStatusArray numInputTaskStatus should be less than NVVIDEO_IEP_MAX_IN_TASKSTATUS
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23189145445] Module_id 81 Severity 2 : NvVideoEncodeFeedFrameForTEGRAV1 ValidateTaskStatusArray failed in IEP FeedFrame
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23189145768] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23192268069] Module_id 81 Severity 2 : ValidateTaskStatusArray numInputTaskStatus should be less than NVVIDEO_IEP_MAX_IN_TASKSTATUS
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23192270278] Module_id 81 Severity 2 : NvVideoEncodeFeedFrameForTEGRAV1 ValidateTaskStatusArray failed in IEP FeedFrame
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23192270590] Module_id 42 Severity 2 : NvMediaIEPFeedFrame Failed to feed frame to NvVideoEncoder
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23195393517] Module_id 24 Severity 2 : sNvMSetNvSciSyncPreFenceWithTaskSlot Failed to set PreFence as the Max preFences are set already
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23198518181] Module_id 24 Severity 2 : sNvMSetNvSciSyncPreFenceWithTaskSlot Failed to set PreFence as the Max preFences are set already
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23201643323] Module_id 24 Severity 2 : sNvMSetNvSciSyncPreFenceWithTaskSlot Failed to set PreFence as the Max preFences are set already
Jul 23 09:45:35 tegra-ubuntu camera_driver_node[9975]: [23204771224] Module_id 24 Severity 2 : sNvMSetNvSciSyncPreFenceWithTaskSlot Failed to set PreFence as the Max preFences are set already

As best I can tell, the error messages about max prefences is because prefences are only cleared on a successful call to NvMediaIEPFeedFrame(). So because many of those calls have failed, we have been adding prefences that are not getting cleared. One question would be if there is some method to clear these prefences so that 16 failed calls to NvMediaIEPFeedFrame() don’t corrupt the state of the NvMediaIEP instance?

The more important question is, can Nvidia help determine any root cause for what those error messages about job submission might mean and if there are any ways we can determine how the encoder got into this state where all calls to NvMediaIEPFeedFrame() are failing?

Thank you.

2024-07-23.log (14.4 KB)

SivaRamaKrishnaNV · July 23, 2024, 6:09pm

Dear @jmanning2,
what is the return status from the NvMediaIEPFeedFrame ? Is it NVMEDIA_STATUS_ERROR ?

could you some more details about the use case or application pipeline to understand the route cause of the issue?

jmanning2 · July 25, 2024, 1:09pm

I believe the initial error codes are NVMEDIA_STATUS_BAD_PARAMETER followed by NVMEDIA_STATUS_NOT_SUPPORTED once we’ve reached the max number of prefences.

As far as our application, we’re using NvSIPL to read images: ICP/ISP0 buffers are read on CPU, ISP1/ISP2 buffers are passed to NvMediaIEP for video encoding.

Here’s some additional debugging info from our application logs:

WARN notification_type=NOTIF_WARN_ICP_FRAME_DROP
WARN notification_type=NOTIF_WARN_ICP_FRAME_DROP
WARN notification_type=NOTIF_WARN_ICP_FRAME_DROP
WARN [ISP0] Failed to get image: NVSIPL_STATUS_TIMED_OUT, current queue count: 0
WARN [ICP] Failed to get image: NVSIPL_STATUS_TIMED_OUT, current queue count: 0
WARN notification_type=NOTIF_WARN_ICP_FRAME_DROP
WARN notification_type=NOTIF_WARN_ICP_FRAME_DROP
WARN notification_type=NOTIF_WARN_ICP_FRAME_DROP
WARN [ISP0] Failed to get image: NVSIPL_STATUS_TIMED_OUT, current queue count: 0
WARN [ICP] Failed to get image: NVSIPL_STATUS_TIMED_OUT, current queue count: 0
WARN notification_type=NOTIF_WARN_ICP_FRAME_DROP
WARN notification_type=NOTIF_WARN_ICP_FRAME_DROP
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
NVENC_H265: Stream flush failed
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
NVENC_H265: Stream flush failed
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
NVENC_H265: Stream flush failed
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
NVENC_H265: Stream flush failed
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
NVENC_H265: Stream flush failed
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
NVENC_H265: Stream flush failed
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
NVENC_H265: Stream flush failed
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
NVENC_H265: Stream flush failed
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] FeedFrame() failed with: NVMEDIA_STATUS_BAD_PARAMETER
ERROR [ISP2] Set encoder pre-fence failed with: NVMEDIA_STATUS_NOT_SUPPORTED
ERROR [ISP2] Set encoder pre-fence failed with: NVMEDIA_STATUS_NOT_SUPPORTED
ERROR [ISP2] Set encoder pre-fence failed with: NVMEDIA_STATUS_NOT_SUPPORTED

SivaRamaKrishnaNV · August 5, 2024, 7:16pm

Dear @jmanning2,
Is it ok to use DW recorder for your usecase?. It fetches camera data and stores in raw/lraw/h264/h265/mp4 formats.

system · September 5, 2024, 5:21am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Disable ISP(Not use .nito) in sample_camera demo DRIVE AGX Orin General driveworks	16	683	June 26, 2024
No camera display when using nvsipl_camera on Group A DRIVE AGX Orin General driveos-nvmedia	8	490	May 9, 2024
The NvMedia sample applications occur build error DRIVE AGX Orin General driveos-nvmedia	7	1088	December 22, 2022
Jetson AGX Orin nvargus_nvraw doesn't show sensors Jetson AGX Orin camera	18	76	October 11, 2024
Driveworks Entron camera queue timeout DRIVE AGX Orin General driveworks	12	91	October 17, 2024
When the camera driver was upgraded from r35.4.1 to r35.5.0, a kernel crash occurred when vi could not receive data Jetson AGX Orin camera	9	238	September 20, 2024
Camera initiation returns error: nvccp_set_cam_unit_pwr_on failed status 1001 DRIVE AGX Xavier General driveworks-cameras	18	1211	January 6, 2023
Driverworks 5.10 on Orin 6.0.6, gives error of NOTIF_WARN_ICP_FRAME_DROP while recording raw videos using recording tool DRIVE AGX Orin General driveworks-cameras	10	1136	May 26, 2023
Driveworks sample_camera_usb failed DRIVE AGX Orin General driveworks-cameras	17	1011	June 22, 2023
What is the actural spec for Orin MIPI CPHY？ Jetson AGX Orin camera , nvbugs	29	1563	September 14, 2023

NvMediaIEP FeedFrame issues

Related topics