Orin NX AV1 encoding limitations

razvanphp · January 28, 2025, 5:37pm

We have successfully tested AV1 streaming from gstreamer to Chrome browser via webrtcbin running on Orin NX 16GB.

Now the problems we are looking to solve:

It works with 2 x 1080p streams at 90 or even 120 fps, but when we switch to 2k / 2712x1538 at 90 fps one stream and 1080p the second, the nvv4l2av1enc does not provide a valid stream anymore, no particular errors in the logs. It works with 2k / 60 fps + 1080p 90 fps tho… According to the datasheet, it should be able to handle it: AV1 / UHP / 4K60 (1) | 4K30 (3) | 1080p60 (6) | 1080p30 (12) 750 MP/S
The encoder does not produce valid keyframes for Chrome browser, so at first PLI, the image freezes forever. How can we debug this further? We tried both profile=0 and 1.

Thank you!
R

DaneLLL · February 3, 2025, 2:16am

Hi,
For information, do you use Jetpack 6.2? Adn please check if you can replicate it by running a gst-launch-1.0 command. We would like to replicate it on developer kit and investigate further.

razvanphp · February 11, 2025, 5:03pm

We’re using JP 6.0 atm.

I was able to reproduce it in CLI with this pipeline:

gst-launch-1.0 --eos-on-shutdown nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM), width=2712, height=1538, framerate=90/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvv4l2av1enc maxperf-enable=1 bitrate=40000000 idrinterval=256 iframeinterval=256 ! 'video/x-av1, stream-format=(string)byte-stream' ! av1parse ! qtmux ! filesink location=/home/jetson/test.mp4

It does not run for more than 13 seconds and then error is thrown:

CONSUMER: Producer has connected; continuing.
NVMEDIA: Need to set EMC bandwidth : 8616000
NvVideo: bBlitMode is set to TRUE
Redistribute latency...
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: UNAVAILABLE
Additional debug info:
Argus Error Status
CONSUMER: ERROR OCCURRED
Execution ended after 0:00:13.379745504
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
^C^C^C(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Freeing pipeline ...
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Argus client is exiting with 4 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366)

Same command with h264 encoder instead, works just fine:

gst-launch-1.0 --eos-on-shutdown nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM), width=2712, height=1538, framerate=90/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvv4l2h264enc maxperf-enable=1 bitrate=40000000 idrinterval=256 iframeinterval=256 ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! qtmux ! filesink location=/home/jetson/test10.mp4

… or if we lower the FPS to 60 instead of 90, keeping the 2k resolution, it also works.

Is this expected? Can we fix it if it’s a bug?

Thank you!
R

DaneLLL · February 12, 2025, 2:32am

Hi,
We run the command on Orin NX developerkit/Jetpack 6.1 and do not hit the issue:

$ gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-raw,format=I420,framerate=90/1,width=1280,height=720' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420,width=2712,height=1538' ! nvv4l2av1enc ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

Could you try it on your system? Would like to know whether this command works at your end.

razvanphp · February 12, 2025, 3:59am

Yes, this works on our test Orin NX as well:

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 6969, dropped: 0, current: 89.69, average: 90.09
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 7015, dropped: 0, current: 90.42, average: 90.09
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 7060, dropped: 0, current: 89.91, average: 90.09
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 7106, dropped: 0, current: 90.06, average: 90.09
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 7151, dropped: 0, current: 89.83, average: 90.09
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 7197, dropped: 0, current: 90.04, average: 90.09
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:01:20.045500830
Setting pipeline to NULL ...
Freeing pipeline ...

but what conclusion can we draw from this?

I don’t expect it to be a problem with the cameras, as it works with identical nvarguscamerasrc + H264 encoder, right?

The strange thing is that this works as well, using the fakesink, why would it fail only with filesink and webrtcbin?

gst-launch-1.0 -v --eos-on-shutdown nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM), width=2712, height=1538, framerate=90/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvv4l2av1enc maxperf-enable=1 bitrate=40000000 idrinterval=256 iframeinterval=256 ! 'video/x-av1, stream-format=(string)byte-stream' ! av1parse ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

Here are the nvargus-daemon errors when the problem happens:

Feb 12 03:51:41 orin60 nvargus-daemon[4040]: Module_id 30 Severity 2 : (fusa) Error: InvalidState Status syncpoint signaled but status value not updated in:/capture/src/fusaViHandler.cpp 861
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: CAM: serial no file already exists, skips storing again=== gst-launch-1.0[5893]: CameraProvider initialized (0xffff79532250)CAM: serial no file already exists, skips storing againSCF: Error InvalidState:  Corr Error 8 Received for sensor 0 .. Continuing!
Feb 12 03:51:41 orin60 nvargus-daemon[4040]:  (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 652)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: Module_id 30 Severity 2 : (fusa) Error: InvalidState  propagating from:/capture/src/fusaViHandler.cpp 803
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: Module_id 30 Severity 2 : (fusa) Error: ResourceAlreadyInUse All captures are already pending, no idle captures available in:/capture/src/fusaViHandler.cpp 672
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: Module_id 30 Severity 2 : (fusa) Error: ResourceAlreadyInUse  propagating from:/capture/src/fusaViHandler.cpp 512
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function startCaptureInternal(), line 875)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCapture(), line 536)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 483)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1565)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1394)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error ResourceAlreadyInUse:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error ResourceAlreadyInUse: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 644)
Feb 12 03:51:41 orin60 nvargus-daemon[4040]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 430)
Feb 12 03:51:42 orin60 nvargus-daemon[4040]: SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
Feb 12 03:51:42 orin60 nvargus-daemon[4040]: SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
Feb 12 03:51:42 orin60 nvargus-daemon[4040]: SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 145)
Feb 12 03:51:42 orin60 nvargus-daemon[4040]: SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
Feb 12 03:51:42 orin60 nvargus-daemon[4040]: SCF: Error Timeout: Sending critical error event for Session 0
Feb 12 03:51:42 orin60 nvargus-daemon[4040]:  (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
Feb 12 03:51:50 orin60 nvargus-daemon[4040]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Feb 12 03:51:50 orin60 nvargus-daemon[4040]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc for sensor guid 0 count -2083027424
Feb 12 03:51:57 orin60 nvargus-daemon[4040]: SCF: Error InvalidState: 6 buffers still pending during EGLStreamProducer destruction (in src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
Feb 12 03:52:02 orin60 nvargus-daemon[4040]: waitForIdleLocked remaining request 2343
Feb 12 03:52:02 orin60 nvargus-daemon[4040]: waitForIdleLocked remaining request 2342
Feb 12 03:52:02 orin60 nvargus-daemon[4040]: waitForIdleLocked remaining request 2341
Feb 12 03:52:02 orin60 nvargus-daemon[4040]: waitForIdleLocked remaining request 2346
Feb 12 03:52:02 orin60 nvargus-daemon[4040]: waitForIdleLocked remaining request 2340
Feb 12 03:52:02 orin60 nvargus-daemon[4040]: waitForIdleLocked remaining request 2339
Feb 12 03:52:02 orin60 nvargus-daemon[4040]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 969)

DaneLLL · February 12, 2025, 6:30am

Hi,
Please run the command for a try:

$ gst-launch-1.0 --eos-on-shutdown nvarguscamerasrc sensor_id=0 ! 'video/x-raw(memory:NVMM), width=2712, height=1538, framerate=90/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12' ! nvv4l2av1enc maxperf-enable=1 bitrate=40000000 idrinterval=256 iframeinterval=256 ! 'video/x-av1, stream-format=(string)byte-stream' ! filesink location=/home/jetson/test.av1

Would like to see if eliminating qtmux works.

system · March 12, 2025, 7:24am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Argus stability on Orin NX Jetson Orin NX camera , gstreamer	28	1354	August 28, 2023
AGX Orin: nvargus-daemon: Error InvalidState Jetson AGX Orin camera	13	1379	February 9, 2023
Encoding speed Jetson AGX Orin encoder	18	976	March 26, 2024
Orin Nano error with IMX219 Camera Jetson Orin Nano camera , gstreamer	47	2958	June 21, 2023
Jetson AGX Orin nvargus_nvraw doesn't show sensors Jetson AGX Orin camera	18	78	October 11, 2024
Adquisition issue with Jetson Orin NX 16GB carried board J4012 and IMX477 camera Jetson Orin NX camera , kernel , gstreamer	8	85	October 7, 2024
Orin NX v4l2/gstreamer starting stream timeout error Jetson Orin NX camera , board-design	11	964	July 19, 2023
Nvargus-daemon: InvalidState Status syncpoint signaled Jetson Orin Nano camera , gstreamer	21	1036	August 23, 2023
Lingering issue 3x USB camera at 60fps 1280x720 Jetson AGX Orin camera , usb , gstreamer , nvbugs	20	607	February 7, 2024
Camera streaming failure on Orin NX Jetson Orin NX camera	25	1858	June 7, 2023

Orin NX AV1 encoding limitations

Related topics