We are experiencing an issue capturing from a camera sensor on a Jetson Orin NX (BSP 35.5.0). When capturing via the GStreamer nvarguscamerasrc plugin, the following error appears in the kernel log:
[RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=547215847104 sof_ts=547281280832 gerror_code=2 gerror_data=400060 notify_bits=20000"
What can we do about this? Based on our research online, this error can occur if the data from the sensor arrives with incorrect timing. It is worth noting that this issue does not occur on another identical device with the same setup.
— 🤖 This is an automated AI response. I’m here to help, but please verify important details! —
To address the issue of frame capture failure and kernel RCE errors on the Jetson Orin Nano or Jetson Orin NX, several steps and considerations can be taken based on the provided context:
Verify Camera Compatibility and Configuration: Ensure that the camera sensor is compatible with the Jetson Orin Nano or NX and that it is properly configured. This includes checking the device tree configuration and ensuring that the camera settings match the sensor’s capabilities.
Check Kernel and Device Tree Versions: The issue might be related to specific kernel or device tree versions. Ensure you are using a compatible and updated version of the kernel and device tree for your Jetson device.
Adjust Camera Settings: The error message suggests a timing issue with the data from the sensor. Adjusting the camera settings, such as the frame rate or exposure, might help synchronize the data flow and resolve the error.
Inspect Power and Reset GPIO Lines: Proper power and reset signaling to the camera is crucial. Verify that the GPIO lines used for camera power and reset are correctly configured and functioning.
Update or Reflash JetPack: Sometimes, updating JetPack to the latest version or reflashing the device can resolve issues related to compatibility or bugs in the software stack.
Consult NVIDIA Documentation and Forums: The NVIDIA developer forums and official documentation are valuable resources for troubleshooting specific errors and finding solutions or workarounds provided by the community or NVIDIA support.
Consider Hardware Issues: Given that the issue does not occur on another identical device with the same setup, there might be a hardware-related problem with the affected device. Checking for any hardware faults or differences in configuration between the devices is necessary.
Review DTS and Kernel Configuration: For custom or specific camera configurations, reviewing the device tree source (DTS) files and kernel configuration can provide insights into potential issues or necessary adjustments for proper camera operation.
Given the information and steps outlined above, if the issue persists, it may be beneficial to consult the NVIDIA Jetson forums or contact NVIDIA support directly for more specific guidance tailored to your setup and configuration.
Supported Content: The provided context supports troubleshooting steps for camera capture issues on Jetson devices, including checking camera compatibility, adjusting settings, and ensuring proper power and reset signaling.
Not Covered: Specific solutions or patches for the RCE error on the Jetson Orin NX (BSP 35.5.0) are not detailed in the provided context. For detailed troubleshooting or patches specific to this error, consulting NVIDIA’s official resources or support channels is recommended.
— We support team will also look into your issue to have further suggestions soon. please stay tuned. —
Hi, I have applied the patches to the kernel, replaced the bootloader partition — camera-rtcpu-t234-rce.img, and the libnvfusacap.so library. Now, the issue seems to be ignored, and GStreamer doesn’t stop, but random purple rectangular areas appear on the video.
Moreover, the first launch after a reboot runs fine (sometimes even a few launches in a row), but all subsequent runs result in these purple artifacts. (Prior to these fixes, GStreamer would stop after a few frames, throwing the error I mentioned earlier). It is worth noting that another identical device does not experience this issue.
What could be the issue here? Maybe I need to enable tracing somewhere to understand why this is happening. Could this be a hardware problem? When I enable raw frame capture via v4l2-ctl and then playback the frame sequence, the image is dark and green, but I don’t see any artifacts on it
Hi, I have collected a pair of logs from 2 devices (*_boot.log taken right after boot). Various errors appear only when starting the video. During the capture process itself, there are no errors in the logs. The startup errors are identical (excluding process IDs and timestamps). The GMAX3405 sensor driver and the DTB are identical (the kernel is built on version 1.156 with patches applied). We tried swapping the motherboard, the sensor, and the processor. On the 1.156 build, random purple rectangles appear, as shown in the screenshot above (these are rectangular areas stretching across the full width, appearing at random vertical positions and varying in height).
There is nothing in the kernel logs (previously there were errors that I mentioned at the beginning of the post, or rather, a single error for each gstreamer start). Is it possible that we need to tweak some configuration parameter to eliminate the hardware impact on this specific build?
The most interesting part is why it works fine the first time, but then consistently produces artifacts.
If we record the video to a file in raw NV12 format, the same purple artifacts are present. If we simply pipe the GStreamer output to fakesink, GStreamer stops after outputting a few frames (this happened before applying the patches, when the error mentioned above occurred due to some synchronization issue).
Hi! The kernel log shows that the exposure is set only at the start of the capture (the sensor driver logs all adjustments to the settings). The GStreamer command parameters specify not to change the exposure. The commands are identical on both devices.
What confuses me is that the failures consistently start after the first stop. It seems like something inside the ISP is either not resetting properly or doesn’t have enough time to do so in this particular build (due to hardware specifics). But how can I figure this out ?
I’ve enabled all the diagnostics you mentioned, but there is way too much data. I managed to save a small snippet of the log during reproduction to an archive. There were no purple artifacts at that exact moment (the time frame is too short, otherwise the log gets huge). Is there anything I can disable to gather more relevant data, or is this enough to pinpoint the issue?
The log class:CORRECTABLE_ERR type:STREAM_VC ... status:0x00000004 means bit 2 is also set in NVCSI_STREAM_0_CORRECTABLE_ERR_INTR_STATUS_VC0_0, i.e. correctable packet payload CRC error on VC0 (err_intr_stat_pd_crc_err_vc0).
So your snippet is essentially showing repeated payload CRC failures on stream 0 / VC0.
Most likely causes:
signal integrity / settle-time issues on the CSI link,
I repeated the log collection on the working device, and there is indeed not a single instance of the specified error (I collected the logs multiple times).
At the same time, the sensors were swapped between the devices, and the driver along with the DTS are identical, as I mentioned before. If the frames are corrupted (and perhaps they are being discarded), why do these purple rectangles appear? Why aren’t they just dropped? (Even though during the capture via v4l2-ctl, the frames arrived strictly at the specified interval without any drops)?
It is also unclear why the image is perfectly fine during the first launch (for any length of time), but artifacts appear during subsequent launches. Is there any parameter in the device tree that could be slightly adjusted to resolve this issue, or is this strictly a hardware failure?
Hi again! Here are the logs for frame capture via v4l2 and Argus.
When capturing via Argus right after a system reboot, there are no errors at all. However, if I restart the capture via Argus a second time, the logs show the exact same errors, resulting in either purple artifacts or GStreamer crashing immediately.
If I run the capture via v4l2-ctl, the errors disappear, and launching GStreamer right after v4l2-ctl also works perfectly fine with clean logs. The root cause seems to be an incorrect reset or shutdown of the ISP, which is somehow recovered by a quick capture initialization via v4l2-ctl. Is there a specific parameter in CSI, VI, or ISP that needs to be adjusted so that stopping the stream doesn’t corrupt subsequent data transmissions?
Now the capture via Argus works perfectly, and I no longer observe any artifacts or freezes. But why does this happen? Why does it always work fine the first time after a reboot, and why does everything work normally on at least two other devices even with the power management mode set to “auto”?
Are these errors related to the RTCPU firmware? Have they been fixed in BSP versions 35.6.*?
Can these errors manifest due to small variations in hardware parameters? Capture via v4l2-ctl works perfectly fine across all devices.
What’s the PHY mode? CPHY/DHPY? What’s the pix_clk_hz
Maybe review the pix_clk_hz for the deskew calbration.
Skew calibration is required if sensor or deserializer is using DPHY, and the output data rate is > 1.5Gbps.
An initiation deskew signal should be sent by sensor or deserializer to perform the skew calibration. If the deskew signals is not sent, the receiver will stall, and the capture will time out.
You can calculate the output data rate with the following equation:
Output data rate = (sensor or deserializer pixel clock in hertz) * (bits per pixel) / (number of CSI lanes)
Hi, according to the sensor documentation, the mode is D-PHY, the output frequency per MIPI lane is 600 MHz, 4 lanes, and the data (12 bit per sample) is transmitted on both the rising and falling edges.
Here are several parameters from the sensor device tree that might be specified incorrectly:
phy_mode = “DPHY”,
discontinuous_clk = “yes”,
cil_settletime = “0”,
line_length = “2592”,
pix_clk_hz = “400000000”,
active_w = “2448”,
active_h = “2048”
pix_clk_hz computed from external clock frequency with sensor PLL settings:
pix_clk_hz = 600 * 2 * 4 / 12 - number of pixels per second ( double rate )
I am not sure how the following parameters might affect the ISP operation:
discontinuous_clk: there is no mention of this in the sensor documentation; a massive register sequence is sent to the sensor when the stream starts;
cil_settletime: by default, this is probably calculated from the pix_clk_hz value;
line_length: It is unclear what value should be here (currently it is set to the maximum number of pixels the sensor can output according to its documentation). At startup, the sensor is programmed to output a 2448x2048 window.
