Orin NX production issues

Autonomous Machines Jetson & Embedded Systems Jetson Orin NX
1

We have identified a production issue with the Jetson Orin NX 16GB. We use the e-cam24 CUNX camera with a custom driver that e-con Systems has made available to us. We have already received 83 Jetson for our series production. However, the processors are exhibiting instability, with 21 of them malfunctioning. We have already ordered another 147 pieces that we need for production.

We unpacked the Jetson Orin NX with great care and in compliance with ESD regulations. We plugged each Jetson into the Xavier NX developer kit and flashed it. Then we checked whether we received a camera image. With 21 pieces we got a frozen or a green picture. Even with repeated flashing, we could not fix the problem.

The hardware and the test setup was exactly the same for every test. Is such a problem known? How can we solve this problem? Such a high error rate is not acceptable to us and makes the system unstable.
We strongly assume that this is a production error as we had no problems with the Xavier NX last year. We urgently need to find a solution for this.

We are currently undertaking an intensive testing process for each processor, which is highly time-consuming and not sustainable in the long run. Our testing procedure is as follows: We only approve the Jetson for mass production if it passes our rigorous tests. Initially, we power up the Jetson and run DeepStream (video stream) for 2 minutes. After shutting the jetson down, we repeat this process ten times. Following these initial tests, we then run DeepStream continuously for 3 hours. It’s not uncommon for the camera to crash during the initial power tests. I use the ‘dmesg | grep -i csi’ command to detect the root of the problem. In some instances, the issue only surfaces after 2 hours of extended testing.

[ 1.571226] SCSI subsystem initialized
[ 2.204085] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 240)
[ 6.736645] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: context isolation disabled due to no IOMMU
[ 6.745925] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: initialized
[ 6.752379] tegra-camrtc-capture-vi tegra-capture-vi: subdev 13e40000.host1x:nvcsi@15a00000- bound
[ 6.761600] tegra-camrtc-capture-vi tegra-capture-vi: subdev 13e40000.host1x:nvcsi@15a00000- bound
[ 39.186506] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[ 39.204882] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0

We have been in communication with E-con systems for several months, initially suspecting an issue with their driver. After exhausting all possible solutions, it now appears to be a hardware issue. This error occurs both with GStreamer and OpenCV. We connected a USB camera for testing and did not find any problems with it. Are you aware of such a problem? What do you suggest we do with the faulty processors? This is a significant concern for us, as our company’s operations heavily rely on the stable and consistent performance of these processors.

e-CAM24_CUNX_JETSON_L4T35.3.1_25-APR-2023_R01_RC1_VERSION3.zip (189.1 KB)
ReadMe.txt (7.7 KB)

We are currently using jetpack 5.1.1

5

Hi,
Please confirm with the vendor whether the camera is supported on Orin NX module + p3509. By default the supported camera is Raspberry pi camera v2. For using other camera, you would need to port camera sensor driver and device tree per hardware design.

6

Yes, we have been in contact with e-consystems for several months. They developed us a constum driver for the camera and the Jetson Orin NX module. They are advertised as Nvidia Elite Partners. I therefore think that they know what they are doing and that they have checked whether the camera is suitable for the application. Otherwise they probably wouldn’t have developed a driver for us.

7

Was the driver set up for that carrier board? If the carrier board differs, then it is likely to need device tree changes. Even if the device tree is correct, if lane impedance changes, then signal quality might also be in question. So basically you need to check that (A) the device tree is valid for that module and carrier board combination, but then you must check (B) that the lane impedance is valid.

8

Yes the driver was developed for the Xavier NX Developer Kit. At the time of driver development, the Orin NX Developer Kit had not yet been released. What can be wrong with the device tree? How can we check whether the lane impedance is valid? Is there a difference in impedance between the Xavier NX and Orin NX developer kit?

9

Hi,
For information, do you use this camera module:
Global Shutter Camera for Jetson Xavier NX / TX2 NX / Nano

Orin NX is not listed so it may not be supported yet.

10

Yes that is correct. This camera is used by us. As I mentioned before, we have been working with the camera manufacturer for several months. So far we have used the camera for the Xavier NX. Since the new Orin processor was announced we have been in contact with the company and they have developed a custom driver for the use case.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.