Jetson NX ttyTHS0 and VI Issues

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
1

Hi,

Our product uses Jetson Nano 4GB version with SDK version 4.5.1 and everything works great,
We are now upgrading our product to use jetson Xavier NX 8GB version also SDK 4.5.1
With the Nano there are no issues at all.
We use the exact same software for both Nano and NX,
For video capture we use V4l2 captuing from a TC358743 device,
The driver seems to be working fine on both systems.

When running the same on the NX, at first everything runs great, exactly like the Nano just much faster,
but after running the NX for a while ( 10m - 30m ) we start to get strange issues,
All of the sudden THS0 of the NX start returing an IO error every time we attempt to write to it,

NvApp5[5110]: [SerialApi]:Failed to write data to the Serial Device: /dev/ttyTHS0: Input/output error (5).

I can say that closing the Serial port and reopening fixes the issue, but still we miss some data when the issue happens, what can cause this ? and how can we fix this.

Acctually this is the minor issue, our bigger issue is that sometime after the same amout of time (10m-30m)
we start to get the following error message over and over:
[ 296.024155] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 296.024329] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 296.024526] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 296.027253] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 298.584151] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 298.584359] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 298.584538] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 298.586868] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 301.144157] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 301.144356] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 301.144522] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 301.147715] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 303.704148] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 303.704328] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 303.704515] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 303.706808] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 306.264188] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 306.264387] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 306.264552] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 306.266785] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 308.824167] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 308.824364] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 308.824544] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 308.826834] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 311.384155] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 311.384360] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 311.384540] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 311.387790] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 313.944188] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 313.944370] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 313.944545] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 313.947773] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 316.504148] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 316.504349] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 316.504520] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 316.506797] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 319.064157] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 319.064355] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 319.064515] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 319.066771] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[ 321.624164] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 321.624359] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[ 321.624546] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[ 321.627145] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel

We do not get any indication about this issue in our software, when this happens we start receving one identical frame per second over and over and the error keeps persisting, even if we detect this we will need to restart the entire application and/or system which is not possible, How can we fix this issue ?

Just some more info:
our system is pretty demanding CPU and GPU wise, I dont know if this has somethig to do with these issue,
Just thought it imporant to state that we are straining the system quite a bit.

This is quite urgent so your prompt reply will be greatly appriciated.
Thanks You
Amir

3

Did you try have system run as performance mode by nvpmodel and jetson_clocks?

4

Hi Shane,

Since yesturday we have tried the following:
Upgrade to 4.6.2 since we saw some fixes in the VI driver.
We tried any power model possibe using nvpmodel, tried 0 to 5,
In all situation we eventualy get the issues.
we allways run jetson_clock to increase the clocks
we tried mode 5 which uses 510M for GPU and still got the issue.

Acctually since we suspected our harware we switched back to using the EVM,
We use the seeed NVIDIA® Jetson Xavier™ NX Developer Kit since it is the only one that as close to the original EVM that cannot be bought currently.
Even when using the EVM we get the exact same issues with the video

Amir

5

Could you clarify if only run the camera preview without any CPU/GPU heavy loading APP.

6

We did try to run only the capture driver without any load at all,
We use v4l2-ctl in order to capture frames and dump them to dev/null,
We have used the following command:
v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat=RG10 --stream-mmap --stream-count=1000000 --stream-to=/dev/null -d /dev/video0

it ran all last night (aprox 20000 seconds ~ 6 hours) and didnt have any issues,
it seems that the issues apper only when we start doing inference on the GPU and strain the CPUs,

7

Could you use taskset assign specific CPU core for VI capture and others CPU core for you APP.

8

Hi Shane,

your idea about using taskset helps us figure out the issue, so thank you very much for that,
we fixed all our issues, so feel free to close the this thread.
FYI
The VI issue that we have been experiencing happens when user mode doesnt return the video buffers back to the VI fast enough, so if our system was heavliy loaded our capture thread didnt manage to dequeue/queue all the captured buffers.
Our tc358743 driver captures at 50FPS and when the system is heavily loaded our capture thread manages to only dequeue/queue 40FPS so eventualy the VI drops frames which is expected from it, if this situation lasts for mode than 10 - 30 minutes the probelm happenes.
I can say that we use the same driver for our NANO plaform which has the same FPS drop when its heavliy loaded and there is no such VI capture issues there.

Thanks
Amir

1 Like
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.