Uartlog.txt (8.6 KB)
If you insist.This was the whole log we got from uart when error occured.It showed the TX2 rebooted and it failed to enter the kernel.
There is another log that recorded some infomation when it worked well.But I still noticed that there were some errors in the middle.I hope it is useful for your analysis.
testlog-tx2.txt (118.7 KB)
Hi,
But the first line I saw is you type the “sudo reboot”. Could you elaborate what is happening here? Is you that type the command to trigger the reboot? Or it is something else prior to the “sudo reboot”?
Also, in the testlog-tx2.txt, what were you trying to do for sudo kill -9 5976? It seems after this line, there is kernel stack dump and then reboot. Is this line what you are talking about? Or it is just a fine log?
How did you recover the system after it fails to boot into kernel? Do the cold boot?
How frequently does this issue happen?
Also, how many TX2 module do you have on your side? Are you able to reproduce this problem on every of them? Or only specific modules have this problem?
Thanks for giving me so many patience.
The “sudo reboot” wasn’t typed by manual.I did’t do that.The command “sudo reboot” maybe be triggered by else,for example ,our program. Actually I did’t know. but it just showed in the log.
[quote=“WayneWWW, post:23, topic:177344”]
Or it is something else prior to the “sudo reboot”?
[/quote]Nothing else,we just let the board ran. The only operations was that we connected the uart to monitor it output.
Cause we wanted to find if this error resulted from our main program “tx2”. "./tx2 " is our main program.
No,it isn’t. In the testlog-tx2.txt,the kernel stack dump happened suddenly when kill the process 5976.
So after that. we rebooted the board and disabled self-starting of our main program in the file “/etc/rc.local”.
It is not fine.But I don’t care that problem at this moment.
Both Pressing the “reset” button and cold boot.(power again)
sometimes two days ,sometimes a week.sometimes a month.
We have more than 200 TX modules in different project.Yeah,it has been used in real project. 10 percent or more have this problem. Not everyone has this problem. But it does happen frequently.And we are under big pressure now.
BTW ,I still don’t understand why the watchdog did’t work? Do I need to enable the PMIC-wdt?
Hi,
Just to clarify this problem. Please help confirm.
The error log and symptom is in Uartlog.txt where the system would reboot but get stuck in “MB2 done”. You don’t know why there is a “sudo reboot” command in the console, but it just reboot.
The WDT does not trigger in such case so the device gets stuck.
Yes ,it is problem.But we only got that once. Most of time,we could not get that.
yes,I don’t know.I think I need to check that again with my colleague.You know ,To check the code.I will let you know if I get more information.
Do you mean neither tegra-WDT nor PMIC-WDT will be trigger in such case.
Can you give us a log that this sudden reboot happens and gets no stuck? It sounds like this issue is not easy to reproduce on your side and you only have one reboot log so far, right?
There is no log that this sudden reboot happens and gets no stuck. Most of the time,when the error happens,we get nothing about the error log. We can just find that all the communication ways,including network ,uart and hdmi, didn’t work.We do only get one log that shows it reboot.but this issue reproduces some times.Just no log.
Is there any specific application running to trigger the device reboot over all the modules?
I am also thinking about that.We can reboot it by web or upper computer software.but no one does that when error happens.When we test,we just check that every two hours or more.Once we find that it don’t work normal ,we check if the uart records any error log.
Will it reboot if you just put it in idle state with basic linux running within hours?
Did you ever monitoring the tegrastats for thermal status ?
Security fuses are something you’d be very aware of if you had worked with them. They are for secure boot. When intentionally burned the boot content cannot be altered except by properly signed content (using a secret key).
I couldn’t tell you much about the watchdog not working. FYI, UART log is needed to know what was set up and how it was set up (or failing) from early in boot. The UART log actually records boot information from stages prior to Linux starting, whereas dmesg is only from the point where Linux runs. If you boot using a serial console log from another computer to see the boot, then the serial console program can create a session log. This means you would be able to attach a plain text file and not need to do screen shots (which are hard to work with and limited…they only show what is on the screen at the moment and not the entire boot…there is a lot of interest in what goes on prior to the error ever occurring, so concentrating around the error for a screen shot is only minimally useful…thus a lot of requests on the forum for UART logs).
It seems like it would not reboot if it is just in idle state within 72 hours.we did not test that for a long time.
And the error happens randomly.Two days,a week or longer.
I have never monitor the thermal status .
Thanks .I did not save a complete log for all the test process from power-on .According our test results,if you combine
and
,That is almost the complete log.You can find log that showed it booted normally in testlog-tx2. I will post the whole log if I am good luck to get the error log.
Actually, it is not easy to find out what is going on.
You have only one log, which has problem stuck in MB2 and fails to reboot, but you said generally there will be no such error log. Thus, I am not sure the correctness of this log.
Also, no other things get monitored and it is also on a custom board.
I would suggest you can also move the module to devkit and see if it also has such random reboot.
Could you please tell me which library API can get the ODMDATA in nvidia-l4t-bootloader? I want to enable PMIC-wdt by restalling nvidia-l4t-bootloader.deb. Is it possible to enable PMIC-WDT not through flashing processing?
it’s uint32_t tegrabl_odmdata_get(void); to get odmdata from br-bct.
nvidia-l4t-bootloader.deb is a debian package that is provided by NV. It only has fixed pattern as what provided by jetpack.
If you want to change the pmic-wdt through that debian package, then you need to modify the content of debian package.
But I guess you don’t know how to do that. The content is similar as update the BUP as below document.
Thanks,@JerryCHang and @WayneWWW.
#if defined(CONFIG_ENABLE_WDT)
tegrabl_wdt_enable(TEGRABL_WDT_LCCPLEX, 0, 0, TEGRABL_WDT_SRC_TSCCNT_29_0);
#endif
In other words,could you please help me to find where to enable the PMIC-WDT in cboot?
4.What is DENVER_WDT? Is it useful?
5. Even if we enable the PMIC-WDT, the problem still happens when we run batch tests.(We test 100 devices at an ambient temperature of 40 degrees Celsius ,and 2 of them repeated the problem). The PMIC-WDT seems to reduce the incidence of problems.So we still want to change the WDT config.
In order to solve the problem ,I check the tegrastats when these devices work normmally and find the temperature is too high.the temperature of some devices have reached 95.9 ℃. the lowest temperature is about 65℃.
So is it possible that this problem ,kernel-crashed,is caused by high temperature of device? Do you have any suggestion to check that?
Dump the log from uart. It will tell what is crashed when your error happens.
Also, you can directly check whether your wdt is enabled by reading the /proc/devicetree/chosen/plugin-manager/odmdata.
