Jetson Agx Xavier stuck on boot after CUDA installation

After Cuda installation and reboot everything was working great but then i rebooted it again and it is stuck in boot.
Last messages are:
[OK] Started PHS deamon.
[OK] Reached target Multi-User System .
[OK] Reached target Graphical Interface.
Starting update UTMP about System Runlevel Changes
[OK] Started update UTMP about System Runlevel Changes.
[Ok] Started Uuser Menager for UID 1001.

After that screen goes black for 2 sek and go back to messeges for 1 sek.
Any help?

It sounds like it is just a video issue. Can you still log in via ssh or serial console?

What was your method of installing CUDA? If the wrong version were installed it might break the video driver, which in turn would cause the repeated attempt to start the GUI, fail, and then try again over and over.

This might also be complicated by any kid of auto login. Is your user set for auto login?

I cannot connect via SSH also if I understand it corectly serial console is available via pins? If it is true this is no option for me.
I installed CUDA with the terminal command and not SDK Menager.
The user was in auto-login but if I remember correctly it actually asked for password last time it booted (already after CUDA installation) which was strange.

I actually managed to log in using serial console (I used mincom for that and usb ttyACM0). I’m sorry for my previous mistake. So does it mean that video driver is broken? Is there any way to fix it from console?

I disabled auto login by commenting out lines in /etc/gdm3/custom.conf.
I also spotted message that is printed out during boot:
[Failure] Failed to start NVIDIA Persistence Deamon.
See ‘systemctl status nvidia-persistenced.service’ for details.
I checked this command and this is the output:
Loaded: loaded (/lib/systemd/system/nvidia-persistenced.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2018-01-28 16:58:19 CET; 6min ago
Process: 5527 ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced/* (code=exited, status=0/SUCCESS)
Process: 5353 ExecStart=/usr/bin/nvidia-persistenced --verbose (code=exited, status=1/FAILURE)
temd[1]: nvidia-persistenced.service: Service hold-off time over, scheduling restart
temd[1]: nvidia-persistenced.service: Scheduled restart job, restart counter is at 5.
temd[1]: Stopped NVIDIA Persistence Daemon.
temd[1]: nvidia-persistenced.service: Start request repeated too quickly.
temd[1]: nvidia-persistenced.service: Failed with result ‘exit-code’.
temd[1]: Failed to start NVIDIA Persistence Daemon.

Could that be the problem?

Yes, that is the problem. You basically confirmed that it is just a GUI issue. The CUDA install on command line should not be an issue, but this probably also updated something in the Xorg driver or the NVIDIA driver which plugs in to the Xorg driver. There is a strong dependency for the ABI version in both Xorg and the modules which plug in to Xorg (the NVIDIA driver is a plug-in to the Xorg software). If one of them changed, or if the wrong one was linked in, then this would cause what you see.

It used to be that there was a specific GL/GLES related library from Xorg which was replaced by the NVIDIA version, and that the replacement got overwritten during updates with the Xorg version. Since going to the dpkg system I no longer know which files should be where, but if someone knows, then a fix might be as simple as copying a single file from the NVIDIA content into the Xorg content. I won’t be able to answer that for the newer releases.