No camera display when using nvsipl_camera on Group A

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hello,

I use the following commands to run the display command

sudo systemctl stop gdm

sudo ./nvsipl_camera -c "IMX728_RGGB_CPHY_x4" --link-enable-masks "0x0001 0x0000 0x0000 0x0000" --enableRawOutput --disableISP0Output --disableISP1Output --disableISP2Output -d 1

The script runs and does not error out but there is no data on the monitor visually.
I also tried -d 0 but it did not help.

I know there is data on the bus, as when I run the FPS script I can see the value reported as 30.
sudo ./nvsipl_camera -c "IMX728_RGGB_CPHY_x4" --link-enable-masks "0x0001 0x0000 0x0000 0x0000" --enableRawOutput --disableISP0Output --disableISP1Output --disableISP2Output --showfps

Not sure if this is related but the Ubuntu Desktop GUI is not showing up either after I run sudo systemctl start gdm

There is a lot of inconsistency here as I just had this working on Friday, and I am unable to boot up the desktop GUI or see any camera data visually on the display.

Dear @chris.hoffman,
Can you check if wfdCreateDevice failed on drive agx orin - #15 by VickNV helps? If it does not help, can you restarting or re-flashing to see if it fixes?

Related section : Connecting the Cameras | NVIDIA Docs

That link to do sudo systemctl disable gdm and sudo systemctl stop gdm did not help in this case.
I was looking into var/log/syslog and sudo dmesg

In syslog I see this:

pid=1 uid=0 auid=4294967295 ses=4294967295 subj=u:r:kernel:s0 msg='avc:  denied  { stop } for auid=1000 uid=1000 gid=1000 path="/lib/systemd/system/gdm.service" cmdline="systemctl stop gdm" scontext=u:r:kernel:s0 tcontext=u:object_r:unlabeled:s0 tclass=service permissive=1
May  7 14:52:45 tegra-ubuntu kernel: [51230.963258]  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
May  7 14:52:49 tegra-ubuntu systemd[1]: Stopping GNOME Display Manager...
May  7 14:52:49 tegra-ubuntu gnome-shell[1722]: gnome-shell: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
May  7 14:52:49 tegra-ubuntu gnome-session[1703]: gnome-session-binary[1703]: WARNING: Lost name on bus: org.gnome.SessionManager
May  7 14:52:49 tegra-ubuntu gnome-session[1703]: gnome-session-binary[1703]: CRITICAL: We failed, but the fail whale is dead. Sorry....
May  7 14:52:49 tegra-ubuntu gsd-a11y-settin[1770]: Error releasing name org.gnome.SettingsDaemon.A11ySettings: The connection is closed
May  7 14:52:49 tegra-ubuntu gnome-session-binary[1703]: WARNING: Lost name on bus: org.gnome.SessionManager
May  7 14:52:49 tegra-ubuntu gsd-housekeepin[1782]: Error releasing name org.gnome.SettingsDaemon.Housekeeping: The connection is closed
May  7 14:52:49 tegra-ubuntu gnome-session-binary[1703]: CRITICAL: We failed, but the fail whale is dead. Sorry....
May  7 14:52:49 tegra-ubuntu gsd-sharing[1795]: Error releasing name org.gnome.SettingsDaemon.Sharing: The connection is closed
May  7 14:52:49 tegra-ubuntu gsd-print-notif[1769]: Error releasing name org.gnome.SettingsDaemon.PrintNotifications: The connection is closed
May  7 14:52:49 tegra-ubuntu gsd-screensaver[1794]: Error releasing name org.freedesktop.ScreenSaver: The connection is closed
May  7 14:52:49 tegra-ubuntu gsd-datetime[1784]: Error releasing name org.gnome.SettingsDaemon.Datetime: The connection is closed
May  7 14:52:49 tegra-ubuntu gsd-sound[1783]: Error releasing name org.gnome.SettingsDaemon.Sound: The connection is closed
May  7 14:52:49 tegra-ubuntu gsd-smartcard[1789]: Error releasing name org.gnome.SettingsDaemon.Smartcard: The connection is closed
May  7 14:52:49 tegra-ubuntu gsd-power[1771]: gsd-power: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
May  7 14:52:49 tegra-ubuntu gsd-media-keys[1773]: gsd-media-keys: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.
May  7 14:52:49 tegra-ubuntu gsd-wacom[1779]: gsd-wacom: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.

Is this the problem? Attaching both files just in case.
dmesg.txt (88.1 KB)
sys.txt (29.0 MB)

As I said above, the display script gets to this point, but there is no visualization on the monitor.

I can also see the fps script is working as expected:

EDIT: after a while I see this while the display script is still running

Message from syslogd@tegra-ubuntu at May  7 18:37:13 ...
 kernel:[ 1576.294676] Internal error: Oops: 96000004 [#1] PREEMPT_RT SMP

Message from syslogd@tegra-ubuntu at May  7 18:37:13 ...
 kernel:[ 1576.313094] Code: f9405a64 8b030002 b94020e1 dac00c42 (f863681b)

Update: After reflashing the Orin with 6.0.5 docker and placing the camera drivers to nvsipl_drv and nito files to nit
I am getting an error on fps script as well:

$ sudo ./nvsipl_camera -c "IMX728_RGGB_CPHY_x4" --link-enable-masks "0x0001 0x0000 0x0000 0x0000" --enableRawOutput --disableISP0Output --disableISP1Output --disableISP2Output --showfps
MAX96712: Revision 5 detected
MAX96712 Link 0: PHY optimization was enabled
nvsipl_camera: ERROR: NvSIPLCamera Init failed

nvsipl_camera: ERROR: Master initialization failed. status: 10

Is there a way to see the initialization log of the deserializer registers?
I see this on /var/log/syslog

May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : devices/MAX96712DeserializerDriver/cdi_max96712.c 2909
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : MAX96712: GMSL2 link lock not detected on link 0
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : devices/MAX96712DeserializerDriver/CNvMMax96712.cpp 247
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : MAX96712: MAX96712SetDefaults failed with NvMedia error 7
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/devblk/common/ddi/CNvMDevice.cpp 71
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Driver failed Init() call 10
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 820
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Deserializer init failed with SIPL error 10
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 564
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Device block init failed. Status: 10
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 221
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Device block action failed for action 1
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 223
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Device block action failed for block 0
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 255
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Failed to initialize device blocks
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/core/CNvMCamera.cpp 1195
May  8 03:49:17 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : DeviceBlock manager init failed
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 1051
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : CNvMDeviceBlock must be in the initialized state or the stopped state before attempting to deinitialize
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 585
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Device block deinit failed. Status: 6
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 221
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Device block action failed for action 4
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 223
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : Device block action failed for block 0
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 1051
May  8 03:49:18 tegra-ubuntu nvsipl_camera: Module_id 30 Severity 2 : CNvMDeviceBlock must be in the initialized state or the stopped state before attempting to deinitialize

Dear @chris.hoffman,
Do you have any other camera module from DRIVE AGX Orin Sensors & Accessories | NVIDIA Developer to test nvsipl_camera display to see if the issue is fixed after reflashing.
Regarding the recent error, we notice such errors were resolved after rebooting the target.
Did you try if rebooting the target fixes the issue. Please use poweroff/ poweron from aurix(or common_if_testapp -mcureset on tegra console) to reboot the target.
Also, check connecting camera to other port to see if issue is specific to camera port?

A hard reboot worked, i.e Turning off the Orin, disconnecting the Fakra cable, reconnecting and switching it back on. I would not say that this is a solution for sure, since we need this to work continuously.

Also, for the display issue, the fix was to ONLY connect a monitor with Display Port to Display Port. Is this the only supported connector of monitors for the Orin modules? If yes, it will help to add it as a note somewhere.

1 Like

Do you run this from minicom ?

Dear @chris.hoffman,
Yes, DP port is used to connect display monitor. common_if_testapp -mcureset need to be run on target to power cycle the target with out accessing aurix console.