Nvgetty udev issues between ZED SDK and UART GPS how to resolve?

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
,
21

It’s the same python program yes. I see it on command line and also in the error logs we gather.

Here are the following errors I get and how and when:

picocom -b 38400 /dev/ttyTHS1
picocom v2.2

port is        : /dev/ttyTHS1
flowcontrol    : none
baudrate is    : 38400
parity is      : none
databits are   : 8
stopbits are   : 1
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
nolock is      : no
send_cmd is    : sz -vv
receive_cmd is : rz -vv -E
imap is        : 
omap is        : 
emap is        : crcrlf,delbs,


FATAL: cannot open /dev/ttyTHS1: Permission denied

But when I use sudo it works, that is why we change permssions.

The GPS python program uses PySerial so the error is:

serial.serialutil.SerialException:[Errno 13] Could not openport /dev/ttyTHS1 [Errno 13] Permission denied: '/dev/ttyTHS1'
22

Is picocom possibly giving this error, and not your program? Are picocom and your program both trying to access /dev/ttyTHS1?

Before and after the error, can you verify the user/group of ttyTHS1 is the same, without change? I’m wondering if something unexpected is changing, or if perhaps the error message is just badly worded (e.g., lack of access is just saying permission denied, but is really some other failure).

It is hard to use strace with scripts, but if picocom has this issue when no other program is running, then you might be able to get a log of system calls:
strace -oTraceLog.txt picocom ...whatever options...

23

when the program doesn’t have access, it exits. Then i use picocom to debug the issue.

24

Is the program in question a scripted language, e.g., Python, or is it a compiled binary? I’d like to see an strace log, but if it is scripted, then it is harder to do that.

25

It’s python

26

Is it python2 or python3? If you examine the Python file itself, the top might have a “shebang” naming which python, e.g.:
```#!/usr/bin/python3`

Whichever python it is, use that, e.g., if it is “python3”, log strace like this in an empty subdirectory if you can (if you need to cd to a particular location, then that is ok, but several logs will be generated, and so an empty directory is more convenient):

strace -o trace_log.txt -ff /usr/bin/python3 <your program file's full path>

There will be at least one (and perhaps more, depending on threads and forked processes) log, name starting with “trace_log.txt.<pid>”, and more, named by PID if threaded. It’ll be a lot of output, and I don’t know which one will have the failure, but go ahead and attach all logs to the forum. This should point out the nature of the system call at the point of failure.

27

Attached are the logs with the application running and giving errors. Please let me know if you need me to kill the existing application and rerun this command.
debug.zip (200.1 KB)

28

Here is another set of debug logs, only for the zed sensor error.
debug_zed.zip (138.1 KB)

29

Neither log indicates any attempt to access any “/dev/tty*”. So the UART is unrelated to any process which was part of those logs. Are you sure those applications use ttyTHS1 (or any UART)?

I do see some permission denied, but I think it is unrelated to the application failing. I think these were GPU debug or performance control files which are supposed to be root-only access:

trace_log.txt.9925:faccessat(AT_FDCWD, "/dev/nvhost-dbg-gpu", R_OK|W_OK) = -1 EACCES (Permission denied)
trace_log.txt.9925:faccessat(AT_FDCWD, "/dev/nvhost-dbg-gpu", R_OK|W_OK) = -1 EACCES (Permission denied)
trace_log.txt.9925:openat(AT_FDCWD, "/dev/nvhost-prof-gpu", O_RDWR|O_CLOEXEC) = -1 EACCES (Permission denied)
trace_log.txt.9925:openat(AT_FDCWD, "/sys/kernel/debug/clock/emc/possible_rates", O_RDONLY) = -1 EACCES (Permission denied)

(maybe NVIDIA can confirm what I suspect: Those files are not used for normal execution and would not cause a regular application to fail)

Did you create an strace log specifically for a session when you saw it fail and named ttyTHS1 as a permission error? Or was the error message different?

30

running sudo lshw shows me that the kernel doesn’t know the driver:

 *-usb:0
          description: USB hub
          product: USB 2.0 Hub
          vendor: Standard Microsystems Corp.
          physical id: 2
          bus info: usb@1:2
          version: b.b3
          capabilities: usb-2.00
          configuration: driver=hub maxpower=2mA slots=2 speed=480Mbit/s
        *-usb UNCLAIMED
             description: Human interface device
             product: ZED-2i HID INTERFACE
             vendor: STEREOLABS
             physical id: 2
             bus info: usb@1:2.2
             version: 3.0a
             serial: 32142444
             capabilities: usb-2.00
             configuration: maxpower=150mA speed=12Mbit/s

I’ve tried updating and reinstalling the ZED SDK but have not yet resolved this issue.

31

It seems the UART is not involved in this, unless it is a UART over USB, but ttyTHS* files are all integrated UARTs. Are we really talking about USB or UART? Is the UART in question a non-integrated USB serial UART? Does the USB issue cause any kind of interference with the camera operation? I don’t actually see a UART issue, other than the earlier need to stop nvgetty and make sure the user is in group dialout.

If I am looking for a USB issue, then the trace logs are still useful, but I’ll need to search differently. In no case do I now see any kind of permission issue in those logs, so if there is still a permission issue, it isn’t from the UART (at least not any UART which would have been used in the programs which produced the strace logs).

A small sample of file opens for something related to USB is (the two trace subdirectories are searched here):

# egrep -i -R usb * | egrep -i '(openat.*= -1)'
debug/trace_log.txt.8865:openat(AT_FDCWD, "/usr/local/zed/lib/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug/trace_log.txt.8865:openat(AT_FDCWD, "/usr/local/cuda/lib64/tls/aarch64/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug/trace_log.txt.8865:openat(AT_FDCWD, "/usr/local/cuda/lib64/tls/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug/trace_log.txt.8865:openat(AT_FDCWD, "/usr/local/cuda/lib64/aarch64/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug/trace_log.txt.8865:openat(AT_FDCWD, "/usr/local/cuda/lib64/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug_zed/trace_log.txt.9925:openat(AT_FDCWD, "/usr/local/zed/lib/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug_zed/trace_log.txt.9925:openat(AT_FDCWD, "/usr/local/cuda/lib64/tls/aarch64/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug_zed/trace_log.txt.9925:openat(AT_FDCWD, "/usr/local/cuda/lib64/tls/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug_zed/trace_log.txt.9925:openat(AT_FDCWD, "/usr/local/cuda/lib64/aarch64/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
debug_zed/trace_log.txt.9925:openat(AT_FDCWD, "/usr/local/cuda/lib64/libusb-1.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Do note that an open failure can, and often is, the result of tracing for multiple locations, and is not necessarily a problem. There is a hierarchy of search locations, and failures just might be early parts of the search path. Of those items which were not found above, all of them were from files in the CUDA content of “/usr/local/cuda”. None of those would cause USB itself to disappear or change.

Is it safe to say that the UART issue and the USB issue are separate? I realize that you said fixing the UART permissions caused USB to go away, but this may just be coincidental to the program making it further in and it might not be caused by UART issues.

Incidentally, if you monitor “dmesg --follow”, does anything show up at the moment the USB goes away?

32

Hey @linuxdev let me clarify, so we make sure we are on the same page. We are discussing 3 different sensors: GPS (on ttyTHS1), the ZED2i camera (USB3) , the ZED2i sensors (-usb Unclaimed)

The ZED Camera works.
The GPS works with nvgetty disabled.

The only remaining issue is that the kernel doesn’t seem to know the correct driver for the ZED Sensor on -usb Unclaimed.

33

Do you have a reference as to which driver the sensor uses? USB is there, so I assume it is the driver itself and not USB which is missing. Also, if you can, do you have an exact model number of the sensor which helps in looking up the driver if you don’t know the exact driver requirement?

34

@linuxdev the ZED sensors require libusb and libhid

When using the ZED Diagnostic tool it also fails with libhid error: -6

35

The exact model of the sensor is ZED2i

36

Any library, e.g., libusb or libhid, is user space. There would also need to be a driver in kernel space. It seems possible it might just use the HID driver (which is not libhid, although related). Perhaps though access to libhid is an issue. What do you see from “dmesg --follow” if you plug in the sensor? What program would you use to test the sensor or see it work (a program where strace might be able to log it…plugging in is not something strace can log since it is kernel space)?

37

I found that stereolabs also has this:
/usr/local/zed/tools/ZED_Diagnostic

Can you run that tool and see if it has a hint? You might want to see how the diagnostic runs with and without sudo.

38 
[ 3689.277206] tegra-xusb-padctl 7009f000.xusb_padctl: power down UTMI pad 1
[ 3689.277229] usb 1-2: USB disconnect, device number 7
[ 3689.277233] usb 1-2.2: USB disconnect, device number 8
[ 3689.279333] soctherm: OC ALARM 0x00000001
[ 3690.018903] usb 2-1: USB disconnect, device number 3
[ 3690.202932] usb usb2: usb_suspend_both: status 0
[ 3702.529899] tegra-xusb-padctl 7009f000.xusb_padctl: power on UTMI pads 1
[ 3702.790633] usb 1-2: new high-speed USB device number 9 using tegra-xusb
[ 3702.811140] usb 1-2: New USB device found, idVendor=0424, idProduct=2512
[ 3702.811146] usb 1-2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3702.812043] hub 1-2:1.0: USB hub found
[ 3702.812112] hub 1-2:1.0: 2 ports detected
[ 3703.098687] usb 1-2.2: new full-speed USB device number 10 using tegra-xusb
[ 3703.121094] usb 1-2.2: New USB device found, idVendor=2b03, idProduct=f881
[ 3703.121100] usb 1-2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3703.121103] usb 1-2.2: Product: ZED-2i HID INTERFACE
[ 3703.121106] usb 1-2.2: Manufacturer: STEREOLABS
[ 3703.121108] usb 1-2.2: SerialNumber: 32142444
[ 3703.129041] hid-generic 0003:2B03:F881.0005: hidraw0: USB HID v1.11 Device [STEREOLABS ZED-2i HID INTERFACE] on usb-70090000.xusb-2.2/input0
[ 3703.299012] usb 2-1: new SuperSpeed USB device number 4 using tegra-xusb
[ 3703.320048] usb 2-1: New USB device found, idVendor=2b03, idProduct=f880
[ 3703.320053] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=4
[ 3703.320056] usb 2-1: Product: ZED 2i
[ 3703.320059] usb 2-1: Manufacturer: Technologies, Inc.
[ 3703.320062] usb 2-1: SerialNumber: OV0001
[ 3703.321465] uvcvideo: Found UVC 1.10 device ZED 2i (2b03:f880)
[ 3703.322576] uvcvideo 2-1:1.0: Entity type for entity ZED 2i was not initialized!
[ 3703.330019] uvcvideo 2-1:1.0: Entity type for entity ZED 2i was not initialized!
[ 3703.337428] uvcvideo 2-1:1.0: Entity type for entity Camera 1 was not initialized!
[ 3703.345282] input: ZED 2i as /devices/70090000.xusb/usb2/2-1/2-1:1.0/input/input4
[ 3886.769211] usb 1-2.2: reset full-speed USB device number 10 using tegra-xusb
[ 3886.799053] hid-generic 0003:2B03:F881.0006: hidraw0: USB HID v1.11 Device [STEREOLABS ZED-2i HID INTERFACE] on usb-70090000.xusb-2.2/input0
[ 3947.911013] usb 1-2.2: reset full-speed USB device number 10 using tegra-xusb
[ 3947.938165] hid-generic 0003:2B03:F881.0007: hidraw0: USB HID v1.11 Device [STEREOLABS ZED-2i HID INTERFACE] on usb-70090000.xusb-2.2/input0
[ 3947.911013] usb 1-2.2: reset full-speed USB device number 10 using tegra-xusb
[ 3947.938165] hid-generic 0003:2B03:F881.0007: hidraw0: USB HID v1.11 Device [STEREOLABS ZED-2i HID INTERFACE] on usb-70090000.xusb-2.2/input0
[22757.987258] soctherm: OC ALARM 0x00000001
39

I ran the ZED_Diagnostic tool 2 times here are the logs, i tried the USB C cable both ways and it still seems like it doesn’t want to connect propertly.
ZED_Diagnostic_Results.zip (3.3 KB)

40

with sudo:

The logs are attached.
ZED_Diagnostic_Results_sudo_1.json (5.1 KB)
ZED_Diagnostic_Results_sudo_2.json (5.1 KB)

The error still persists
image