FTDI Device not recognized on Jetson

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
1

I am attempting to use a camera that uses a USB interface board with an FTDI chip to handle serial communication.

This is the usb interface board: USB HDMI camera interface board | Product

The camera with the board works well in the Xavier NX, I can stream HD video at 30FPS, and can use UVC to control a few parameters like zoom level and auto exposure.

But to control the whole range of parameters, I need the virtual serial device provided by the FTDI chip on the board.

This works fine in a Ubuntu 20.04 x86_64 desktop, but I canā€™t get it to work on the Xavier NX with a J202 recomputer board.

I have stopped all the other software accessing the camera to ensure no conflicts caused by UVC.

With lsusb I can see the device:

Bus 002 Device 003: ID 0403:602a Future Technology Devices International, Ltd USB3.1 Hub             
Bus 002 Device 002: ID 2109:0822 VIA Labs, Inc. USB3.1 Hub             
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:2822 VIA Labs, Inc. USB2.0 Hub             
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

But the SDK from the board manufacturer cannot communicate with the board.

I went directly to the FTDI site to use their SDK: D3XX Drivers - FTDI

Downloaded the arm v8 SDK https://ftdichip.com/wp-content/uploads/2023/06/libftd3xx-linux-arm-v8-1.0.14.tgz

Built the examples and run. With super user to ensure device permissions, but I always get no devices available:

$ sudo LD_LIBRARY_PATH=. ./FTD3xx_Notification 
FTD3xx_Notification.c:370: Attempting to open FT60X device...
FTD3xx_Notification.c:152: ERROR: FT_Create failed (DEVICE_NOT_FOUND)
$ sudo LD_LIBRARY_PATH=. ./streamer 3 1 1
Driver version:1.0.14
Library version:1.0.26
Total 0 device(s)

I even built the x86_64 libraries inside an emulated docker in the Xavier NX with access to the device, to make sure that the problem was not related to the armv8 SDK. And the same code that worked on the x86_64 desktop, didnā€™t work when emulated in the jetson.

3

Hi,
Not sure but probably the driver does not support arm64. We would suggest check and confirm this with the device vendor.

4

The vendor has confirmed that they have used it on armv8 devices with Ubuntu ranging from 18.04 to 22.04.

I had no trouble compiling or executing with valgrind, which would indicate some binary incompatibility.

Please notice that I also ran an emulated version of x86_64 and it did not work either on the Jetson. But on the desktop PC with the same camera, board and cable ran without issues.

5

Is the above the device in question? If so, can you provide these logs?

  • Monitor ā€œdmesg --followā€, and note what new log lines occur as a result of plugging in the USB connector (I am assuming USB is on the Jetson side).
  • lsusb -t 2>&1 | tee log_tree.txt
    (this should be performed with the device plugged in)
  • sudo lsusb -d 0403:602a -vvv 2>&1 | tee log_verbose.txt
    (this should be performed with the device plugged in)
  • lsmod 2>&1 | tee lsmod.txt
    (this should be performed with the device plugged in)
  • ls /dev/ttyUSB* 2>&1 | tee ttyUSB.txt
    (this should be performed with the device plugged in)
6

Yes it is that device.

dmesg --follow

[337288.040201] tegra-xusb 3610000.xhci: Firmware timestamp: 2022-03-16 11:07:43 UTC, Version: 60.13 release
[337288.450625] usb 2-3.3: new SuperSpeed Gen 1 USB device number 14 using tegra-xusb
[337288.471467] usb 2-3.3: New USB device found, idVendor=0403, idProduct=602a, bcdDevice= 0.01
[337288.471485] usb 2-3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[337288.471494] usb 2-3.3: Product: Active Silicon Harrier S9500-2
[337288.471503] usb 2-3.3: Manufacturer: Active Silicon
[337288.471512] usb 2-3.3: SerialNumber: C-1696329785-SY9500-2
[337288.477866] uvcvideo: Found UVC 1.10 device Active Silicon Harrier S9500-2 (0403:602a)

lsusb -t 2>&1 | tee log_tree.txt

/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
|__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
|__ Port 3: Dev 14, If 0, Class=Video, Driver=uvcvideo, 5000M
|__ Port 3: Dev 14, If 1, Class=Video, Driver=uvcvideo, 5000M
|__ Port 3: Dev 14, If 2, Class=Vendor Specific Class, Driver=, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
|__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M

lsmod 2>&1 | tee lsmod.txt

Module Size Used by
xt_nat 16384 0
xt_tcpudp 16384 0
veth 28672 0
xt_conntrack 16384 1
xt_MASQUERADE 16384 1
nf_conntrack_netlink 45056 0
nfnetlink 20480 2 nf_conntrack_netlink
xt_addrtype 16384 2
iptable_filter 16384 1
iptable_nat 16384 1
nf_nat 45056 3 xt_nat,iptable_nat,xt_MASQUERADE
nf_conntrack 131072 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
nf_defrag_ipv6 24576 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 2 nf_conntrack,nf_nat
br_netfilter 32768 0
fuse 118784 5
lzo_rle 16384 36
lzo_compress 16384 1 lzo_rle
zram 32768 6
overlay 114688 0
snd_soc_tegra186_asrc 36864 1
snd_soc_tegra210_ope 32768 1
snd_soc_tegra186_arad 24576 2 snd_soc_tegra186_asrc
snd_soc_tegra210_iqc 16384 0
snd_soc_tegra210_mvc 20480 2
snd_soc_tegra186_dspk 20480 2
snd_soc_tegra210_afc 20480 6
snd_soc_tegra210_dmic 20480 4
snd_soc_tegra210_adx 28672 4
snd_soc_tegra210_mixer 45056 1
snd_soc_tegra210_amx 32768 4
snd_soc_tegra210_admaif 118784 1
snd_soc_tegra210_i2s 24576 6
snd_soc_tegra_pcm 16384 1 snd_soc_tegra210_admaif
snd_soc_tegra210_sfc 57344 4
aes_ce_blk 36864 0
crypto_simd 24576 1 aes_ce_blk
cryptd 28672 1 crypto_simd
aes_ce_cipher 20480 1 aes_ce_blk
ghash_ce 28672 0
sha2_ce 20480 0
sha256_arm64 28672 1 sha2_ce
sha1_ce 20480 0
snd_soc_spdif_tx 16384 0
uvcvideo 102400 0
videobuf2_vmalloc 20480 1 uvcvideo
snd_soc_tegra_machine_driver 16384 0
leds_gpio 16384 0
max77620_thermal 16384 0
snd_soc_tegra210_adsp 753664 1
snd_soc_tegra_utils 28672 3 snd_soc_tegra210_admaif,snd_soc_tegra_machine_driver,snd_soc_tegra210_adsp
snd_soc_tegra210_ahub 1228800 3 snd_soc_tegra210_ope,snd_soc_tegra210_sfc
snd_soc_simple_card_utils 24576 1 snd_soc_tegra_utils
nvadsp 110592 1 snd_soc_tegra210_adsp
tegra_bpmp_thermal 16384 0
tegra210_adma 28672 2 snd_soc_tegra210_admaif,snd_soc_tegra210_adsp
snd_hda_codec_hdmi 57344 4
snd_hda_tegra 16384 0
snd_hda_codec 118784 2 snd_hda_codec_hdmi,snd_hda_tegra
realtek 24576 1
userspace_alert 16384 0
snd_hda_core 81920 3 snd_hda_codec_hdmi,snd_hda_codec,snd_hda_tegra
nv_imx219 20480 0
spi_tegra114 32768 0
loop 36864 1
binfmt_misc 24576 1
ina3221 24576 0
pwm_fan 24576 0
nvgpu 2494464 20
nvmap 192512 62 nvgpu
ip_tables 36864 2 iptable_filter,iptable_nat
x_tables 49152 7 xt_conntrack,iptable_filter,xt_tcpudp,xt_addrtype,xt_nat,ip_tables,xt_MASQUERADE

ls /dev/ttyUSB* 2>&1 | tee ttyUSB.txt

ls: cannot access ā€˜/dev/ttyUSB*ā€™: No such file or directory

Notice that on my x86_64 desktop thereā€™s no /dev/ttyUSBX either and I can read data.

I did some more testing, I can send commands to the camera, they are received and executed, but I can never get a reply on the jetson, but I can on the x86_64 desktop machine.

EDIT:
Additionally, Iā€™ve seen that in the x86_64 system I can send and receive commands while streaming, whereas in the Xavier NX sending commands interrupts the video transmission and I still get no answer.

7

A summary of some notes:

  • The dmesg log on insert says that the device plugin succeeded, and that the hotplug associated the standard USB video class driver to at least part of it.
  • The tree log says you now have two cameras, e.g., a stereo camera, or two cameras on one HUB.
  • One device is a vendor specific class, meaning that it needs a custom driver. This is not a standard USB class. Perhaps this is the control of the cameras?
  • Your earlier FTDI listing was this:
    Bus 002 Device 003: ID 0403:602a Future Technology Devices International, Ltd USB3.1 Hub
    ā€¦this is a HUB, not a serial UART.
  • I see the uvcvideo module loaded, this handles the cameras (but not control of cameras).

So I am thinking the custom device is the UART, and that it is not FTDI. FTDI is a HUB. However, that ā€œcustomā€ device might really be FTDI underneath everything. A lot of manufacturers use a serial UART from FTDI, but arrange for it to list as their device, and thus the driver does not associate with the FTDI driver. One would need to install a driver that is not really a driver. To explain this requires mentioning how udev works.

Everything which is ā€œhot plugā€, in which devices can identify themselves, e.g., USB and PCI, are examined by udev upon plugin. Things like manufacturer ID and device class are examined. A lot of devices report themselves as being compliant with some standard drivers, e.g., the uvcvideo driver is not a USB driver so much as it is a driver known to handle devices reporting themselves as compatible with that standard. The FTDI driver is custom, but it is so common that if a device reports itself as being an FTDI serial UART, udev will normally associate the UART to the driver without any help.

Manufacturers will often change the hot plug device description to ā€œbrandā€ their product, and then supply a custom driver which isnā€™t really a driverā€¦what it does is to associate the hot plug information to the standard driver or the common driver which is probably already present. It might be that the ā€œVendor Specific Classā€ device is an FTDI serial UART, but that udev needs to be told that this is the case.

If Iā€™m correct about the above, then the camera manufacturer needs to supply a driver (which isnā€™t really a driver, but is instead just a udev customization to tell the hot plug to associate this with the FTDI driver).

If the FTDI driver loads for a serial UART, then this is what produces the ā€œ/dev/ttyUSB*ā€ files (udev can actually change the naming of those resultant device special files as well). This idea that the manufacturer customized a standard FTDI device matches well with the lack of ā€œ/dev/ttyUSB*ā€ on the host PC; udev must be renaming the device special file, which is likely part of what the extra ā€œdriverā€ (udev trigger) is.

A udev trigger is likely architecture independent. Is there some sort of custom driver or software associated with this camera/UART (it would be for control, and not for the video itself)?

8

The only rules provided are these, which are for not requiring sudo to access the device:

SUBSYSTEM=="usb", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="601e", MODE="0666"
SUBSYSTEM=="usb", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="601f", MODE="0666"
SUBSYSTEM=="usb", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="602a", MODE="0666"
SUBSYSTEM=="usb", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="602b", MODE="0666"
SUBSYSTEM=="usb", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="602c", MODE="0666"
SUBSYSTEM=="usb", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="602d", MODE="0666"
SUBSYSTEM=="usb", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="602f", MODE="0666"

Regarding these 2 video devices, I have seen this with other cameras, depending on kernel version, and I believe it is related to this: Multiple /dev/video for one physical device - Unix & Linux Stack Exchange

I understand what you are explaining, in fact I managed to force a ttyUSB to appear by executing
echo 0403 602a > /sys/bus/usb-serial/drivers/ftdi_sio/new_id

As indicated here: How do I add a custom PID to the ftdi_sio Linux COM port driver? - FTDI

But this ttyUSB0 device is not usable by the libraries provided by the camera manufacturer or FDTI.

Also keep in mind that on the Ubuntu 20.04 x86_64 none of this is necessary, the ttyUSB* device doesnā€™t appear either and the code can send/receive commands to the camera.

While on the Xavier NX I can send, but not receive.

9

The point here is that we know the FTDI chipset would run with defaults. However, the manufacturer of the USB devices has probably made modifications to customize what device special file is used. That means the default udev rules have to be amended to detect this exact device, and make changes which the camera software depends on (such as a name other than ā€œ/dev/ttyUSB#ā€). On a desktop PC, where this camera works, can you find anything for that idVendor within the udev system which might create or alter a name of a device? I think this is what the vendor libraries are looking for (something with a custom name which in turn would exist because the manufacturer probably provides a custom udev naming rule).

10

Thereā€™s no difference in the created devices on the desktop and jetson:

I ran ls /dev before and after the camera is plugged on both systems. On both systems thereā€™s only these newly added devices:

 /dev/media0 (media1 on jetson)
/dev/video0
/dev/video1
/dev/v4l/by-path/platform-3610000.xhci-usb-0:3.3:1.0-video-index0
/dev/v4l/by-path/platform-3610000.xhci-usb-0:3.3:1.0-video-index1
/dev/v4l/by-id/usb-Active_Silicon_Active_Silicon_Harrier_S9500-2_C-1696329785-SY9500-2-video-index0
/dev/v4l/by-id/usb-Active_Silicon_Active_Silicon_Harrier_S9500-2_C-1696329785-SY9500-2-video-index1
11

Do you have set SW1-7=on, SW1-6=off SW1-5=off ?

12

Yes, all off except SW1-7.

13

Maybe itā€™s easier to use and FT2232D mor FT2232H and use the TTL serial ports on the camera interface boards.

14

It might be easier to use the Serial interface indeed.
But we are building a product with this, and weā€™d like to minimize assembly steps and leave the TTL pins free for other applications that might require them.

The capability is there since itā€™s working fine on the desktop PC, we just need to get it running on the Jetson.

15

Does unplug/replug of the USB improve what is visible? You might want to monitor ā€œdmesg --followā€, and post the log of the unplug/replug. Maybe retriggering will make something show up which did not show up previously.

16

No, I have tried several unplug/replug and itā€™s always the same devices.

Hereā€™s the dmesg --follow log, the camera now appears as ā€œHarrier 40LHD-2-Testā€ because I am testing a configuration provided by the manufacturer, but the behavior was the same before I applied this configuration.

[691468.904724] usb 2-3.3: new SuperSpeed Gen 1 USB device number 24 using tegra-xusb
[691468.925670] usb 2-3.3: New USB device found, idVendor=0403, idProduct=602a, bcdDevice= 0.01
[691468.925686] usb 2-3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[691468.925695] usb 2-3.3: Product: Harrier 40LHD-2-Test
[691468.925702] usb 2-3.3: Manufacturer: Active Silicon
[691468.925710] usb 2-3.3: SerialNumber: 40LHD-2
[691468.944177] uvcvideo: Found UVC 1.10 device Harrier 40LHD-2-Test (0403:602a)
[691477.120705] tegra-xusb 3610000.xhci: entering ELPG done
[691485.378269] tegra-xusb 3610000.xhci: Firmware timestamp: 2022-03-16 11:07:43 UTC, Version: 60.13 release
[691485.709139] usb 2-3.3: USB disconnect, device number 24
[691487.978778] tegra-xusb 3610000.xhci: entering ELPG done
[691493.660596] tegra-xusb 3610000.xhci: Firmware timestamp: 2022-03-16 11:07:43 UTC, Version: 60.13 release
[691494.080348] usb 2-3.3: new SuperSpeed Gen 1 USB device number 25 using tegra-xusb
[691494.101121] usb 2-3.3: New USB device found, idVendor=0403, idProduct=602a, bcdDevice= 0.01
[691494.101135] usb 2-3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[691494.101173] usb 2-3.3: Product: Harrier 40LHD-2-Test
[691494.101185] usb 2-3.3: Manufacturer: Active Silicon
[691494.101193] usb 2-3.3: SerialNumber: 40LHD-2
[691494.117402] uvcvideo: Found UVC 1.10 device Harrier 40LHD-2-Test (0403:602a)
[691504.368335] tegra-xusb 3610000.xhci: entering ELPG done
[691506.425025] tegra-xusb 3610000.xhci: Firmware timestamp: 2022-03-16 11:07:43 UTC, Version: 60.13 release
[691506.756591] usb 2-3.3: USB disconnect, device number 25
[691509.015652] tegra-xusb 3610000.xhci: entering ELPG done
[691511.659015] tegra-xusb 3610000.xhci: Firmware timestamp: 2022-03-16 11:07:43 UTC, Version: 60.13 release
[691512.076152] usb 2-3.3: new SuperSpeed Gen 1 USB device number 26 using tegra-xusb
[691512.096868] usb 2-3.3: New USB device found, idVendor=0403, idProduct=602a, bcdDevice= 0.01
[691512.096884] usb 2-3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=7
[691512.096893] usb 2-3.3: Product: Harrier 40LHD-2-Test
[691512.096902] usb 2-3.3: Manufacturer: Active Silicon
[691512.096943] usb 2-3.3: SerialNumber: 40LHD-2
[691512.115322] uvcvideo: Found UVC 1.10 device Harrier 40LHD-2-Test (0403:602a)
[691516.060115] tegra-xusb 3610000.xhci: entering ELPG done
17

The UVC side is for camera sensors, and would not really count so far as the FTDI serial UART goes. However, it keeps mentioning firmware. Iā€™m wondering if maybe this needs some sort of firmware. If it does, some firmware uploads each time it runs, but that tends to be for Wi-Fi or cell devices. I would wonder though if this has a firmware update available? Does the support page for this device have any kind of firmware download?

Note: The 0x0403 idVendor is the FTDI device, so this part is not UVC, and is not a sensor. I see it working just fine, but no driver is seen loading. The implication is that either (A) the driver is not present, or (B) it is customized via udev before it is recognized as being an FTDI serial UART.

18

Do you know any way of having verbose udev logs to see whatā€™s going on in more detail when I plug the camera?

19

Are you sure there is an actual FTDI chip on the USB/HDMI interface board? I have never seen any references in de documentation. They say

ā€œCamera VISCA control from Linux
The Harrier Virtual Serial COM port is only supported on Windows. For Linux users we provide an API that enables Linux applications to send VISCA commands to the camera over the USB connection. This gives the Linux application full control of the camera. This API can be downloaded from the Active Silicon website and is available for PC and ARM Linux platforms. Note that the UVC video system expects to be in control of the camera video mode and frame rate, so these settings should not be changed using VISCA commands.ā€

1 Like
20

What you see in dmesg is what udev logs. You can manually trigger a udev detect event (making it detect again), but each USB connection does that already. Also, dmesg confirmed that udev triggered. The part which fails is an association of the device with a driver which handles the device (the USB drivers handle the data pipe, but the device itself must have its own driver which is separate from USB; it is this latter which fails). Actually, to say the driver fails is incorrect, because no driver attempted to load. It seems logical that if what @fchkjwlsq mentions is the case for your deviceā€¦a virtual interfaceā€¦then this would be why no driver associates. Normally any FTDI serial UART would ā€œjust workā€ since it is so common and rarely hits any kind of bug.

21

I am 100% positive they have an FTDI chip, in fact the FTDI support team is looking into the issue because they have not seen this on other armv8 boards.

Today I flashed a new Jetson Xavier NX with stock Jetpack 5.1.1 and the problem remains. Just to make sure there wasnā€™t some misconfiguration on my side.

What I fail to understand is who is supposed to create this virtual interface, and if I can detect that it has been created. On the x86 desktop there are different devices in /dev/.