Issue when using simultaneously usb audio device microphone and speaker through xhci driver

Hi,

I have a problem with a specific usb audio device.

The issue appears on all the 3 different TK1 boards that I have (from 3 different producers), releases R21.5 and R21.7.

The usb device has those interfaces:

/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
    |__ Port 3: Dev 10, If 0, Class=Hub, Driver=hub/7p, 480M
        |__ Port 6: Dev 11, If 0, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 11, If 1, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 11, If 2, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 11, If 3, Class=Human Interface Device, Driver=usbhid, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M


Playback:
  Status: Stop
  Interface 1
    Altset 1
    Format: S24_3LE
    Channels: 2
    Endpoint: 1 OUT (ASYNC)
    Rates: 48000, 48000, 48000

Capture:
  Status: Stop
  Interface 2
    Altset 1
    Format: S24_3LE
    Channels: 2
    Endpoint: 2 IN (ASYNC)
    Rates: 48000, 48000, 48000

The Speaker audio is ok when I play sound on the playback interface with this pipeline:

gst-launch-1.0 audiotestsrc ! audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=2 ! audioconvert ! audio/x-raw,format=S24LE,layout=interleaved,rate=48000,channels=2 ! alsasink device=dmix:1,0 sync=0 async=0

Also the Mic is ok with this pipeline, the recorded sound is good:

gst-launch-1.0 alsasrc device=hw:1,0 ! capsfilter caps=audio/x-raw,format=S24LE,layout=interleaved,rate=48000,channels=2 ! filesink file.raw sync=0 async=0

The issue appears when I use the Mic and the Speaker simultaneously (when I start both the pipelines together): the speaker reproduces noise and the recording pipeline creates a 0 byte file. If I stop the Mic pipeline, the speaker audio returns good. You can check the reproduced noise on this files:

https://drive.google.com/file/d/1A5ilgyI-qq5qQtPihcbxW6uUMZkbPJ1p/view?usp=sharing

I have the same results also using a powered USB hub. Note that the device works correctly on a Linux PC and on a MAC (with the same pipelines, same USB cables, …).

Can you help me?

Thank you, best regards.

Ivan

Hi,
Does it happen in running arecord and aplay commands simultaneously? Or specific to gstreamer commands?
Not sure but it seems to be a hardware issue in the TK1 board itself.

You may also consider to use Jetson Nano.

Hi DaneLLL,

yes, the same issue appears also using arecord and aplay.

I have another usb audio device with similar characteristics:

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
        |__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 3: Dev 6, If 0, Class=Hub, Driver=hub/5p, 480M
                |__ Port 2: Dev 7, If 0, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 2: Dev 7, If 1, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 2: Dev 7, If 2, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 2: Dev 7, If 3, Class=Human Interface Device, Driver=usbhid, 12M
                |__ Port 5: Dev 8, If 0, Class=Vendor Specific Class, Driver=, 480M

Playback:
  Status: Stop
  Interface 1
    Altset 1
    Format: S24_3LE
    Channels: 2
    Endpoint: 1 OUT (ASYNC)
    Rates: 44100, 48000, 48000, 48000

Capture:
  Status: Stop
  Interface 2
    Altset 1
    Format: S24_3LE
    Channels: 2
    Endpoint: 2 IN (ASYNC)
    Rates: 44100, 48000, 48000, 48000

and the issue appears also with this one.

Instead, this usb audio device works correctly (using the same sample rate, I have changed only the format to S16_LE):

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
    |__ Port 1: Dev 8, If 0, Class=Hub, Driver=hub/7p, 480M
        |__ Port 6: Dev 9, If 0, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 9, If 1, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 9, If 2, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 9, If 3, Class=Human Interface Device, Driver=usbhid, 12M

Playback:
  Status: Stop
  Interface 2
    Altset 1
    Format: S16_LE
    Channels: 2
    Endpoint: 1 OUT (ADAPTIVE)
    Rates: 48000

Capture:
  Status: Stop
  Interface 1
    Altset 1
    Format: S16_LE
    Channels: 2
    Endpoint: 1 IN (ASYNC)
    Rates: 8000, 16000, 24000, 32000, 44100, 48000

Maybe the issue is something related to the two ASYNC Endpoints and/or the S24_LE format?

I can’t consider to use a different platform because we have an already in production system based on TK1.

Thank you, best regards.

Ivan

So far as I know the TK1 interrupt aggregator works only with CPU0 (other newer multi-core systems can spread the interrupt across cores). If for some reason the driver is receiving interrupts such that both cannot exist at the same time, then I’d think that the device which runs is not giving up the interrupt. A form of interrupt starvation.

For the TK1, if you “head /proc/interrupts”, then you’ll see only “CPU0” if the current model does not enable the other cores. If you do enable the other cores, then you will see software interrupts spread across other cores, but you won’t see hardware interrupts move.

I’m not positive of what you would look at to see if this is happening, and you could just save a copy of the file “/proc/interrupts” over time and then compare; but if you have “egrep” installed (“sudo apt-get install egrep”), then this might be of interest:

sudo -s
watch -n 1 egrep '\(usb\|USB\|CPU[0-3]\|Err\|IPI\[0-9\]\)' 'interrupts'

The “Err” line at the bottom is important. See if you ever get an Err, you should never see this as anything but “0”. However, here is a key: When you use only record or only playback, and have never run both at the same time, if you then see something related to failing or rescheduled interrupts suddenly climb up very fast, then you might have found an issue of driver resources not being available.

Keep in mind that if you know an IRQ for a process which is involved, then that IRQ should appear in the left column. Each time you stop/start a process that PID is likely to change. You could use “ps” to find a PID, but I generally just use “htop” (“sudo apt-get install htop”) to find a PID unless I’m willing to write a script to find the PID or to launch and report PID. You can do things like launch htop to view only a given user’s processes ("-u username" or “–user=username”)…then you get a live update of PID every time that user launches something if that user doesn’t already have a lot of PIDs running.

If you wanted to change the egrep above to add some PIDs, then basically you are searching for a “:” pattern (the left column). You have to escape some of the egrep search pattern when you run in “watch”, but lets say you wanted to watch PIDs 42 and 142 in addition to what is already there (sudo not shown, assumed), you’d separate with an escaped pipe, “|”, and add the number plus colon:

watch -n 1 egrep '\(usb\|USB\|CPU[0-3]\|Err\|IPI\[0-9\]\|42:\|142:\)' 'interrupts'

I’ll attach my TK1 scripts for performance settings, and you’ll want to make sure all four CPU cores are running. If you were already running in a mode which enabled only one core, then you will probably get a dramatically better performance just by enabling all the cores. If the cores are enabled at max performance, then you’ll know any kind of IRQ starvation is not due to slower settings.

FYI, a good hardware driver design does only the minimum work possible in the direct hardware access steps. Splitting off the driver for non-mandatory hardware access steps allows migrating the rest of the work to a software-IRQ thread, the kworker content. Much of the time spent in direct hardware access of a driver is time which is atomic and you cannot context switch. Soft IRQs (such as kworker) allow context switching. Even on a multi-core system a bad driver design will spend more time in uninterruptible code. If no other process needs the core, this won’t matter and might even be a bit more efficient, but as soon as different hardware access threads start needing access at the same time this becomes a problem.

Sorry, this whole long thread is just a way of saying to check if the issue is merely lack of CPU time for the drivers.

NOTE: I had to use a “.txt” name extension on the files to get the forum to allow. Waiting for them to be scanned.
performance_ls.txt (447 Bytes)
performance_set_max.txt (736 Bytes)
performance_set_default.txt (736 Bytes)

Hi linuxdev,

thank you for the explanation.

I have already set the “max perfomance” state (all the cpu enabled, at max frequency) and also when I use both the speaker and mic together, the “/proc/interrupts” Err field value remains 0:

Every 1.0s: egrep \(usb\|USB\|CPU[0-3]\|Err\|IPI\[0-9\]\) /proc/interrupts

           CPU0       CPU1       CPU2       CPU3
 53:          0          0          0          0       GIC  pmc_usb_phy_wake_isr
 71:     292083          0          0          0       GIC  tegra-xhci:usb1
129:          0          0          0          0       GIC  pmc_usb_phy_wake_isr
IPI0:          0          0          0          0  CPU wakeup interrupts
IPI1:          0          0          0          0  Timer broadcast interrupts
IPI2:      29237      29149      23910      44552  Rescheduling interrupts
IPI3:        353        485        570        576  Function call interrupts
IPI4:          1          8          3          3  Single function call interrupts
IPI5:          0          0          0          0  CPU stop interrupts
IPI6:          0          0          0          0  CPU backtrace
Err:          0

The tegra-xhci interrupts don’t seem to rise too much.

I have found out an interesting thing: if I connect both the “bad” devices through an external USB hub and start recording through device 1) and playing sound through device 2), the reproduced and recorded sounds are good. If, in the same situation, I play and record through the same device, I have the issue.

So I think it is a more USB driver related problem, right?

Thank you, best regards.

Ivan

It appears the IRQs show up in a fairly “typical” way. In particular, if using both record and play at the same time during an error does not show an extreme IRQ interrupt or rescheduling on CPU0, this probably is unrelated to IRQ starvation.

Having behavior change depending on using a HUB or not using a HUB is a very strong indication that the USB signal is an issue. Perhaps only an issue when both devices run at the same time due to bandwidth availability, but perhaps for other reasons.

It would be good to understand something about HUBs are designed, and why not all HUBs (which seem otherwise equal) are not the same.

In USB a higher speed controller tends to have the ability to back off to lower speeds. Your “lsusb -t” from above says the root HUB (built in to the TK1) is USB2 (the “480M” is the 480Mb/s of USB2). The actual audio devices, and the keyboard (or mouse, all I know is it is an input device), are all USB1.1 (the “12Mb/s” is USB1.1 speed). All of these devices are working at the same speed.

In describing a “Transaction Translator” (“TT” for short) this will at first sound like you do not need this on the HUB. When I describe the base use of the TT, you would be correct to believe you don’t need a HUB with TT, but then I’ll add some other comments as to why that may not be the case.

Before I continue, I want to ask a specific question: Is this a cheap HUB? Is this an expensive HUB? I am going to guess you will say it was inexpensive.

When a HUB can handle more than one device speed, if you have no TT, then all devices must revert to the lower speed. So if a USB2 and a USB1.1 device were on the same HUB where there is no TT, then even the USB2 device would be forced to revert to slower USB1.1 as soon as the USB1.1 device is plugged in. When a TT is present, then the two devices, one being USB2 and the other being USB1.1, are essentially decoupled. The USB2 device would then actually operate at USB2 speeds, and only the USB1.1 device would run at the slower USB1.1 speed.

You only have USB1.1 devices, so this does not seem to be a problem. However, adding a TT usually also adds buffering. In a way a HUB is a multiplexer, and the final connection path from HUB to root HUB only communicates regarding one device at a time. Even when all devices run at USB1.1 speeds it is possible there is performance lost if there is no TT.

Your extra HUB experiment leads to two possible issues.

The first is that if signal quality is bad, that a different HUB may have good signal quality. The signal path for USB depends on the impedance of both ends of the connection, and the two interact. Perhaps some HUB combination (or even different cables) between host (the TK1) and the devices (microphones and playback device) have different qualities.

The second possibility is that a lack of TT combined with simultaneous use of several devices is acting as a data choke point. If this is the case, then any individual device will function correctly by itself, but additional devices will begin to fail because that bandwidth is not buffered and unable to multiplex in real time (microphones and audio playback are usually isochronous, and thus can’t handle waiting while traffic becomes available).

Note that everything I see is on a single external HUB, and that this HUB connects to a single USB2 root HUB (remember the root HUB is the TK1 itself). The “lsusb -t” also shows a USB3 root HUB which isn’t used. As a necessary side note, do beware that internally some hardware, when USB3 capable, and then detecting USB2 or slower devices, may actually reroute the signal to a separate USB2 controller instead of handling that traffic itself (a USB3 root HUB can be dedicated to that, while offloading USB2 to a legacy controller). In that case “lsusb -t” might show the 5000M USB3 speed root HUB is not used even though you connected your external HUB into the USB3 port. For the moment, unless you say otherwise, I’m going to assume the USB3 port is free.

If your microphone is stereo, then you will see the two separate microphones in “lsusb -t” even though it is physically one device. On the other hand, you could have two separate microphones. I don’t know which case it is. You are referring to this as “the mic”, so I am going to guess it is a single stereo microphone. You may have to adjust for what I’m about to tell you if I am wrong.

If you were to move the microphone and playback devices to separate root HUBs, and the problem were to go away, then I believe your HUB is at fault. Keep in mind that you can avoid using extra external HUBs on at least one of the USB2 or USB3 native sockets of the TK1 carrier board. Your goal is to test while eliminating the external HUB, and while separating traffic such that output sound and input sound do not use a common HUB.

If this helps, then you can buy a higher quality HUB and things should work (lower priced HUBs are not all equal…they are all “compatible”, but how performance is changed can be drastically different among HUBs if there is simultaneous use of devices). If not, then perhaps more questions will provide more clues.

Hi linuxdev,

I don’t have a different behaviour depending on using an external USB hub or not using an external USB hub. I have the same issue also with the external USB hub when I use the microphone and speaker from the same device.

Anyway, the USB hub I’m using is this one:

https://www.amazon.co.uk/AFENDO®-Portable-Aluminum-Powered-adapter-7-port-black/dp/B00SWBEG1I

Maybe I have not explained well my situation. I have a single device (let’s call it device A) that provide microphone (stereo) and speaker interfaces. The issue appears when I use them simultaneously. I have also another different single device (let’s call it device B) that provide microphone and speaker interfaces that have the same issue. Then I connect those two devices through the external USB hub, so in this situation I have 2 microphones and 2 speakers: if I use simultaneously the microphone from device A) and the speaker from device B) I don’t have the issue, but if I use simultaneously the microphone from device A) and the speaker from device A) I have the issue, also if the device is connected through the external USB hub (and so if I use simultaneously the microphone from device B) and the speaker from device B)).

I will paste here the “lsusb -t” output in various situations. For example, on one of the TK1 boards I have (Toradex Ixora Carrier Board - Apalis TK1):

Nothing connected to the USB ports:

root@apalis-tk1:~# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M

Only device A) connected:

root@apalis-tk1:~# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
    |__ Port 3: Dev 32, If 0, Class=Hub, Driver=hub/7p, 480M
        |__ Port 6: Dev 33, If 0, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 33, If 1, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 33, If 2, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 33, If 3, Class=Human Interface Device, Driver=usbhid, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M

Only device B) connected:

root@apalis-tk1:/sys/bus/usb/devices/usb4# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
    |__ Port 3: Dev 29, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 2: Dev 30, If 0, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 2: Dev 30, If 1, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 2: Dev 30, If 2, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 2: Dev 30, If 3, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 5: Dev 31, If 0, Class=Vendor Specific Class, Driver=, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M

Only external USB hub connected:

root@apalis-tk1:~# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
    |__ Port 1: Dev 10, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 4: Dev 11, If 0, Class=Hub, Driver=hub/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
    |__ Port 3: Dev 14, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 4: Dev 15, If 0, Class=Hub, Driver=hub/4p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M

Device A) and Device B connected to the external USB hub:

/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
    |__ Port 1: Dev 10, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 4: Dev 11, If 0, Class=Hub, Driver=hub/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
    |__ Port 3: Dev 34, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 4: Dev 35, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 3: Dev 36, If 0, Class=Hub, Driver=hub/7p, 480M
                |__ Port 6: Dev 37, If 0, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 6: Dev 37, If 1, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 6: Dev 37, If 2, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 6: Dev 37, If 3, Class=Human Interface Device, Driver=usbhid, 12M
            |__ Port 4: Dev 38, If 0, Class=Hub, Driver=hub/5p, 480M
                |__ Port 2: Dev 39, If 0, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 2: Dev 39, If 1, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 2: Dev 39, If 2, Class=Audio, Driver=snd-usb-audio, 12M
                |__ Port 2: Dev 39, If 3, Class=Human Interface Device, Driver=usbhid, 12M
                |__ Port 5: Dev 40, If 0, Class=Vendor Specific Class, Driver=, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M

You said to try with a higher quality external USB hub. This is for testing only since it can help to understand where the issue is?

Thank you, best regards.

Ivan

Still examining the setup, I want to first ask about the device which requires a custom driver:

...
Port 5: Dev 40, If 0, Class=<i><b>Vendor Specific Class, Driver=</b></i>, 480M
...

The “HID” part on both devices is likely just some buttons which can be used to control the devices. The “Audio” devices also use generic USB audio drivers. The device with the vendor specific class has a function which no driver was found for. This part of the device may not matter, but I’d like to know what exactly it is that this custom driver does. Do you have any information about the device which has the non-generic missing driver?

About the “higher quality external USB hub”: The hub you are using does not have details specifications. We might be able to gather a detailed verbose lsusb and get hints about transaction translators (“TT”). If the hub you are using has multi-TT (versus single-TT), then there would be no need for another HUB. However, if the hub is single-TT, then there is no way you are going to get rated performance on all devices at the same time. Switching to a hub with a known multi-TT design could eliminate if this is the issue.

If you do just a regular “lsusb” with no options while the hub is connected, then you will get an “ID” for each device. I don’t know what the ID is for that hub, but if for example it is “0955:7140”, then you could get a verbose lsusb of the hub like this:

sudo lsusb -d 0955:7140 -vvv

That report might mention the transaction translators. If this is multi-TT, then there is no reason to try another hub. If this is not multi-TT, then I’m afraid you’ll probably need to test with a new hub. A multi-TT hub might not help, but it probably would, and would never hurt (other than being more expensive). Very few consumers know anything about hubs which are multi-TT versus single-TT, and it is difficult to find such information on the hub without going to detailed documents (and cheaper hubs never provide this…some higher quality hubs do provide this since they want people to know they are multi-TT).

Regardless of anything else, the hub does matter because it changes how multiple devices share on a single hub (all the way down to a shared root hub).

Hi linuxdev,

no, unfortunately I don’t have any info on the interface which has the non-generic missing driver, but I don’t think the problem is there, this interface appears only on device B) and not on device A).

My external USB hub has single-TT:

root@apalis-tk1:~# lsusb -d 2109:2812 -vvv
Bus 003 Device 007: ID 2109:2812 VIA Labs, Inc. VL812 Hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x2109 VIA Labs, Inc.
  idProduct          0x2812 VL812 Hub
  bcdDevice           90.90
  iManufacturer           1 VIA Labs, Inc.         
  iProduct                2 USB2.0 Hub             
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           25
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             4
  wHubCharacteristic 0x00e9
    Per-port power switching
    Per-port overcurrent protection
    TT think time 32 FS bits
    Port indicators
  bPwrOn2PwrGood       50 * 2 milli seconds
  bHubContrCurrent    100 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0100 power
   Port 2: 0000.0100 power
   Port 3: 0000.0100 power
   Port 4: 0000.0100 power
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength           42
  bNumDeviceCaps          3
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000002
      HIRD Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat           4 micro seconds
    bU2DevExitLat         231 micro seconds
  Container ID Device Capability:
    bLength                20
    bDescriptorType        16
    bDevCapabilityType      4
    bReserved               0
    ContainerID             {2d207069-441e-de4c-8f73-78ec5964eca7}
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0001
  Self Powered

but note that also if the issue will be resolved using a different (multi-TT) external USB hub, I can’t propose this solution to our customers, the devices A) or B) have to work without an external USB hub.

Anyway, I see that also the TK1 board “internal” USB hub has single-TT:

root@apalis-tk1:~# lsusb -d 1d6b:0002 -vvv
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            3.10
  iManufacturer           3 Linux 3.10.40-2.8.6+g2c7a3c3af726 tegra-xhci
  iProduct                2 Nvidia xHCI Host Controller
  iSerial                 1 tegra-xhci
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           25
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             6
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
    TT think time 8 FS bits
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0100 power
   Port 2: 0000.0100 power
   Port 3: 0000.0100 power
   Port 4: 0000.0100 power
   Port 5: 0000.0100 power
   Port 6: 0000.0100 power
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0003
  Self Powered
  Remote Wakeup Enabled

Is maybe this the problem?

Thank you, best regards.

Ivan

The internal hub has only one PHY on the port…it is a root hub. Thus there are no other ports to add a TT to on the Jetson SoC. There could be more ports on the root hub, but the Jetson does not do this on dev boards…and the third party boards tend to follow this same design. A multi-TT attaches an individual TT to every port in a group of ports; a single-TT design attaches just one TT to a shared group of ports (after which all ports must operate at the same speed and may suffer more latency as more devices share). Each port on a Jetson uses a separate controller unless the carrier board strayed from the reference design. Thus, only if multiple ports directly interface with the same controller without some other hardware intervening in the topology will a TT change anything. HUBs are a special case since they always have multiple ports directly sharing an interface.

Now for the case of a single physical device using a single cable for multiple “devices”, there is more than one way the manufacturer can share that cable. The manufacturer could have each internal device aware of the other, e.g., because an MCU is running both devices, and the MCU handles knowing which function goes to which device; or the two devices could be independent with no knowledge of each other. In this latter case there would need to be an internal hub, and it would be best if the internal hub has multi-TT. This latter does not seem to be the case because the tree view does not show a hub. Presumably all internal devices have some awareness of each other and it is up to an MCU to essentially multiplex.

If the solution is to use a multi-TT hub, then that solution would only matter if simultaneous use of two devices using separate cables is an issue. Two devices on one cable won’t benefit from this. For the other failure scenario of two devices using separate cables (on a single hub) there would be no other choice but to go to a multi-TT hub instead of a single-TT hub. Can we verify that a single physical device suffers the issue using a single cable with no other device connected to that root hub (the port directly on the Jetson…the lsusb tree view would not show any hub unless it is integrated into the device itself)? If a hub is used, then for testing purposes, can we verify the issue exists when no other device is on the hub, e.g., no keyboard or mouse? The best test case is directly connected to the port (unless external power is required).

If this is the case where a single cable directly connected to the Jetson fails, then no hub will help and it isn’t an issue of single-TT versus multi-TT. If it is the case where two separate cables show up in the same tree view (sharing the same root hub), and only then does a failure occur, I will be tempted to say you must get a better hub.

Hi linuxdev,

in the previous “lsusb -t” output, where I said “Only device A) connected”:

root@apalis-tk1:~# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
    |__ Port 3: Dev 32, If 0, Class=Hub, Driver=hub/7p, 480M
        |__ Port 6: Dev 33, If 0, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 33, If 1, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 33, If 2, Class=Audio, Driver=snd-usb-audio, 12M
        |__ Port 6: Dev 33, If 3, Class=Human Interface Device, Driver=usbhid, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M

I meant that this device is connected to the port directly on the TK1 board, with no other devices/hubs connected.

You can see that the device provides an internal hub to share the other devices. I checked with lsusb, this hub is a multi-TT. When I use the microphone and the speaker simultaneously I have the issue.

Thank you, best regards.

Ivan

Ok, so this guarantees it is not a hub issue. You are correct then that there is probably something either in how the device itself works in combination with characteristics of the Jetson, or else just from the Jetson end.

I won’t be able to answer in any more detail, but in case someone else can, do you have a USB1.1 protocol analyzer available? Someone else will probably still be able to help debug, but if you have the analyzer I’m sure it would speed things up to see if USB errors are occurring (if it isn’t a protocol error on the wire, then you can narrow the search to the drivers or user space software).

One thing I will suggest, even if it is probably unrelated, is to use the device with the custom type on a PC and see if there is a known driver loading. Imagine if the driver does something like upload firmware…in that case it could still be at fault.

Hi linuxdev,

no, unfortunately I don’t have a USB1.1 protocol analayzer.

I have connected the device with the custom type on a PC but also there no driver is loading.

I have tried to flash the Apalis TK1 board with the mainline kernel (Toradex provides also a mainline image) but the issue is still there. Is there maybe something in the xusb firmware?

Thank you, best regards.

Ivan

There could be firmware required by the Toradex module or carrier board, but since USB is functioning correctly, then this would not be related to the current issue. If some USB works, e.g., one at a time, where firmware is more or less an “all or nothing” thing (though performance throughput could still be an issue), it implies the USB side firmware is valid.

Note that since this occurs with multiple carrier boards and TK1 modules I also doubt it is anything related to the particular manufacturer’s board support package.

An example of a plausible issue is that the bandwidth required by two devices exceeds the USB ability to transfer data, but would work with a single device. However, USB1.1 is only 12Mbit/s, and the USB2 port this connects to handles 480Mbit/s. You could have a very very large number of USB1.1 devices without ever noticing bandwidth requirements.

The point of joining traffic among multiple devices could be at fault, but you are not using an external HUB, so the only such point is the HUB within the device itself. That device reports as multi-TT, so this cannot be an issue. Add to this that the internal HUB within the device was created by the manufacturer of the device and that this works on other systems, and so the internal HUB can be ruled out as a cause.

If the Jetson were in a low power mode, e.g., the system ran and then went into a USB auto-suspend mode, then a mode such as this could cause issues, but in this case I doubt even a single device would work, and we know the devices work one at a time. Plus you’ve set for max performance mode in testing.

Something much more subtle and difficult to understand is going on. As an example, if USB itself is not failing, something in the end programs consuming the data could have a race condition or priority inversion. gstreamer itself is very well tested, but it doesn’t mean this can’t happen for your case. I am unable to help with gstreamer, but unless you have a protocol analyzer for the USB1.1 (which is likely to show all is working well in the USB side), then end user software is by far where the issue is.

One could still argue that other drivers could be getting in the way, but it won’t be the USB drivers. Also, the actual IRQ information shows drivers are “more or less” behaving normally.

If you were to run a program such as “htop” (“sudo apt-get install htop”, a “better” top), and compare what shows up in a working single device operation, and then compare to how CPU use appears in a failed dual device test, then you might be able to notice a trend (such as a CPU core which did not peg at max now pegging at max, or a core which had relatively normal or random use suddenly go to near zero).

I’m not equipped to help with gstreamer, and most people do not have this specific device to test with, and so it may be difficult to narrow things down further.

Hi linuxdev,

note that the issue appears also with aplay/arecord, and not only with gstreamer.

I have compared the CPU usage in the working single device operation with the usage in the failed dual device test and it is more or less the same, I can’t see any difference.

Thank you, best regards.

Ivan

Possibly aplay/arecord and gstreamer share some linked library, so despite the end programs being different, it does not necessarily mean the consuming/producing user space software does not have a common component. However, your situation is a bit perplexing.

I probably sound like a broken record, but if other people debugging do not have the actual devices to test with, then you almost require a USB protocol analyzer to see if USB is doing something wrong. Or perhaps building with both debug symbols and profiling support would offer a way to find out if some function is suddenly changing or hanging up when both record and playback run simultaneously. When you look at CPU usage it is equivalent to using a large stone axe for surgery, versus profiling and seeing times of individual functions. There are a large number of gprof tutorials on the internet, here is one example:
https://www.thegeekstuff.com/2012/08/gprof-tutorial/

Keep in mind also that you can run “ldd /where/ever/an/executable/or/library/is” and find out what outside libraries are linked in, and then see which parts are in common between the aplay/arecord and gstreamer programs. This is not an easy way to go, but then if you knew this you might be able to install debug symbol packages for those libraries to help gprof out somewhat. You won’t find most libraries with a profiler enabled version, but you may not need that. If you do trace to a library which has a problem, then you could report that particular library and it would narrow things down.

The biggest problem here is that someone else will need to reproduce this to do anything more direct than profiling or using a USB analyzer. The need for someone else to evaluate this with the same USB audio devices is high on the list of being able to actually resolve this. As such I’ll recommend you give URLs to show the exact USB audio devices and/or the specs on the devices.

Hi linuxdev,

this are the libraries that is used both by gstreamer and arecord/aplay:

librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xb6f33000)
libasound.so.2 => /usr/lib/arm-linux-gnueabihf/libasound.so.2 (0xb6e98000)
libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6e7d000)
libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6d95000)
/lib/ld-linux-armhf.so.3 (0xb6f65000)
libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xb6d29000)
libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xb6d1e000)

I have also compiled the latest libasound version and nothing changed.

This are the devices that have the issue:

DEVIO SCR-25:
https://www.biamp.com/products/devio/devio-huddle-room-solutions

Clearone Converge Huddle:
http://www.clearone.com/converge®-huddle-0

Thank you, best regards.

Ivan

I noticed the Converge Huddle does have a firmware download. I couldn’t say if this has anything to do with it, but you might start (for that one) seeing if the latest firmware is loaded, and if the firmware needs to be present in the “/lib/firmware/” tree.

For the DEVIO SCR-25, it says it supports both Mac and Windows using standard audio drivers. This basically says it is also compatible with Linux. However, it does also have firmware updates. So the first thing I’d do for this unit as well is to see if the most recent firmware is installed.

Some firmware loads into the device and the device remembers the update. Probably your devices are in this category. However, some devices will need to load firmware each time they power on. Since the two functions running simultaneously is when things fail, it is possible this firmware is involved in the issue.

After that, then probably someone with one of those specific devices needs to debug this.

Hi linuxdev,

yes, I have installed the latest firmware on both the Devio SCR-25 and Converge Huddle devices (the firmware loads in the device, no need to load it at power on and no firmware needs to be present in the /lib/firmware directory) and the issue is still present.

Thank you, best regards.

Ivan

At this point I think someone will need to reproduce this on one of those devices. There may be some sort of USB debug setting to provide clues I don’t know about, but it is hard to go further without some hands on testing.