Nothing showing up with lspci -v

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Need to install a 2 port pcie serial card in my TX1. Instructions say to verify the card is active run lspci -v. Nothing shows up. Not even the controller. This is the output from dmesg | grep pci. The 2 port is a startech card 16550 uarts with a linux driver for 3.0 and above kernel. Works fine in the intel box. I’ve used them before in linux boxes. Other than having to compile the .ko module with a cross compiler the install is straightforward. But it looks like something is wrong with the pcie interface. I saw the other post about making sure the kernel headers and all were the same so I reflashed with a fresh install still the same deal.

[ 2.739387] tegra-pcie 1003000.pcie-controller: PCIE: Enable power rails
[ 2.740436] tegra-pcie 1003000.pcie-controller: probing port 0, using 4 lanes and lane map as 0x14
[ 2.742504] tegra-pcie 1003000.pcie-controller: probing port 1, using 1 lanes and lane map as 0x14
[ 3.144202] tegra-pcie 1003000.pcie-controller: link 0 down, retrying
[ 3.550259] tegra-pcie 1003000.pcie-controller: link 0 down, retrying
[ 3.956137] tegra-pcie 1003000.pcie-controller: link 0 down, retrying
[ 3.962620] tegra-pcie 1003000.pcie-controller: link 0 down, ignoring
[ 4.369666] tegra-pcie 1003000.pcie-controller: link 1 down, retrying
[ 4.779976] tegra-pcie 1003000.pcie-controller: link 1 down, retrying
[ 5.189883] tegra-pcie 1003000.pcie-controller: link 1 down, retrying
[ 5.196382] tegra-pcie 1003000.pcie-controller: link 1 down, ignoring
[ 5.200838] tegra-pcie 1003000.pcie-controller: PCIE: no ports detected
[ 5.207754] tegra-pcie 1003000.pcie-controller: PCIE: Disable power rails
[ 6.585769] ehci-pci: EHCI PCI platform driver

Can’t use the serial port on pin 8 and 10 of the expansion header. Even though I managed to get the console off of it if its connected the machine won’t boot. This is a known problem I found out its in the docs. So any suggestions? Pretty generic card no reason why it shouldn’t work. Are there any other tests I can run to see if the pcie is dead or not functioning correctly? Seems to me I should see the pcie controller and bridge even with nothing plugged into the slot.

Dan

2

J21 pins 8/10 require a 3.3V logic level for the serial UART. Other serial UARTs will fail. The ground pin should also be connected, else noise would be a problem. If CTS/RTS is connected, you should use CTS/RTS flow control, otherwise software flow control; as noise increases in the environment, CTS/RTS will begin to be required. If this does not work I’d be suspicious of hardware failure.

I am doing a clone of my JTX1 at the moment, so I can’t check PCIe. However, I believe the lack of output for the PCIe link from lspci may be normal (if so, it would likely be related to how the PCIe is accessing memory differing from a desktop motherboard).

3

Yea I read that in the docs but even with a 3.3 level shifter in place and the ground grounded its still 1 in 9 or so boots. The only way I can get it to work every time is to disconnect the controller while it boots. In the docs it said to ground one of pins 8 I think. So I was trying to get one of my multiport cards to work. I moved the console over to the uart that comes out on the debug port. So nothing comes out on the video at all till the desktop is up. I’m using sparkfun 3.3 to full rs232 shifters. The roboteq controllers have a full power rs232 and less likely motor noise etc will mess with it. I’d use usb but they recommend not too. Something about it can get into a state that you have to unplug the controller to get it back. And I noticed ROS nodes written for usb connections always have a procedure to reset the usb port. Rather not deal with that mess. When I do a lspci on one of my TK’s I get a whole slew of stuff. But seems to me you should see at least the controller even if nothing is attached. Samtech makes a cable for that debug port so are all the ports sensitive like the one on the expansion connector I think its uart1. If so I can get that cable and put that port back to the console. Nothing is coming out of the port that I can see with the analyzer. Initially I thought some garbage was coming out and the controller was responding in a way that stopped the boot. Like it was waiting for further input. But I put the analyzer on it and don’t see anything like that. It looks quiet to me no spikes or anything but my analyzer isn’t pro quality. Man I’m so close to having this bot running. I just need one more serial port. If I don’t have to use the can bus it makes things so much easier but to get away from that I gotta have 2 ports. The info in the docs doesn’t say much about what is going on just says noise can cause it not to boot. I first saw it trying to get the serial ports reconfigured. Left my jumper on for testing and all of a sudden it wouldn’t boot. Looping the tx and rx it should be grounded by the board right? On the TK1 I see the controller the bridge and ethernet controller and the wireless card in the mini pcie slot. What makes me wonder if its working at all is the retrys then ignores in the dmesg output. It very well might not put out any out with nothing on the bus. It was the first step in the install directions. I went ahead and tried both cards and still don’t see anything. I have an old graphics card around here somewhere I was going to try next. I’d be interested to see dmesg | grep pci from your machine. That is what is causing me to think something isn’t right. Looking through /var/log/boot.log I see a fail on restore sound card state. Is the sound card on the pci bus?

Dan

4

A USB serial device can be either self-powered or bus powered. The trick is that regardless of power method, the USB and UART circuits must be through their startup sequence prior to serial USB working as a serial console (J21). Since power can be taken from either the JTX1 or the host, there can be an inconsistency between serial USB devices which are powered differently. Sometimes this means you have to disconnect and reconnect something if default is not working. The docs on the serial port at that URL on the serial port is from a known working serial USB UART which behaves well.

Level shifters should work for 3.3V to typical RS232 voltages, but there can be issues you don’t expect. I would not trust level shifters as a deciding factor if the port works or fails (and this is J21, not the debug port). Even among “real” RS-232 serial ports there can be an issue with some ranges of voltage swing…the standard allows +3 to +15V, and -3 to -15V…how the shifters behave under such a range may not be as expected.

No data line of a serial port should ever be grounded.

As for the PCIe bus, I can verify that a normal and working JTX1 does not show a PCIe bridge. I suspect PCIe is directly connected to the memory controller (I have not verified this), which might account for lspci not showing a PCIe bridge or controller on a JTX1. The device actually connected to the slot should show up. If boot fails when a PCIe card is installed, I have to wonder if pins are correctly matched.

Sound state is not related to PCIe on a Jetson. This is actually part of the Linux distribution, and unless you’ve created a saved state, there will be no file to restore from.

Seeing the lspci output from a video card would be interesting, but I would not expect it to function by default…the video driver on a Jetson is designed for non-PCI devices custom to Tegra chips. Certainly PCIe video should be showing up for query via lspci, just don’t expect it to function.

5

I managed to get the .ko file compiled and loaded. No devices are created though. Can’t talk to the board with the utilities. Did a little research and looks like udev creates devices when the kernel detects a device so if the kernel isn’t detecting the card in the pcie bus no devices will be created. They are /dev/ttyFx. Checked the kernel config to make sure all the pcie stuff is enabled. Its all good. Talked to someone else that has a TX1 and they didn’t get any output with a lspci -v either. So its puzzling. Checked the board in my intel linux box. Worked just fine both of them in fact. cat /proc/modules shows it on the TX1.

99xx 44292 0 - Live 0xffffffbffc00c000 (O)

This is the output from file on the .ko 99xx.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=b994d9d34814d4bfbe05b747524872c6aa39aec4, not stripped That looks right to me.

lspci shows nothing still. Which is really cat /proc/bus/pci/devices which also shows nothing just to make sure its not lspci. The way I remember pci enumeration works is at boot each device is probed and its signature is used to set up devices etc. So it should see the device even if no driver is present. This is all based on experience with intel boxes though. So this might be very different. There are some utilities to set up baud rate etc on the 4 port they all fail which is understandable since there are no devices. Looked up the device in devices.txt and created a node with mkdev still no workie. Seems like if it wasn’t working at all it would throw some errors. However I did notice when I perused the kernel config none of the pci debug flags are set so thinking I should set those and recompile the kernel. Will the TX1 try and boot off the SD card first? That would make all this easier to deal with. If so can you point me to how to make it happen?

Dan

6

You are correct that enumeration should show the PCIe devices even if there is no driver for what is contained on the PCIe card.

Under udev any file in “/dev” would be generated by a kernel feature. Manually creating such a file with no kernel feature will not be any more successful than no file at all…communications to and from a “/dev” file is really communications with that kernel feature.

So far as PCIe errors go, some cards have more or less in the way of advanced error reporting and correction…advanced error reporting and correction is not mandatory, and could be eliminated in order to make the card less expensive (in which case some errors won’t be shown in lspci). Mostly when thinking about errors on PCIe it is the data which runs at fairly fast speeds which is thought of as being sensitive to errors…a PCIe v3 device might throttle back the data to v2 or v1 speeds. The control channel is designed to be slower and very unlikely to fail even when data lanes have gone missing; however, if it is a control channel which has failed, lspci won’t know there is a card there…it isn’t the physical use of the PCIe slot which lspci reports, it is information from non-data-lane query which serves this function.

I do not know if the TX1 will try to boot SD card first, but extlinux.conf can be used to point at any device you want for rootfs even though configuration is on eMMC…it’s easiest to just try and find out. Just make sure you have the “/boot” content on both SD card and eMMC, and that the content is identical, using “root=/dev/mmcblk0p1” on both for eMMC, or “/dev/mmcblk1p1” on both for SD (in which case SD needs a full rootfs and not just “/boot”).

FYI, I wouldn’t bother with an entire SD card install for testing if all you are changing is the kernel…just use a serial console for which config to use at boot time and add different configs in extlinux.conf for any kernel or firmware changes. Here’s a sample extlinux.conf with some selectable entries via serial console:

TIMEOUT 30
DEFAULT primary

MENU TITLE p2371-2180 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      FDT /boot/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
      APPEND fbcon=map:0 console=tty0 console=ttyS0,115200n8 androidboot.modem=none androidboot.serialno=P2180A00P00940c003fd androidboot.security=non-secure tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=${lp0_vec} nvdumper_reserved=${nvdumper_reserved} core_edp_mv=1125 core_edp_ma=4000 gpt android.kerneltype=normal androidboot.touch_vendor_id=0 androidboot.touch_panel_id=63 androidboot.touch_feature=0 androidboot.bootreason=pmc:software_reset,pmic:0x0 root=/dev/mmcblk0p1 rw rootwait

LABEL testing1
      MENU LABEL testing on eMMC
      LINUX /boot/Image_test_kernel
      FDT /boot/dtb/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
      APPEND fbcon=map:0 console=tty0 console=ttyS0,115200n8 androidboot.modem=none androidboot.serialno=P2180A00P00940c003fd androidboot.security=non-secure tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=${lp0_vec} nvdumper_reserved=${nvdumper_reserved} core_edp_mv=1125 core_edp_ma=4000 gpt android.kerneltype=normal androidboot.touch_vendor_id=0 androidboot.touch_panel_id=63 androidboot.touch_feature=0 androidboot.bootreason=pmc:software_reset,pmic:0x0 root=/dev/mmcblk0p1 rw rootwait

LABEL testing2
      MENU LABEL testing on SD
      LINUX /boot/Image_test_kernel
      FDT /boot/dtb/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
      APPEND fbcon=map:0 console=tty0 console=ttyS0,115200n8 androidboot.modem=none androidboot.serialno=P2180A00P00940c003fd androidboot.security=non-secure tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=${lp0_vec} nvdumper_reserved=${nvdumper_reserved} core_edp_mv=1125 core_edp_ma=4000 gpt android.kerneltype=normal androidboot.touch_vendor_id=0 androidboot.touch_panel_id=63 androidboot.touch_feature=0 androidboot.bootreason=pmc:software_reset,pmic:0x0 root=/dev/mmcblk1p1 rw rootwait

One thing which could help is if you can run sudo lspci -vvv for your PCIe card on another Linux machine where you know it works. See what the card should be when it is detected and whether any data lane speeds had to be throttled back.

7

All the cards work in my intel machine. Just don’t show up in the TX1. I was using a direct loopback on the serial port to test with. No level shifter involved I haven’t gotten to that point yet. The port I used the level shifter on is on the 6 pin header works just fine. I make sure the port will work in loopback then add the level shifter loop that back to make sure its working then connect to the controller. So how are you supposed to debug the pci interface if you can’t see anything? I tried a sound card as well it doesn’t show up either. Another test I ran was to use the commands on this page
https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci after putting the card into the intel machine so I could see the id of the card. Roboteq says no usb on their controllers for regular operation. Only for config. Does the SATA go through the pcie hub? What other subsystems use it maybe I can look at those and figure out what is going on. The serial cards have different chipsets made by different makers. Surely one of them would show up. Both have linux drivers work with kernel above 3.0. What cards was the pcie bus tested with during development? Any serial cards known to work with the TX1? Is there any docs on how the pcie is implemented on the TX1? This should be pretty much plug and play. The only issue I even thought might be a problem was getting the module compiled but wasn’t a problem at all. I saw the post about throttled back lanes its not doing that. Saw the post about making sure the kernel modules headers and kernel have the same local name. Don’t have the video card now must have chucked it during the move. Do you know what video cards or any cards show up maybe I can find one to make sure the pcie is working. Other than that I’m stuck again. To me the fact it has problems with throttling means its not implemented correctly to the spec.

Dan

8

So far as integrated UARTs go, which ones are you now using? J17 6-pin is associated with the camera mezzanine, J21 has the serial console. If J21 does not work then something is wrong.

If you have two unused RS232 add-on connectors, you could attach a serial terminal to each “/dev” file as it seems you have done, with TX on one crossing to RX on the other, plus grounds (if it is on the same physical hardware ground would of course already be there). This would fail if not using software flow control. You could add CTS-to-DTS and DTS-to-CTS and use CTS/DTS flow control.

In the case of the NULL-modem cable connection of two RS232 connections on a single add-on card, you can assume they use the same signaling levels. If you are going between two different sets of hardware, they are likely (but not guaranteed) compatible signal levels if both use the D-sub connector. Once you start going through headers you need to verify both using the same signal levels…it’s all supposed to be interchangeable, but it isn’t…3.3V or 5V levels may not cross-function to 12V or other levels (and it is possible damage could be done).

The FTDI serial UART chipsets are almost universally supported by default, others maybe not. For a PCIe serial port add-on, unless PCIe first sees the card being present, no driver would be detected or loaded.

I do not know if SATA goes through PCIe, but I suspect it does not.

The deal is that PCIe has both data lanes and a control lane. When data lanes lack signal quality, they throttle back to previous generation speed, and lspci notes this. If the control signal itself is not working, then all bets are off, it isn’t even going to show up. I was interested in lspci -vvv output on the card from another Linux machine to see what the card’s abilities are, including any kind of error handling, and whether on a “working” machine there was any need to throttle back.

Sometimes heavier PCIe cards will torque on the connector slot and cause issues, but it is rather unlikely something as small as a serial port card would do this (unless cables add to the torque). If the card is slightly flawed in how it is seated, this could account for issues; the way the card is mechanically supported by a desktop chassis may mean things work better on a desktop system, whereas sitting loose without support on a JTX1 just sitting unsupported on a table could contribute to failure. Since you have more than one card doing this, you might want to closely inspect the connector socket and very carefully make sure the card is properly seated.

9

Ok I’ve tried 2 serial cards a sound card and a wifi card. None of them work or show up. And your manual says to ground pin 10 of that serial port I didn’t dream it up.

On the Jetson TX1 developer kit, if there is no serial device attached and driving the 2x20 connector J21
pin 10, there can be noise on this input to the Tegra serial terminal. This noise can cause an interruption of
the L4T boot process specifically in U-Boot. If you have an issue booting the system or as a preventative measure, please add a jumper across J21 pins 9 and 10.

And what I’m saying is that behavior happens with stuff connected as well. What I need to know is if some stray characters from the controller come down that line while the TX1 is trying to boot will it stop? Is that what they are referring to as “noise”. I can keep the controllers from turning on if I can find a pin on the TX1 to turn on a relay. Haven’t looked that hard but doesn’t seem to be any general purpose gpio pins. I can use one on my touch screen they are easy enough to access though. That would be the best way but there are others. They have a 2 pin header one pin is ground and when they are connected the controller is turned off but they are really for keeping the bot from generating current from the regen circuits if you happen to roll it while its not turned on. The whole power side of the card is disconnected.

The card is fastened down in an enclosure. It uses a native pci chipset. No phony stuff. I’ve used this brand for a long time. Seems like one of those cards would have showed up. But I read up on pcie and looks like this one is a virtual set up in which case you won’t see any output from lspci until there is an actual enumerated device attached. There is a name for it but basically its a virtual bus which makes sense on the X1 because its used all over the place looks like. Tired of looking at it right now but I’ll get that data in a day or so. Don’t want to put anything in the otg connector I’ve already had one fall off. And I’d rather keep the usb3 free for the Zed. Looked at the Samtech cable its not going to be good for this application. But I’m hoping they have one like that for the connector under the camera. I need to point it in a different direction. Now it points out the side instead of forward. Basically it just brings out the same connector over a short ribbon cable. But they did sample me a couple of the mating connectors for the debug port. Don’t know if I can actually see those pins to solder onto but I’m going to try and use that port too. The 6 pin port works just fine with the level shifters I have so I suspect the others will as well. They are built for RS232 connections have a 9 pin connector on them. I used them on my robotic telescope. They work great.

Dan

10

For reference, this is the J21 serial port document I look at:
[url]Jetson TX1 - eLinux.org

J21 serial console with nothing connected should never cause enough noise to fool boot into thinking there is a key press. This would probably be a case of hardware failure (such as a cold joint on a ground). Connecting a wire with serial console RX sitting in the air unshielded could indeed do this. However, even an RS232 cable of significant length with a ground shield and floating RX/TX twisted pair should not be an issue. That much noise would make me consider hardware failure or ground failure. If wires are connected to the serial console, then to protect against noise the data and other internal wires to the cable should be twisted pair…there would be added protection via an outer shield. Lengths of wire not shielded or not twisted would be what to look out for. If you are operating in an environment with extreme noise, I suppose something more drastic may apply.

During early boot there are two points where u-boot would halt if it thinks there is a keystroke, so serial port activity (regardless of source being legitimate or noise or hardware failure) would stop boot within the u-boot stage. If early on, u-boot would be in its command line environment; a very short time later, noise would stop and wait for which extlinux.conf entry to boot.

In terms of the PCIe issue, I wonder if the pins in the connector are damaged. If not, then it sounds like other parts of the system have failed (I would consider RMA). Take a close look at the PCIe connector pins. Certainly sound cards and other cards should have shown up. Considering how many different PCIe cards it sounds like you’ve tested, hardware failure seems more likely.

11

For what it’s worth, I’m having the same problem here. I’ve installed a PCIe serial port card in the slot, and lspci shows nothing. Did danpollock ever come up with a solution?

-Dennis

12

I believe both Jetson TK1 and TX1 use Spread Spectrum Clocking (SSC), some older PCIe interface chips can’t handle SSC. There was a thread on TK1 forum about how to disable SSC of TK1 to have PCIe cards working. It involved modifying and rebuilding uboot and kernel.

You may want to get help from Nvidia on how to disable SSC.

For some FPGA cards, it’s necessary to restart Ubuntu after FPGA PCIe interface is configured.

13

I bought a couple of keyspan adapters and I’m using those via usb. So far they are working great so I haven’t revisited the pci serial card issue. However since I saw this post about SSC I might see if I can get it to work.

14

Here is the link for disable SCC for TK1:

https://devtalk.nvidia.com/default/topic/819646/embedded-systems/jetson-tk1-disable-spread-spectrum-clocking-on-pcie/post/4490416/post/4489600/

uboot probes PCIe at start up, if disabling SSC in uboot works, it should show in serial console output.