asm-generic/io.h?

Hello, John here.

I’m trying to get some port-mapped 16550A-style UARTs on a PCI Express card working.

First, I asked how to debug ttyS issues:
https://devtalk.nvidia.com/default/topic/1035557/-dev-ttysn-n-gt-3-minicom-debugging-serial-peripheral-connected-to-pci-express-/#5263545

Now I’m trying to reach the UART directly, using the IO registers, but TX2 is ARM so ioremap?
https://devtalk.nvidia.com/default/topic/1036156/jetson-tx2/does-the-tx2-support-i-o-registers-/

I grepped and found under src/ an “asm-generic/io.h” file but “CC” doesn’t find it (it isn’t located in usr/include, nor anywhere outside of my user tree’s copy of TX2 src/

How can I / can I #include <asm-generic/io.h> and compile/link/run on the TX2?

My attempt using /dev/port, lseek, and read/write no longer segmentation faults, but always reads “0x00” from all 64 UART registers, so I wanted to try inb() or ioread8() or or or but none have header files … or not installed, or not on path, or something. <-- me Windows

danke!

The “” syntax uses the include search path. To add a path to this use “-I”. Adding this would imply the ability to translate “#include <asm-generic/io.h>” to something the build understands:

-I /where/ever/kernel/source/is/include

FYI, by default the include path is “/usr/include”. You could copy “asm-generic/io.h” to create “/usr/include/asm-generic/io.h”, but this is a bad idea since “/usr/include” is a system path. Using “-I” instead will allow everything in your kernel source at “/where/ever/kernel/source/is/include/” to be found…which becomes even more important if io.h includes headers from elsewhere in the kernel source.

NOTE: I am assuming a build from Linux, I’ve not compiled from Windows in a very very long time.

I am compiling from the TX2 (which is unusual I suspect, but I’m just writing “hello world” style code trying to get the PCI Express attached UARTs working).\

I assume you mean “use -I /where/ever/include/files/are on the command line to “CC””?

like,

cc -I /home/myusername/kernel/kernel4.4/src/arch/long/path/asm-generic/io.h t0.c

right?

I do apologize for my ignorance; my IDEs normally take care of such things via GUI settings. I haven’t had to compile anything on the command line since using the “2500AD” cross-assembler suite for 8051 development back in the DOS era.

My primary concern with simply changing #include <asm-generic/io.h> to #include “full/path/to/io.h” regards the actual library location: I can’t believe I’ve got the associated bin correctly built and installed on the TX2 if the include file is not, no? Doesn’t the MAKEFILE ensure if one, then the other, and imply the inverse?

Thank you again for your timely assistance.
(note to self: I need to figure out why I don’t receive email notifications of responses (presumably a spam filter issue))

It isn’t at all unusual to build from the Jetson. I’ve been compiling software all day :) …though I found x86_64 code, so it failed :(

No space between the “-I” (captal letter I) and path. You can have more than one “-I”. And it is the parent of a search location, not a specific file. So:

cc <b>-I/home/myusername/kernel/kernel4.4/src/arch/long/path</b> t0.c
...in code:
#include <asm-geneneric/io.h>
...this will find:
/home/myusername/kernel/kernel4.4/src/arch/long/path/asm-generic/io.h

The “-I” supplements paths from includes of the format:

#include <whatever>

…this makes the code portable, especially if you plan on a later release as a library with a dev package.

8051…those are actually still sometimes used. I think I have some in a pile of advanced stone knives/chisels and stone tablets.

NOTE: Sometimes I think email notifications do not work. Some time back they never worked…they now work, but not always. Be sure to click the “Follow” icon at the top if it is present.

@linuxdev
Thank you, again.

Please note: it turns out my TX2 doesn’t have a copy of SRC with asm-generic/io.h … I was grepping my Ubuntu 16.04 LTS “host computer”, not the TX2.

Also, my grepping didn’t restrict itself to “arch/arm64”: the arch/arm64 tree doesn’t have io.h, although arch/arm does.

Should I try my hand at porting arch/arm/…/asm-generic/io.h to arm64? (a task for which I suspect I am decidedly underqualified.)

Which puts me back in the lurch: no access to io.h functions, no ioremap(), no pci_iomap(), etc.

I found “cat /proc/ioports”, and it does report the PCIe serial card and the correct address &range thereof, I just need to know how to read/write those registers. … or why my ttySn aren’t working, the root of the issue.

You’re probably using the wrong source…“include/asm-generic/io.h” definitely exists. I can’t answer whether what you are doing is correct or not, but you can start with making sure you’re using the right source.

If you have the driver package (which produces the host side “Linux_for_Tegra/” subdirectory), then you will have the script “source_sync.sh”. I’m going to suggest doing this on the Jetson itself to avoid cross compile issues (you can switch to PC host cross compile if you’re sure the rest is correct). Copy “source_sync.sh” to some location on the Jetson you want to build from. I actually put it in “/usr/local/src/” (which also implies “sudo” is needed to operate there since it is owned by root…but then I’ve used “chown” to change it to owner “ubuntu” and no longer need sudo). My example uses “/usr/local/src/”, but it could be anywhere.

# From the host PC with the driver package...
cd /where/ever/it/is/Linux_for_Tegra
scp source_sync.sh ubuntu@<your_jetson_IP_address>:~/Downloads
# From user ubuntu login on the Jetson...
sudo mkdir /usr/local/src
sudo chown ubuntu.ubuntu /usr/local/src
cp ~/Downloads/source_sync.sh /usr/local/src
cd /usr/local/src
chmod u+x source_sync.sh
./source_sync.sh -u tegra-l4t-r28.2
# You now have "sources/" subdirectory and kernel within that...

You could do the same thing from some other temporary location for testing first and then delete it, or you could use source_sync.sh from your PC in the same way. Regardless of where you run source_sync.sh from you will have this subdirectory content:

sources/kernel/kernel-4.4/include/asm-generic/io.h

Using “/usr/local/src/” as my example you could add this in order to make “<asm-generic/io.h>” available (this would be one of the options to the “cc” or “gcc” command):

-I/usr/local/src/kernel/kernel-4.4/include

You might still be mixing things together which shouldn’t be mixed, but see where this takes you. Are you using some third party code not in the kernel tree? If so, do you know which kernel version that code was intended to run with (the Jetson is using a 4.4.x kernel)?

A comment on the topic, but a bit generic, is that some of the earlier kernels did not use the IOMMU for virtual addressing on PCIe. This meant that all of the earlier code would use physical addresses in many places where current code requires a mapping to virtual addresses. If the code previously in place tries to use an address which is a physical addres…which is in reality being remapped to a virtual address through the IOMMU…then you will crash and burn. Make sure any address you are looking at hasn’t been remapped to a virtual address while still attempting to use a physical address. Errors listed in “dmesg” after an attempt to read/write might offer a clue to this.

I do not.
I am using a TX2 one of my customers shipped to me pre-configured.

I downloaded https://developer.nvidia.com/embedded/dlc/l4t-jetson-tx2-driver-package-28-1, but it doesn’t contain Linux_for_Tegra, nor source_sync.sh.

I haven’t had to deal with flashing this unit, I’m merely modifying the kernel on the volume my customer created. I suspect the “driver package” you’re referring to is inside Jetpack, the Host system component? I’ve never installed it.

Do you have a link to the proper package, Jetpack or otherwise?

Sorry for the trouble!

Unpacking that file will produce those as subdirectory content. R28.2 is current, but R28.1 is very similar. Using R28.1 as an example you will have file “Tegra186_Linux_R28.1.0_aarch64.tbz2”. To unpack from some location:

tar xvfj /where/ever/it/is/Tegra186_Linux_R28.1.0_aarch64.tbz2
cd Linux_for_Tegra
ls source_sync.sh

Just to be complete, if you were to also add the sample rootfs:

# ...still in Linux_for_Tegra/...
cd rootfs
<b>sudo</b> tar xvfj /where/ever/it/is/Tegra_Linux_Sample-Root-Filesystem_R28.1.0_aarch64.tbz2
cd ..
<b>sudo</b> ./apply_binaries.sh
# Example flash:
sudo ./flash.sh -S 28318MiB jetson-tx2 mmcblk0p1
# Example downloading kernel source for R28.1 (R28.2 is newer):
./source_sync.sh -k tegra-l4t-r28.1
cd source/kernel/kernel-4.4
ls

If your file Tegra186_Linux_R28.1.0_aarch64.tbz2 does not have subdirectory content, then the download was incorrect. Sometimes people use wget, but the web site won’t send the file with wget…it’ll send a web page.

NOTE: Behind the scenes JetPack downloads these files for you (including driver package and sample rootfs). JetPack is just a front end to the flash process and uses those command line tools behind the scenes. JetPack itself is only necessary when installing extra files after the flash…and even then if you know the package URLs you can install packages without JetPack (JetPack is recommended though, it is a real pain to do manually).

So, I’ve been dinking around. There were a bunch of issues with the definition of things not being identical between the already-installed includes and the new set, so I basically overwrote the /usr/include with /usr/local/src/sources/kernel/…/include, now those issues are gone.

My code, however, now throws an exception:

a.out[9636]: unhandled level 2 translation fault (11) at 0x00001007, esr 0x92000046
[ 9723.352616] pgd = ffffffc153cda000
[ 9723.356050] [00001007] *pgd=00000001f3a1a003, *pud=00000001f3a1a003, *pmd=0000000000000000

[ 9723.365887] CPU: 4 PID: 9636 Comm: a.out Not tainted 4.4.38-tegra #4
[ 9723.372244] Hardware name: quill (DT)
[ 9723.376135] task: ffffffc1492de400 ti: ffffffc139468000 task.ti: ffffffc139468000
[ 9723.384027] PC is at 0x4005f0
[ 9723.387138] LR is at 0x40063c
[ 9723.390136] pc : [<00000000004005f0>] lr : [<000000000040063c>] pstate: 60000000
[ 9723.397630] sp : 0000007ff4262960
[ 9723.401014] x29: 0000007ff4262970 x28: 0000000000000000 
[ 9723.406475] x27: 0000000000000000 x26: 0000000000000000 
[ 9723.411935] x25: 0000000000000000 x24: 0000000000000000 
[ 9723.417461] x23: 0000000000000000 x22: 0000000000000000 
[ 9723.422947] x21: 0000000000000000 x20: 0000000000000000 
[ 9723.428449] x19: 0000000000400690 x18: 0000000000000a03 
[ 9723.433795] x17: 0000000000411000 x16: 0000007fa1a7a7c0 
[ 9723.439260] x15: 0000007fa1bef000 x14: 0000000000000000 
[ 9723.444655] x13: 0000000000000402 x12: 0000007fa1bf0028 
[ 9723.450031] x11: 0000040000000000 x10: 0101010101010101 
[ 9723.455456] x9 : 00ffffffffffffff x8 : ffffffffffffffff 
[ 9723.460805] x7 : 0000040000000000 x6 : 0000000000000000 
[ 9723.466156] x5 : 0000000000000000 x4 : 0000007ff4262a18 
[ 9723.471568] x3 : 0000000000400648 x2 : 0000007ff4262b08 
[ 9723.476916] x1 : 00000000000000aa x0 : 0000000000001007 

[ 9723.483785] Library at 0x4005f0: 0x400000 /home/nvidia/a.out
[ 9723.489515] Library at 0x40063c: 0x400000 /home/nvidia/a.out
[ 9723.495190] vdso base = 0x7fa1bee000

Here’s my entire program:

#define __must_check		__attribute__((warn_unused_result))
#include <stdio.h>
#include <asm/io.h>

#define BAR0 (0x1000)
#define UART_SCRATCHPAD (0x07)
void main()
{
  unsigned char DataOut = 0xAA;
  writeb( DataOut, (volatile void *)(BAR0 + UART_SCRATCHPAD) );
  unsigned char DataIn = readb((const volatile void *)(BAR0 + UART_SCRATCHPAD));
  printf("Read Back %02X from PCIe UART #0 scratchpad register, should have read %02X\r\n", DataIn, DataOut);
}

Line 1 is to hack out this error:

Line 10&11 have a cast (volatile void *) and (const volatile void *) to avoid this error:

I’m going to keep hacking away. My best guess is I need to issue an ioremap or pci_iomap or something. Or run as actual root instead of ./a.out and sudo ./a.out as I’ve done so far

Just speculating, perhaps you are using a physical address which should have been mapped first to a virtual address. DMA on PCIe in earlier releases used physical addresses, but the IOMMU is now enabled and one has to use the virtual address instead. Regardless of what the final cause is, the kernel doesn’t like your process using that address.