Changing memory page size to 64KB on TX1

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Hello,

I have used M4000/M6000 GPU before where GPU memory page size was 64KB. On TX1 module it seems that the memory page size is 4KB only. I first thought just changing the kernel configuration by enabling the flag CONFIG_ARM64_64K_PAGES=y will do the trick but with this flag kernel would not build. So I want to first ask if there is any way to make the page size to 64KB.
The reason why I want 64KB page is that we are doing huge amount to DMA transfers from FPGA to GPU memory and FPGA has limited entries for translation table. Also I see that there is no rDMA support on TX1. Could someone tell me why is that?

Thanks,
RPG

2

Hi,

64 KB pages won’t work since it breaks 32-bit applications.
That is CONFIG_COMPAT(AArch32 userspace) requires !CONFIG_ARM64_64K_PAGES.
The problem is that almost all 32-bit applications are linked to start at 0x04000.
With 64 KB pages, the kernel is unable to map code at 0x04000 and a zero page at 0x00000 because pages are 0x10000 bytes long.

We don’t investigate this issue further since the lack of motivation.
We will discuss this feature internally and update information to you later.

If someone also requires this feature, please also let us know.
Thanks.

3

Hi,

Just want to confirm that is your use-case via PCIe?

4

Thanks for the reply.

Yes, it is via PCIe.

Is there any alternative to rDMA on TX1?

Also, I noticed that nVidia IPC mechanism does not exist on TX1. Could I know the reason for that ans what could be an alternative there if this is our use case:
Some memory controlled by FPGA → DMA to GPU memory on TX1 over PCIe (first app) → Access the same memory on second app

Thanks

5

Hi,

Thanks for your feedback.
We are discussing this internally, will update information to you later.

6

I’m not sure if I understood rDMA correctly in this context. By ‘rDMA’, you mean remote DMA present in TX1’s root port? If yes, then, TX1’s root port doesn’t have any DMA, although there are generic DMA engines in TX1 but they can’t be used for PCIe.

7

@vidyas - Thanks for confirming that DMA cannot be used for PCIe.