PCIe dynamic link state management and PCIe device power management

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Dear All,

We are using a typical communication scheme between TX1 and FPGA - PCIe x4.

The whole system sleep/wake behavior is working in our system.

The problem is partial down of the FPGA to save power.
I mean we need to turn off FPGA which looks as a PCIe device for the TX1. I.e. to turn it into D3cold state.

Unfortunately our pcie-axi ip-core is not supporting any advanced power saving modes. L1 L2 link states are not available.

Any methods based on unbinging or removal PCIe drivers do not turn the PCIe link off (L3 state).

Please advise how to dynamically turn off PCIe device (link to L3, device to D3cold) and restore it back.

Thank you in advance,
Alex

2

Any ideas?

any runtime functions, sysfs files, documents, howtos…

3

No ideas? No experience?

4

Dear NVIDIA experts,

I would appreciate your comments.

5

Does your device support ASPM L1 Sub-States (L1.1 and L1.2), if yes, based on entry into L1.1 or L1.2, majority of your FPGA circuitry can be turned off.
(you have already clarified in your initial post that your device doesn’t support L1, I’m wondering if it supports ASPM-L1SS)

6

Dear vidyas,

Unfortunately, our PCIe device is not supporting substates, it should be turned off completely.

The problem is a persistent link. The PCIe host is never sleep. We cannot find a way to turn all the subsystem off keeping the rest CPU run…

Please comment.

Thank you,
Alex

7

How about this flow?

  1. Make both PCIe host controller driver and your PCIe device driver as modules
  2. Initially Boot the system without loading PCIe host controller driver
  3. Power-up your FPGA based end point
  4. load PCIe host controller driver (this enumerated your device)
  5. load your device driver and work with the device
  6. when you are done with it, rmmod your device driver first
  7. followed by rmmod'ing PCIe host controller driver
  8. power-off FPGA device

you can repeat steps 3 ~ 8 when you want to use your device again.
Does this flow work for you?

8

Hi again!

Thank you for the recommendation.

We have tried to turn off, unbind and remove the drivers via sysfs, but never through a rmmod.
So, I should check it.

But do we have a chance to reach such behavior with monolith kernel?
We had a trouble with modular drivers for our device…

9

But, unbinding the driver and rmmod’ing do the same job.
Did you try the same sequence mentioned in #7 replacing ‘rmmod’ with ‘unbind’ ?

10

Sure!

But I have written I tried to unbind both my drv and pcie host’s one, but links had kept up… :(

11

Possible to share the exact steps including commands you used?

12

Sure. I will do it little bit later…

13

Hi Vidyas,

We have found some mistakes in our sleep/wake logic in drivers.
Device can sleep if only all the children can…

Debugging…

14

Hi Vidyas,

We have rewritten all our PCIe related drivers taking into account some special power management states.
So, I should say, it was a hard job and finally code is almost completely different.
Now it works. It can sleep and awake, the PCIe subsystem can be put into low power mode or completely turned off taking all the rest in active states.

So, the problem was in code of our drivers. Drivers should be written with support of advanced power management states.

Thank you,
Alex

15

Thanks for the update and good to hear that things are working fine now.