Rtcwake for nvidia tx2

Good day

I am trying to make a cronjob that will freeze or suspend TX2 with rtcwake, but only off mode is working. What should I do to understand why mem / freeze mods don’t work ?

nvidia:~$ sudo rtcwake -s 120 -m mem -v
rtcwake: assuming RTC uses UTC ...
Using UTC time.
	delta   = -1
	tzone   = 0
	tzname  = UTC
	systime = 1593787356, (UTC) Fri Jul  3 14:42:36 2020
	rtctime = 1593787357, (UTC) Fri Jul  3 14:42:37 2020
alarm 0, sys_time 1593787356, rtc_time 1593787357, seconds 120
rtcwake: wakeup from "mem" using /dev/rtc0 at Fri Jul  3 14:44:38 2020
suspend mode: mem; suspending system
rtcwake: write error
  

nvidia:~$ sudo rtcwake -s 120 -m freeze -v
rtcwake: assuming RTC uses UTC ...
Using UTC time.
	delta   = -1
	tzone   = 0
	tzname  = UTC
	systime = 1593787484, (UTC) Fri Jul  3 14:44:44 2020
	rtctime = 1593787485, (UTC) Fri Jul  3 14:44:45 2020
alarm 0, sys_time 1593787484, rtc_time 1593787485, seconds 120
rtcwake: wakeup from "freeze" using /dev/rtc0 at Fri Jul  3 14:46:46 2020
suspend mode: freeze; suspending system
rtcwake: write error

What happens for below test?

1. just call suspend:
`$ echo mem > /sys/power/state`
2. Enable alarm and go to suspend:
`$ echo +10 > /sys/class/rtc/rtc0/wakealarm; echo mem > /sys/power/state`

In both cases:

bash: echo: write error: Device or resource busy

Have sudo su then type those command.

That was under sudo su

home/nvidia# echo +10 > /sys/class/rtc/rtc0/wakealarm; echo mem > /sys/power/state
bash: echo: write error: Device or resource busy

same behaviour for freeze

Hi,
I just try the J4.3 there’s no problem to run the rtcwake

root@nvidia-desktop:/home/nvidia# cat /etc/nv_tegra_release
# R32 (release), REVISION: 3.1, GCID: 18186506, BOARD: t186ref, EABI: aarch64, DATE: Tue Dec 10 07:03:07 UTC 2019

root@nvidia-desktop:/home/nvidia# rtcwake -s 60 -m mem -v
rtcwake: assuming RTC uses UTC ...
Using UTC time.
        delta   = 0
        tzone   = 0
        tzname  = UTC
        systime = 1594800952, (UTC) Wed Jul 15 08:15:52 2020
        rtctime = 1594800952, (UTC) Wed Jul 15 08:15:52 2020
alarm 0, sys_time 1594800952, rtc_time 1594800952, seconds 60
rtcwake: wakeup from "mem" using /dev/rtc0 at Wed Jul 15 08:16:53 2020
suspend mode: mem; suspending system

root@nvidia-desktop:/home/nvidia# rtcwake -s 30 -m freeze -v
rtcwake: assuming RTC uses UTC ...
Using UTC time.
        delta   = 0
        tzone   = 0
        tzname  = UTC
        systime = 1594801158, (UTC) Wed Jul 15 08:19:18 2020
        rtctime = 1594801158, (UTC) Wed Jul 15 08:19:18 2020
alarm 0, sys_time 1594801158, rtc_time 1594801158, seconds 30
rtcwake: wakeup from "freeze" using /dev/rtc0 at Wed Jul 15 08:19:49 2020
suspend mode: freeze; suspending system

I have the same output for

cat /etc/nv_tegra_release

The dmesg looks like that

[1014756.328808] PM: suspend entry 2020-07-15 08:27:36.941717321 UTC
[1014756.328815] PM: Syncing filesystems … done.
[1014756.328984] PM: Preparing system for sleep (mem)
[1014756.330772] tegradc 15210000.nvdisplay: blank - powerdown
[1014756.331700] tegradc 15210000.nvdisplay: blank - powerdown
[1014756.338015] Freezing user space processes … (elapsed 0.003 seconds) done.
[1014756.341881] Freezing remaining freezable tasks … (elapsed 0.001 seconds) done.
[1014756.343101] PM: Suspending system (mem)
[1014756.349449] nvlogger_suspend_work
[1014756.350330] usb usb2: usb_suspend_both: status 0
[1014756.350397] usb usb1: usb_suspend_both: status 0
[1014756.351557] Wake24 for irq=241
[1014756.351562] Enabling wake24
[1014756.352275] nct1008_nct72 7-004c: success in disabling tmp451 VDD rail
[1014756.352329] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)
[1014756.352332] Enabling wake29
[1014756.352345] tegra_cec 3960000.tegra_cec: suspended
[1014756.357686] gpio tegra-gpio wake53 for gpio=159(X:7)
[1014756.357689] Enabling wake53
[1014756.361725] dpm_run_callback(): platform_pm_suspend+0x0/0x78 returns -16
[1014756.368655] PM: Device 17000000.gp10b failed to suspend: error -16
[1014756.375035] PM: Some devices failed to suspend, or early wake event detected
[1014756.382878] nvlogger_resume_work
[1014756.387004] gpio tegra-gpio wake53 for gpio=159(X:7)
[1014756.387009] Disabling wake53
[1014756.388946] extcon-gpio-states external-connection:extcon@1: Cable state:0, cable id:0
[1014756.388996] tegra_cec 3960000.tegra_cec: Resuming
[1014756.393933] tegra_cec 3960000.tegra_cec: tegra_cec_init started
[1014756.400074] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)
[1014756.400077] Disabling wake29
[1014756.400919] nct1008_nct72 7-004c: success in enabling tmp451 VDD rail
[1014756.433224] nct1008_nct72 7-004c: LOC shutdown limit 107
[1014756.433779] nct1008_nct72 7-004c: EXT shutdown limit 107
[1014756.466673] nct1008_nct72 7-004c: initial LOC temp: 56000
[1014756.466997] nct1008_nct72 7-004c: initial EXT temp: 58500
[1014756.467683] nct1008_nct72 7-004c: hi_b:-11, lo_b:0
[1014756.469547] Wake24 for irq=241
[1014756.469554] Disabling wake24
[1014756.470341] PM: resume of devices complete after 87.870 msecs

So there is a device that didn’t want to be suspended

[1014756.361725] dpm_run_callback(): platform_pm_suspend+0x0/0x78 returns -16
[1014756.368655] PM: Device 17000000.gp10b failed to suspend: error -16
[1014756.375035] PM: Some devices failed to suspend, or early wake event detected

If I understand correctly Device 17000000.gp10b is a videocard?
Can the error be related to jetson_clocks ?

Have a try below patch.

diff --git a/kernel-dts/tegra186-quill-p3310-1000-a00-00-base.dts b/kernel-dts/tegra186-quill-p3310-1000-a00-00-base.dts
index f498a54..e64c806 100644
--- a/kernel-dts/tegra186-quill-p3310-1000-a00-00-base.dts
+++ b/kernel-dts/tegra186-quill-p3310-1000-a00-00-base.dts
@@ -44,7 +44,6 @@
                bootargs ="console=ttyS0,115200";
 #endif
                stdout-path = &uarta;
-               nvidia,tegra-joint_xpu_rail;
        };

        firmware {

I don’t know where to apply that patch.
On my tx2 those files are only in binary format.

nvidia:~$ locate tegra186-quill
/boot/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb
/boot/tegra186-quill-p3310-1000-a00-00-base.dtb
/boot/tegra186-quill-p3310-1000-as-0888.dtb
/boot/tegra186-quill-p3310-1000-c03-00-base-adafruit-sph0645lm4h.dtbo
/boot/tegra186-quill-p3310-1000-c03-00-base-fe-pi-audio-z-v2.dtbo
/boot/tegra186-quill-p3310-1000-c03-00-base-hdr40.dtbo
/boot/tegra186-quill-p3310-1000-c03-00-base.dtb
/boot/tegra186-quill-p3310-1000-c03-00-dsi-hdmi-dp.dtb
/boot/tegra186-quill-p3489-0888-a00-00-base.dtb
/boot/tegra186-quill-p3489-1000-a00-00-ucm1.dtb
/boot/tegra186-quill-p3489-1000-a00-00-ucm2.dtb

Can I use dtc tools to convert dtb file to dts and then apply the patch and compile back to dtb, and finally replace the existing dtb file ?

Yes, you can use dtc.
But you need to apply the new dtb by flash the dtb partition by below command.

  1. replace the dtb file at …/JetPack_4.3_Linux_P3310/Linux_for_Tegra/kernel/dtb
  2. sudo ./flash.sh -k kernel-dtb jetson-tx2 mmcblk0p1

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fflashing.html

I have modified dts and flashed the tx2.

time sudo rtcwake -m mem -s 60
rtcwake: assuming RTC uses UTC …
rtcwake: wakeup from “mem” using /dev/rtc0 at Wed Jul 22 13:16:55 2020
rtcwake: write error

real 0m2,513s
user 0m0,008s
sys 0m0,252s

There is no error with Device 17000000.gp10b. But still rtcwake wakes up earlier than it should.

[ 465.803492] PM: suspend entry 2020-07-22 13:15:53.862933881 UTC
[ 465.803502] PM: Syncing filesystems … done.
[ 465.803695] PM: Preparing system for sleep (mem)
[ 465.806893] tegradc 15210000.nvdisplay: blank - powerdown
[ 465.809194] tegradc 15210000.nvdisplay: blank - powerdown
[ 465.811697] Freezing user space processes … (elapsed 0.004 seconds) done.
[ 465.815749] Freezing remaining freezable tasks … (elapsed 1.259 seconds) done.
[ 467.075447] PM: Suspending system (mem)
[ 467.095194] nvlogger_suspend_work
[ 467.096626] usb usb1: usb_suspend_both: status 0
[ 467.096962] usb usb2: usb_suspend_both: status 0
[ 467.097192] Wake24 for irq=241
[ 467.097220] Enabling wake24
[ 467.097922] nct1008_nct72 7-004c: success in disabling tmp451 VDD rail
[ 467.097979] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)
[ 467.097982] Enabling wake29
[ 467.097996] tegra_cec 3960000.tegra_cec: suspended
[ 467.103054] gpio tegra-gpio wake53 for gpio=159(X:7)
[ 467.103058] Enabling wake53
[ 467.123391] tegra-xudc-new 3550000.xudc: device mode off: 0
[ 467.124079] tegra-xudc-new 3550000.xudc: ep 3 disabled
[ 467.124122] tegra-xudc-new 3550000.xudc: ep 2 disabled
[ 467.124156] tegra-xudc-new 3550000.xudc: ep 5 disabled
[ 467.124185] tegra-xudc-new 3550000.xudc: ep 4 disabled
[ 467.124208] tegra-xudc-new 3550000.xudc: ep 7 disabled
[ 467.124282] tegra-xudc-new 3550000.xudc: ep 9 disabled
[ 467.124319] tegra-xudc-new 3550000.xudc: ep 13 disabled
[ 467.124355] tegra-xudc-new 3550000.xudc: ep 8 disabled
[ 467.124387] tegra-xudc-new 3550000.xudc: ep 15 disabled
[ 467.124682] android_work: sent uevent USB_STATE=DISCONNECTED
[ 467.124762] tegra-usb-cd usb_cd: notification status (0x7, 0x0)
[ 467.124771] tegra-usb-cd usb_cd: disconnected USB cable/charger
[ 467.124779] tegra-xudc-new 3550000.xudc: entering ELPG
[ 467.124932] l4tbr0: port 1(rndis0) entered disabled state
[ 467.125988] l4tbr0: port 2(usb0) entered disabled state
[ 467.126506] tegra-xudc-new 3550000.xudc: entering ELPG done
[ 467.126520] tegra-xusb 3530000.xhci: entering ELPG
[ 467.129924] tegra-xusb 3530000.xhci: entering ELPG done
[ 467.129931] Wake76 for irq=199
[ 467.129934] Wake77 for irq=199
[ 467.129937] Wake78 for irq=199
[ 467.129940] Wake79 for irq=199
[ 467.129943] Wake80 for irq=199
[ 467.129945] Wake81 for irq=199
[ 467.129948] Wake82 for irq=199
[ 467.129951] Enabling wake76
[ 467.129954] Enabling wake77
[ 467.129957] Enabling wake78
[ 467.129959] Enabling wake79
[ 467.129962] Enabling wake80
[ 467.129965] Enabling wake81
[ 467.129967] Enabling wake82
[ 467.129994] Wake73 for irq=42
[ 467.129997] Enabling wake73
[ 467.130069] tegradc 15210000.nvdisplay: suspend
[ 467.133818] gpio tegra-gpio wake71 for gpio=125(P:5)
[ 467.133823] Enabling wake71
[ 467.134264] sdhci-tegra 3440000.sdhci: Tuning done, restoring the best tap value : 61
[ 467.134887] PM: suspend of devices complete after 40.788 msecs
[ 467.137185] host1x 13e10000.host1x: suspended
[ 467.137264] PM: late suspend of devices complete after 2.370 msecs
[ 467.176259] tegra-xusb 3530000.xhci: exiting ELPG
[ 467.179996] tegra-xusb 3530000.xhci: Firmware timestamp: 2019-07-08 19:32:42 UTC, Version: 55.15 release
[ 467.180972] tegra-xusb 3530000.xhci: exiting ELPG done
[ 467.181993] PM: noirq resume of devices complete after 40.381 msecs
[ 467.182106] PM: noirq suspend of devices failed
[ 467.189582] host1x 13e10000.host1x: resuming
[ 467.190132] PM: early resume of devices complete after 3.427 msecs
[ 467.198907] sdhci-tegra 3440000.sdhci: Tuning done, restoring the best tap value : 61
[ 467.200515] vdd-1v8: voltage operation not allowed
[ 467.205363] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[ 467.212337] gpio tegra-gpio wake71 for gpio=125(P:5)
[ 467.212342] Disabling wake71
[ 467.212357] nvlogger_resume_work
[ 467.215185] tegradc 15210000.nvdisplay: resume
[ 467.215200] Wake73 for irq=42
[ 467.215204] Disabling wake73
[ 467.215215] Wake76 for irq=199
[ 467.215217] Wake77 for irq=199
[ 467.215220] Wake78 for irq=199
[ 467.215222] Wake79 for irq=199
[ 467.215225] Wake80 for irq=199
[ 467.215227] Wake81 for irq=199
[ 467.215230] Wake82 for irq=199
[ 467.215232] Disabling wake76
[ 467.215235] Disabling wake77
[ 467.215237] Disabling wake78
[ 467.215240] Disabling wake79
[ 467.215242] Disabling wake80
[ 467.215244] Disabling wake81
[ 467.215247] Disabling wake82
[ 467.215275] tegra-xudc-new 3550000.xudc: exiting ELPG
[ 467.217173] tegra-xudc-new 3550000.xudc: exiting ELPG done
[ 467.217180] tegra-xudc-new 3550000.xudc: vbus state: 1
[ 467.388546] tegra-usb-cd usb_cd: notification status (0x0, 0x7)
[ 467.388552] tegra-usb-cd usb_cd: connected to CDP
[ 467.388561] tegra-xudc-new 3550000.xudc: device mode on: 0
[ 467.388771] tegra-xudc-new 3550000.xudc: active: 0 => 1
[ 467.412826] gpio tegra-gpio wake53 for gpio=159(X:7)
[ 467.412834] Disabling wake53
[ 467.414751] extcon-gpio-states external-connection:extcon@1: Cable state:1, cable id:1
[ 467.414797] tegra_cec 3960000.tegra_cec: Resuming
[ 467.419559] tegra_cec 3960000.tegra_cec: tegra_cec_init started
[ 467.425545] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)
[ 467.425548] Disabling wake29
[ 467.426452] nct1008_nct72 7-004c: success in enabling tmp451 VDD rail
[ 467.458770] nct1008_nct72 7-004c: LOC shutdown limit 107
[ 467.459333] nct1008_nct72 7-004c: EXT shutdown limit 107
[ 467.492194] nct1008_nct72 7-004c: initial LOC temp: 39000
[ 467.492535] nct1008_nct72 7-004c: initial EXT temp: 39250
[ 467.493196] nct1008_nct72 7-004c: hi_b:-11, lo_b:0
[ 467.494979] Wake24 for irq=241
[ 467.494984] Disabling wake24
[ 467.495518] Suspended for 0.445 seconds
[ 467.495856] PM: resume of devices complete after 305.646 msecs
[ 467.505282] PM: Finishing wakeup.
[ 467.505288] Restarting tasks …
[ 467.505897] usb usb2: usb_suspend_both: status 0
[ 467.508226] done.
[ 467.508534] usb usb1: usb_suspend_both: status 0
[ 467.510831] tegra-xudc-new 3550000.xudc: ep 11 disabled
[ 467.510848] tegra-xudc-new 3550000.xudc: ep 6 disabled
[ 467.524920] PM: suspend exit 2020-07-22 13:15:56.029647609 UTC

The problem is with this device right now. Should I change some boot options?

[ 467.181993] PM: noirq resume of devices complete after 40.381 msecs
[ 467.182106] PM: noirq suspend of devices failed
[ 467.189582] host1x 13e10000.host1x: resuming
[ 467.190132] PM: early resume of devices complete after 3.427 msecs

Did you try on TX2 reference carrier board?
Any customize kernel?

No, the kernel is not customized. It is J4.3. With jetson_clocks turned on.

nvidia@nvidia:~$ cat /etc/nv_tegra_release
R32 (release), REVISION: 3.1, GCID: 18186506, BOARD: t186ref, EABI: aarch64, DATE: Tue Dec 10 07:03:07 UTC 2019

We have a TX2 dev kit

R32 (release), REVISION: 1.0, GCID: 14531094, BOARD: t186ref, EABI: aarch64, DATE: Wed Mar 13 07:41:08 UTC 2019

and rtcwake works without problems (even with jetson_clocks)

root@nvidia:/home/nvidia# time sudo rtcwake -m mem -s 60
rtcwake: assuming RTC uses UTC …
rtcwake: wakeup from “mem” using /dev/rtc0 at Thu Jul 23 13:02:00 2020

real 1m1,496s
user 0m0,004s
sys 0m0,084s

Could you check if the TX2 dev kit have the same message in your early comment?

[ 467.181993] PM: noirq resume of devices complete after 40.381 msecs
[ 467.182106] PM: noirq suspend of devices failed
[ 467.189582] host1x 13e10000.host1x: resuming
[ 467.190132] PM: early resume of devices complete after 3.427 msecs

No such message on TX2 dev kit.

[ 389.150675] PM: suspend entry 2020-07-23 15:48:55.394997937 UTC
[ 389.150682] PM: Syncing filesystems … done.
[ 389.150822] PM: Preparing system for sleep (mem)
[ 389.153579] tegradc 15210000.nvdisplay: blank - powerdown
[ 389.156682] tegradc 15210000.nvdisplay: blank - powerdown
[ 389.159544] Freezing user space processes … (elapsed 0.002 seconds) done.
[ 389.161922] Freezing remaining freezable tasks … (elapsed 0.001 seconds) done.
[ 389.163291] PM: Suspending system (mem)
[ 389.166479] [07-23 15:48:55.410] wl_cfg80211_disconnect:
[ 389.166480] usb usb1: usb_suspend_both: status 0
[ 389.166482] Reason 3
[ 389.166497] usb usb2: usb_suspend_both: status 0
[ 389.167990] sdhci-tegra 3400000.sdhci: Tuning done, restoring the best tap value : 29
[ 389.168296] Wake24 for irq=241
[ 389.168299] Enabling wake24
[ 389.169046] nct1008_nct72 7-004c: success in disabling tmp451 VDD rail
[ 389.169112] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)
[ 389.169113] Enabling wake29
[ 389.169325] gpio tegra-gpio wake53 for gpio=159(X:7)
[ 389.169328] Enabling wake53
[ 389.174128] wl_is_linkdown :
[ 389.174131] Link down Reason : WLC_E_LINK
[ 389.174139] [07-23 15:48:55.418] wl_notify_connect_status: link down if wlan0 may call cfg80211_disconnected. event : 16, reason=2 from 0c:f4:d5:3b:ac:3c
[ 389.189022] tegra-xusb 3530000.xhci: exiting ELPG
[ 389.192558] tegra-xusb 3530000.xhci: Firmware timestamp: 2018-12-26 10:30:04 UTC, Version: 55.12 release
[ 389.193568] tegra-xusb 3530000.xhci: exiting ELPG done
[ 389.193571] tegra-xusb 3530000.xhci: entering ELPG
[ 389.196439] tegra-xusb 3530000.xhci: entering ELPG done
[ 389.196447] Wake76 for irq=199
[ 389.196450] Wake77 for irq=199
[ 389.196452] Wake78 for irq=199
[ 389.196454] Wake79 for irq=199
[ 389.196456] Wake80 for irq=199
[ 389.196458] Wake81 for irq=199
[ 389.196460] Wake82 for irq=199
[ 389.196463] Enabling wake76
[ 389.196465] Enabling wake77
[ 389.196467] Enabling wake78
[ 389.196469] Enabling wake79
[ 389.196470] Enabling wake80
[ 389.196472] Enabling wake81
[ 389.196473] Enabling wake82
[ 389.196549] Wake73 for irq=42
[ 389.196552] Enabling wake73
[ 389.196607] tegradc 15210000.nvdisplay: suspend
[ 389.197602] sdhci-tegra 3400000.sdhci: Tuning done, restoring the best tap value : 29
[ 389.197645] gpio tegra-gpio wake71 for gpio=125(P:5)
[ 389.197647] Enabling wake71
[ 389.488547] wl_cfg80211_disconnect :
[ 389.488551] Link down event is not received
[ 389.496968] nvlogger_suspend_work
[ 389.497012] sdhci-tegra 3440000.sdhci: Tuning done, restoring the best tap value : 64
[ 389.497537] PM: suspend of devices complete after 331.165 msecs
[ 389.499885] host1x 13e10000.host1x: suspended
[ 389.499974] PM: late suspend of devices complete after 2.429 msecs
[ 389.504403] PM: noirq suspend of devices complete after 4.421 msecs
[ 389.504408] Disabling non-boot CPUs …
[ 389.520988] CPU3: shutdown
[ 389.523702] psci: CPU3 killed.
[ 389.552936] CPU4: shutdown
[ 389.555647] psci: CPU4 killed.
[ 389.580874] CPU5: shutdown
[ 389.583585] psci: CPU5 killed.
[ 389.585538] Entered SC7
[ 389.585538] Wake[31-0] level=0x40000000
[ 389.585538] Wake[63-32] level=0x0
[ 389.585538] Wake[95-64] level=0x7f2a0
[ 389.585538] Wake[31-0] enable=0x21000000
[ 389.585538] Wake[63-32] enable=0x200000
[ 389.585538] Wake[95-64] enable=0x7f2a0
[ 389.585538] Wake[31-0] route=0x21000000
[ 389.585538] Wake[63-32] route=0x200000
[ 389.585538] Wake[95-64] route=0x7f2a0
[ 389.585538] Wake[32:0] status=0x1000000
[ 389.585538] Resume caused by WAKE24, max77620-top
[ 389.585538] Wake[64:32] status=0x0
[ 389.585538] Wake[96:64] status=0x0
[ 389.585538] Exited SC7
[ 389.585538] bpmp: waiting for handshake
[ 389.585538] bpmp: synchronizing channels
[ 389.585538] bpmp: channels synchronized
[ 389.585538] Suspended for 20.400 seconds
[ 389.585612] Enabling non-boot CPUs …
[ 389.585814] CPU3: Booted secondary processor [411fd073]
[ 389.585954] cache: parent cpu3 should not be sleeping
[ 389.591386] CPU3 is up
[ 389.591603] CPU4: Booted secondary processor [411fd073]
[ 389.591752] cache: parent cpu4 should not be sleeping
[ 389.597151] CPU4 is up
[ 389.597343] CPU5: Booted secondary processor [411fd073]
[ 389.597505] cache: parent cpu5 should not be sleeping
[ 389.603084] CPU5 is up
[ 389.636531] tegra-xusb 3530000.xhci: exiting ELPG
[ 389.638995] tegra-xusb 3530000.xhci: Firmware timestamp: 2018-12-26 10:30:04 UTC, Version: 55.12 release
[ 389.639918] tegra-xusb 3530000.xhci: exiting ELPG done
[ 389.640863] PM: noirq resume of devices complete after 37.745 msecs
[ 389.641463] eqos 2490000.ether_qos eth0: Link is Down
[ 389.644458] host1x 13e10000.host1x: resuming
[ 389.645075] PM: early resume of devices complete after 3.371 msecs
[ 389.653040] sdhci-tegra 3440000.sdhci: Tuning done, restoring the best tap value : 64
[ 389.654560] vdd-1v8: voltage operation not allowed
[ 389.659381] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[ 389.666065] gpio tegra-gpio wake71 for gpio=125(P:5)
[ 389.666068] Disabling wake71
[ 389.666093] nvlogger_resume_work
[ 389.668533] sdhci-tegra 3400000.sdhci: Tuning done, restoring the best tap value : 29
[ 389.668625] sdhci-tegra 3400000.sdhci: Tuning done, restoring the best tap value : 29
[ 389.669778] tegradc 15210000.nvdisplay: resume
[ 389.669794] Wake73 for irq=42
[ 389.669796] Disabling wake73
[ 389.669803] Wake76 for irq=199
[ 389.669804] Wake77 for irq=199
[ 389.669805] Wake78 for irq=199
[ 389.669806] Wake79 for irq=199
[ 389.669807] Wake80 for irq=199
[ 389.669808] Wake81 for irq=199
[ 389.669809] Wake82 for irq=199
[ 389.669810] Disabling wake76
[ 389.669811] Disabling wake77
[ 389.669812] Disabling wake78
[ 389.669813] Disabling wake79
[ 389.669813] Disabling wake80
[ 389.669814] Disabling wake81
[ 389.669815] Disabling wake82
[ 389.669825] tegra-xudc-new 3550000.xudc: exiting ELPG
[ 389.671157] tegra-xudc-new 3550000.xudc: exiting ELPG done
[ 389.671163] tegra-xudc-new 3550000.xudc: vbus state: 0
[ 389.694314] gpio tegra-gpio wake53 for gpio=159(X:7)
[ 389.694316] Disabling wake53
[ 389.696024] extcon-gpio-states external-connection:extcon@1: Cable state:0, cable id:0
[ 389.696272] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)
[ 389.696274] Disabling wake29
[ 389.697669] nct1008_nct72 7-004c: success in enabling tmp451 VDD rail
[ 389.729422] nct1008_nct72 7-004c: LOC shutdown limit 107
[ 389.729922] nct1008_nct72 7-004c: EXT shutdown limit 107
[ 389.761991] nct1008_nct72 7-004c: initial LOC temp: 29000
[ 389.762288] nct1008_nct72 7-004c: initial EXT temp: 30000
[ 389.762880] nct1008_nct72 7-004c: hi_b:-11, lo_b:0
[ 389.766253] Wake24 for irq=241
[ 389.766256] Disabling wake24
[ 389.790005] sdhci-tegra 3400000.sdhci: Tuning done, restoring the best tap value : 29
[ 389.790030] PM: resume of devices complete after 144.949 msecs
[ 389.790723] PM: Finishing wakeup.
[ 389.790725] Restarting tasks … done.
[ 389.809909] usb usb2: usb_suspend_both: status 0
[ 389.810007] usb usb1: usb_suspend_both: status 0
[ 389.811513] tegra-xudc-new 3550000.xudc: entering ELPG
[ 389.812839] tegra-xudc-new 3550000.xudc: entering ELPG done
[ 389.814918] PM: suspend exit 2020-07-23 15:49:16.459394698 UTC

Could you check the wake up source for the dev kit and your carrier board.

cat /sys/kernel/debug/wakeup_sources

The difference is that the carrier board (tx2 where rtcwake doesn’t work) has additional 3960000.tegra_cec

Carrier board

nvidia@nvidia:~$ sudo cat /sys/kernel/debug/wakeup_sources
[sudo] password for nvidia: 
name					active_count	event_count	wakeup_count	expire_count	active_since	total_time	max_time	last_change	prevent_suspend_time
mmc1:0001:2                     	27811		27811		0		0		0		490179		700		583537713		0
usb2                            	0		0		0		0		0		0		0		4308		0
usb1                            	0		0		0		0		0		0		0		4308		0
bluedroid_pm                    	0		0		0		0		0		0		0		3598		0
gpio-keys                       	0		0		0		0		0		0		0		1586		0
3400000.sdhci                   	0		0		0		0		0		0		0		1326		0
c2a0000.rtc                     	867		867		0		0		0		927		40		583062728		0
max77620-rtc                    	868		868		0		0		0		86761		253		583062872		0
vbus-wakelock                   	0		0		0		0		0		0		0		1155		0
3530000.xhci                    	0		0		0		0		0		0		0		1149		0
fuse_wake_lock                  	0		0		0		0		0		0		0		1066		0
3960000.tegra_cec               	0		0		0		0		0		0		0		1048		0
alarmtimer                      	0		0		0		0		0		0		0		803		0
external-connection:extcon@1    	0		0		0		0		0		0		0		520		0
extcon-suspend-lock             	0		0		0		0		0		0		0		516		0
serial-debug                    	1		1		0		0		0		504		504		992		0
deleted                         	0		0		0		0		0		0		0		0		0

Dev kit

nvidia@nvidia:~$ sudo cat /sys/kernel/debug/wakeup_sources
[sudo] password for nvidia: 
name					active_count	event_count	wakeup_count	expire_count	active_since	total_time	max_time	last_change	prevent_suspend_time
usb2                            	0		0		0		0		0		0		0		8746		0
usb1                            	0		0		0		0		0		0		0		8746		0
mmc1:0001:2                     	16324		16324		0		0		0		120691		640		57171022		0
bluedroid_pm                    	0		0		0		0		0		0		0		6535		0
gpio-keys                       	0		0		0		0		0		0		0		4661		0
3400000.sdhci                   	0		0		0		0		0		0		0		4417		0
c2a0000.rtc                     	87		87		0		0		0		83		5		57137984		0
max77620-rtc                    	88		88		0		0		0		6879		100		57138104		0
vbus-wakelock                   	0		0		0		0		0		0		0		3242		0
3530000.xhci                    	0		0		0		0		0		0		0		3237		0
fuse_wake_lock                  	0		0		0		0		0		0		0		822		0
alarmtimer                      	0		0		0		0		0		0		0		622		0
external-connection:extcon@1    	0		0		0		0		0		0		0		528		0
extcon-suspend-lock             	0		0		0		0		0		0		0		524		0
serial-debug                    	1		1		0		0		0		528		528		1028		0
deleted                         	0		0		0		0		0		0		0		0		0

I think the tegra_cec doesn’t matter with it. My system have the tegra_cec and can suspend by rtcwake.
Does your carrier board can suspend by sudo systemctl suspend?

Carrier Board wakes up right away, Dev kit suspends normally

[112185.210490] wl_android_wifi_off in                                                                                                                                                                             
[112185.212275] dhd_wlfc_deinit():3472, maintain HOST RXRERODER flag in tvl                                                                                                                                        
[112185.213108] dhdsdio_isr : bus is down. we have nothing to do                                                                                                                                                   
[112185.213180] gpio tegra-gpio-aon wake69 for gpio=59(FF:3)                                                                                                                                                       
[112185.213190] Disabling wake69                                                                                                                                                                                   
[112185.213348] sdhci-tegra 3440000.sdhci: Tuning done, restoring the best tap value : 60                                                                                                                          
[112185.214586] wifi_platform_set_power = 0                                                                                                                                                                        
[112185.482503] CFGP2P-ERROR) wl_cfgp2p_del_p2p_disc_if :                                                                                                                                                          
[112185.482509] P2P interface unregistered                                                                                                                                                                         
[112185.541284] PM: suspend entry 2020-07-30 15:49:11.260991826 UTC                                                                                                                                                
[112185.541292] PM: Syncing filesystems ... done.
[112185.549510] PM: Preparing system for sleep (mem)
[112185.550589] tegradc 15210000.nvdisplay: blank - powerdown
[112185.554983] tegradc 15210000.nvdisplay: blank - powerdown
[112185.557040] Freezing user space processes ... (elapsed 0.003 seconds) done.
[112185.560313] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[112185.561853] PM: Suspending system (mem)
[112185.568521] nvlogger_suspend_work
[112185.569347] usb usb2: usb_suspend_both: status 0
[112185.569414] usb usb1: usb_suspend_both: status 0
[112185.570304] Wake24 for irq=241
[112185.570331] Enabling wake24
[112185.570981] nct1008_nct72 7-004c: success in disabling tmp451 VDD rail
[112185.571054] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)
[112185.571056] Enabling wake29
[112185.571070] tegra_cec 3960000.tegra_cec: suspended
[112185.576222] gpio tegra-gpio wake53 for gpio=159(X:7)
[112185.576226] Enabling wake53
[112185.595812] tegra-xusb 3530000.xhci: entering ELPG
[112185.599212] tegra-xusb 3530000.xhci: entering ELPG done
[112185.599241] Wake76 for irq=199
[112185.599245] Wake77 for irq=199
[112185.599248] Wake78 for irq=199
[112185.599251] Wake79 for irq=199
[112185.599254] Wake80 for irq=199
[112185.599256] Wake81 for irq=199
[112185.599259] Wake82 for irq=199
[112185.599262] Enabling wake76
[112185.599265] Enabling wake77
[112185.599267] Enabling wake78
[112185.831748] Disabling wake79                                                                                                                                                                                   
[112185.831753] Disabling wake80                                                                                                                                                                                   
[112185.831757] Disabling wake81                                                                                                                                                                                   
[112185.831761] Disabling wake82                                                                                                                                                                                   
[112185.831787] tegra-xudc-new 3550000.xudc: exiting ELPG                                                                                                                                                          
[112185.833755] tegra-xudc-new 3550000.xudc: exiting ELPG done                                                                                                                                                     
[112185.833762] tegra-xudc-new 3550000.xudc: vbus state: 0                                                                                                                                                         
[112185.857864] gpio tegra-gpio wake53 for gpio=159(X:7)                                                                                                                                                           
[112185.857869] Disabling wake53                                                                                                                                                                                   
[112185.859807] extcon-gpio-states external-connection:extcon@1: Cable state:0, cable id:0                                                                                                                         
[112185.859867] tegra_cec 3960000.tegra_cec: Resuming                                                                                                                                                              
[112185.864750] tegra_cec 3960000.tegra_cec: tegra_cec_init started                                                                                                                                                
[112185.870878] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)                                                                                                                                                       
[112185.870883] Disabling wake29                                                                                                                                                                                   
[112185.871736] nct1008_nct72 7-004c: success in enabling tmp451 VDD rail                                                                                                                                          
[112185.904050] nct1008_nct72 7-004c: LOC shutdown limit 107                                                                                                                                                       
[112185.904616] nct1008_nct72 7-004c: EXT shutdown limit 107                                                                                                                                                       
[112185.937456] nct1008_nct72 7-004c: initial LOC temp: 36000                                                                                                                                                      
[112185.937790] nct1008_nct72 7-004c: initial EXT temp: 36250                                                                                                                                                      
[112185.938493] nct1008_nct72 7-004c: hi_b:-11, lo_b:0                                                                                                                                                             
[112185.940332] Wake24 for irq=241                                                                                                                                                                                 
[112185.940339] Disabling wake24                                                                                                                                                                                   
[112185.941144] PM: resume of devices complete after 120.884 msecs                                                                                                                                                 
[112185.949674] PM: Finishing wakeup.                                                                                                                                                                              
[112185.949681] Restarting tasks ...                                                                                                                                                                               
[112185.953543] tegra-xudc-new 3550000.xudc: entering ELPG                                                                                                                                                         
[112185.953836] done.                                                                                                                                                                                              
[112185.955202] usb usb1: usb_suspend_both: status 0                                                                                                                                                               
[112185.955285] usb usb2: usb_suspend_both: status 0                                                                                                                                                               
[112185.960957] PM: suspend exit 2020-07-30 15:49:11.680662579 UTC                                                                                                                                                 
[112185.961708] tegra-xudc-new 3550000.xudc: entering ELPG done                                                                                                                                                    
[112186.032505] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready                                                                                                                                                 
[112186.078653] tegradc 15210000.nvdisplay: blank - powerdown                                                                                                                                                      
[112186.111282] gpio tegra-gpio wake18 for gpio=101(M:5)                                                                                                                                                           
[112186.113608] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready                                                                                                                                                 
[112186.115658] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready                                                                                                                                                
[112186.115784]                                                                                                                                                                                                    
                                                                                                                                                                                                                   
                Dongle Host Driver, version 1.201.82 (r)                                                                                                                                                           
                Compiled from                                                                                                                                                                                      
[112186.115794] wl_android_wifi_on in                                                                                                                                                                              
[112186.115799] wifi_platform_set_power = 1                                                                                                                                                                        
[112186.326902] vdd-1v8: voltage operation not allowed                                                                                                                                                             
[112186.331903] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[112186.358835] vdd-1v8: voltage operation not allowed
[112186.363850] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[112186.391849] vdd-1v8: voltage operation not allowed
[112186.396917] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[112186.406227] vdd-1v8: voltage operation not allowed
[112186.411169] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[112186.526892] vdd-1v8: voltage operation not allowed
[112186.531911] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[112186.541180] vdd-1v8: voltage operation not allowed
[112186.546141] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[112186.558488] vdd-1v8: voltage operation not allowed
[112186.563607] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[112186.570509] sdhci-tegra 3440000.sdhci: Tuning done, restoring the best tap value : 60
[112186.572060] F1 signature read @0x18000000=0x17214354
[112186.582178] F1 signature OK, socitype:0x1 chip:0x4354 rev:0x1 pkg:0x2
[112186.583549] DHD: dongle ram size is set to 786432(orig 786432) at 0x180000
[112186.762852] dhdsdio_write_vars: Download, Upload and compare of NVRAM succeeded.
[112186.826551] dhd_bus_init: enable 0x06, ready 0x06 (waited 0us)
[112186.827594] gpio tegra-gpio-aon wake69 for gpio=59(FF:3)
[112186.827706] gpio tegra-gpio-aon wake69 for gpio=59(FF:3)
[112186.827718] Enabling wake69
[112186.830898] wifi_platform_get_mac_addr
[112186.835770] Firmware up: op_mode=0x0005, MAC=00:04:4b:c5:8f:6d
[112186.845769] clm path from default:/lib/firmware/brcm/bcmdhd.clm_blob
[112186.845876] Skipping the clm download. len:0 memblk:          (null)
[112186.852777] dhd_preinit_ioctls pspretend_threshold for HostAPD failed  -23
[112186.864314] Firmware version = wl0: May 17 2019 16:59:40 version 7.35.349.79 (r714996 CY) FWID 01-e527e2ad
[112186.871523] dhd_interworking_enable: failed to set WNM info, ret=-23
[112186.874693] tegra_cec 3960000.tegra_cec: tegra_cec_init Done.

You may need to figure which device cause the system can’t suspend.