SPI TX Time Variability

System Specs:

Model: Jetson AGX Orin – Jetpack 5.1 [L4T 35.2.1]

Platform:

  • Distribution: Ubuntu 20.04 (Focal)
  • Kernel Release: 5.10.104-tegra

Good morning,

I am working with SPI communication on the Jetson AGX Orin and am encountering variability in TX times when sending data via SPI. While most transfers complete in approximately 100µs, I frequently observe significant outliers, which can impact applications requiring fast and consistent communication. I’ve attached a graph illustrating this variation.

I reviewed the Orin Jetson Series Pinmux Config Template to identify potential optimisations, but couldn’t figure out any obvious changes in order to make progress. Additionally, I have tested with Jetson clocks enabled to ensure maximum performance, which has noticeably improved results compared to when they are disabled (the graph attached reflects that).

I’d appreciate any guidance on improving SPI transfer consistency. Are there specific configurations, kernel parameters, or NVIDIA documentation that might help us reduce these outliers? Any relevant examples or insights would be greatly appreciated.

Thanks in advance for your help!

Hi,
On Jetson platforms, we support using peripherals like CAN, SPI, I2C, UART… etc. for the user.
For CAN usage:
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/HR/ControllerAreaNetworkCan.html
For the connection of peripheral on the devkit, please refer to expansion-headers guide and carrier board specification:
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/HR/ConfiguringTheJetsonExpansionHeaders.html

For pin configuration, please refer to pinmux spreadsheet:
https://developer.nvidia.com/embedded/secure/jetson/agx_orin/jetson_agx_orin_pinmux_config_template.xlsm

By default, the configuration is used for the devkit. If you are using the custom carrier board, please configure it according to your custom board design.

There are also several examples which have been verified from us, please check
https://elinux.org/Jetson/L4T/peripheral/
Please share the full dmesg and device tree for us to check your status in detail.

Thanks!

Hi carolyuu,

Thank you for replying and for the docs and links provided.

Please find the dmesg log and also the .dts file content below. For clarity, this is how I obtained them:

  1. sudo dmesg > dmesg.log
  2. dtc -I dtb -O dts -o tegra234-p3701-0000-p3737.dts /boot/dtb/kernel_tegra234-p3701-0000-p3737-0000.dtb

Please note that I changed the .dts file extension to .txt only so I could upload the file as the file uploader does not seem to support .dts extension.

tegra234-p3701-0000-p3737_dts.txt (525.8 KB)
dmesg.log (87.3 KB)

Please let me know if you can spot anything in there that may be related to the issue in discussion.

Thanks!

Hi francisco.silverio,

Are you using the devkit or custom board for AGX Orin?

Is your issue about the delay in SPI data transaction rather than it transfer failed?

It seems no error in the dmesg you shared.

Could you update to the latest Jetpack 5.1.4(r35.6.0) to verify if there’s still similar issue?

Please also try to apply the patch from Orin NX SPI delay between transfers - #21 by KevinFFF to check if it could help.

Hi KevinFFF,

Thank you for your reply.

I am using the AGX Orin Developer Kit with no custom configuration. The only modification I made was enabling the SPI pins via jetson-io.py. I have since tested this on Jetpack 6.2 (L4T 36.4.3), and I’m still encountering similar issues with SPI performance.

Here are the details of my setup:

Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 6.2 [L4T 36.4.3]
NV Power Mode[0]: MAXN
Serial Number: [XXX Show with: jetson_release -s XXX]
Hardware:
 - P-Number: p3701-0000
 - Module: NVIDIA Jetson AGX Orin
Platform:
 - Distribution: Ubuntu 22.04 Jammy Jellyfish
 - Release: 5.15.148-tegra
jtop:
 - Version: 4.3.1
 - Service: Active
Libraries:
 - CUDA: 12.6.68
 - cuDNN: 9.3.0.75
 - TensorRT: 10.3.0.30
 - VPI: 3.2.4
 - Vulkan: 1.3.204
 - OpenCV: 4.12.0-dev - with CUDA: YES

I’ve modified the spidev_test.c code from this repository to print the time it takes for SPI transfers (specifically the write time), and I keep getting values around 300 microseconds, with jitter present as well. Below is an example output:

$ sudo ./spidev_test -D /dev/spidev0.0 -v -p "TestTransferTime"

spi mode: 0x0
bits per word: 8
max speed: 500000 Hz (500 KHz)
SPI write took: 306 microseconds
TX | 54 65 73 74 54 72 61 6E 73 66 65 72 54 69 6D 65 __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __  | TestTransferTime
RX | FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __  | ................

Even with higher frequencies (10 MHz, 20 MHz), the results remain the same. I am not testing loopback feedback—there’s nothing connected to the MOSI and MISO pins, so I am only measuring the transfer time itself. Is this method appropriate for testing?

Since I’ve encountered the same jitter and latency issues with both Jetpack 5.1 and Jetpack 6.2, I wanted to ask if the patch mentioned in this forum post would still be applicable to my setup. If so, could you provide some instructions on how to apply it?

Thanks again for your support!

I think it is available for JP5.x and would help for the delay issue in SPI.
If you are using JP6.x, please use the following one:

diff --git a/drivers/spi/spi-tegra114.c b/drivers/spi/spi-tegra114.c
index 9fa66500ab54..020c8217ee11 100644
--- a/drivers/spi/spi-tegra114.c
+++ b/drivers/spi/spi-tegra114.c
@@ -155,6 +155,7 @@
 #define RX_FIFO_FULL_COUNT_ZERO			SPI_RX_FIFO_FULL_COUNT(0)
 #define MAX_HOLD_CYCLES				16
 #define SPI_DEFAULT_SPEED			25000000
+#define AUTOSUSPEND_TIMEOUT			300 /* in millisec */
 
 struct tegra_spi_soc_data {
 	bool has_intr_mask_reg;
@@ -1024,7 +1025,8 @@ static int tegra_spi_setup(struct spi_device *spi)
 	tegra_spi_writel(tspi, tspi->def_command1_reg, SPI_COMMAND1);
 	spin_unlock_irqrestore(&tspi->lock, flags);
 
-	pm_runtime_put(tspi->dev);
+	pm_runtime_mark_last_busy(tspi->dev);
+	pm_runtime_put_autosuspend(tspi->dev);
 	return 0;
 }
 
@@ -1425,6 +1427,8 @@ static int tegra_spi_probe(struct platform_device *pdev)
 
 	init_completion(&tspi->xfer_completion);
 
+	pm_runtime_set_autosuspend_delay(&pdev->dev, AUTOSUSPEND_TIMEOUT);
+	pm_runtime_use_autosuspend(&pdev->dev);
 	pm_runtime_enable(&pdev->dev);
 	if (!pm_runtime_enabled(&pdev->dev)) {
 		ret = tegra_spi_runtime_resume(&pdev->dev);
@@ -1453,7 +1457,8 @@ static int tegra_spi_probe(struct platform_device *pdev)
 	tspi->spi_cs_timing1 = tegra_spi_readl(tspi, SPI_CS_TIMING1);
 	tspi->spi_cs_timing2 = tegra_spi_readl(tspi, SPI_CS_TIMING2);
 	tspi->def_command2_reg = tegra_spi_readl(tspi, SPI_COMMAND2);
-	pm_runtime_put(&pdev->dev);
+	pm_runtime_mark_last_busy(&pdev->dev);
+	pm_runtime_put_autosuspend(&pdev->dev);
 	ret = request_threaded_irq(tspi->irq, tegra_spi_isr,
 				   tegra_spi_isr_thread, IRQF_ONESHOT,
 				   dev_name(&pdev->dev), tspi);
@@ -1477,6 +1482,7 @@ static int tegra_spi_probe(struct platform_device *pdev)
 	pm_runtime_disable(&pdev->dev);
 	if (!pm_runtime_status_suspended(&pdev->dev))
 		tegra_spi_runtime_suspend(&pdev->dev);
+	pm_runtime_dont_use_autosuspend(&pdev->dev);
 	tegra_spi_deinit_dma_param(tspi, false);
 exit_rx_dma_free:
 	tegra_spi_deinit_dma_param(tspi, true);
@@ -1528,7 +1534,8 @@ static int tegra_spi_resume(struct device *dev)
 	tegra_spi_writel(tspi, tspi->command1_reg, SPI_COMMAND1);
 	tegra_spi_writel(tspi, tspi->def_command2_reg, SPI_COMMAND2);
 	tspi->last_used_cs = master->num_chipselect + 1;
-	pm_runtime_put(dev);
+	pm_runtime_mark_last_busy(dev);
+	pm_runtime_put_autosuspend(dev);
 
 	return spi_master_resume(master);
 }

And also apply the patch from Jetson orin nano SPI Speed not changing - #9 by KevinFFF in case you hit any CS related issue.

To apply the patch and rebuild kernel image, please refer to Kernel Customization — NVIDIA Jetson Linux Developer Guide for details.

I successfully applied the recommended patch to the SPI driver (spi-tegra114.c), along with the CS pin patch, and built a custom kernel on JetPack 5.1 (L4T 35.2.1). The kernel was compiled directly on the Jetson, without cross-compilation

Here are the steps taken:

  1. Download sources from https://developer.nvidia.com/embedded/jetson-linux-r3521:

    • copy link from Driver Package (BSP) Sources
  2. On the Jetson:

    mkdir custom-kernel
    cd custom-kernel
    
    wget https://developer.nvidia.com/downloads/public-sourcestbz2 # copied on prev step
    tar -xvf public-sourcestbz2
    
    cd Linux_for_Tegra/source/public/kernel
    tar -xvf kernel_src.tbz2
    
    cd kernel-5.10
    

    These steps will make files available such as the SPI driver we want to patch (Linux_for_Tegra/source/public/kernel/kernel-5.10/drivers/spi/spi-tegra114.c).

  3. After applying the suggest patch for JP 5.1 in Orin NX SPI delay between transfers - #21 by KevinFFF and the CS pin patch from Jetson orin nano SPI Speed not changing - #9 by KevinFFF we get the resulting file:

spi-tegra114-patch-JP5.1-l4t35.2.1.c.txt (57.5 KB)
(note that I had to change the file extension to .txt to upload, but that is a .c file)

  1. Copy the contents of that file into Linux_for_Tegra/source/public/kernel/kernel-5.10/drivers/spi/spi-tegra114.c and save it.

  2. Get possible default kernel configurations:

    $ ls custom-kernel/Linux_for_Tegra/source/public/kernel/kernel-5.10/arch/arm/configs/
    defconfig                tegra_defconfig
    defconfig_debug.config   tegra_defconfig_debug.config
    tegra_android_defconfig
    
  3. Kernel build and installation:

	cd custom-kernel/Linux_for_Tegra/source/public/kernel

	# Load the correct kernel configuration
	sudo make O=../build/ tegra_defconfig

	# Prepare the kernel build environment
	sudo make O=../build/ modules_prepare

	# Compile the full kernel (including changes to SPI driver)
	sudo make O=../build/ -j$(nproc)

	# Compile and install kernel modules (including spi-tegra114.ko)
	sudo make O=../build/ modules_install

	# Update module dependencies
	sudo depmod -a

	# Copy the new kernel Image to the boot directory
	sudo cp ../build/arch/arm64/boot/Image /boot/Image-Custom-Kernel
  1. Update the boot configuration
  • Check the current contents of /boot/extlinux/extlinux.conf. In my case, there was a config labelled as JetsonIO that was being used and running successfully. So I copied that and only changed the boot label and path to the new kernel image:

TIMEOUT 30
#DEFAULT JetsonIO
DEFAULT CustomKernelTest

LABEL CustomKernelTest
     MENU LABEL Custom Kernel Test
     LINUX /boot/Image-Custom-Kernel
     FDT /boot/kernel_tegra234-p3701-0000-p3737-0000-user-custom.dtb
     INITRD /boot/initrd
     APPEND ${cbootargs} root=PARTUUID=4c1c9478-c986-4839-8cc2-f773fbfac931 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0

LABEL JetsonIO
	MENU LABEL Custom Header Config: <HDR40 User Custom [2024-10-08-125801]>
	LINUX /boot/Image
	FDT /boot/kernel_tegra234-p3701-0000-p3737-0000-user-custom.dtb
	INITRD /boot/initrd
	APPEND ${cbootargs} root=PARTUUID=4c1c9478-c986-4839-8cc2-f773fbfac931 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0

Be aware that if the kernel image used by CustomKernelTest is faulty, the Jetson may fail to boot. In such a case, you’ll need to connect a monitor and keyboard to manually roll back to the JetsonIO option (in this example) during boot.
It appears that /boot/extlinux/extlinux.conf supports configuring a fallback kernel, based on the default comments in the file. However, I have not explored this option.

  1. Make a note of what uname -r returns to compare after rebooting.

  2. Reboot the Jetson.

  3. Run uname -r, it should have changed from the previous value. In my case it changed from 5.10.104-tegra to 5.10.104 (I basically used this to flag that something had actually changed after the reboot).

  4. When back up:

	# Enable spidev
	sudo modprobe spidev
	$ lsmod | grep spi
	spidev                 28672  0
	spi_tegra114           28672  0

Which I believe is what’s required to run SPI applications on the Jetson.

  1. Run your application.

Before:

Avg: ~240 µs
Median: 240 µs
Stdev: ~51 µs

After

Avg: ~58 µs
Median: 56 µs
Stdev: ~31 µs

The patch seemed to result in better SPI performance and more consistent write times.

As a sanity check confirmed on a logic analyser that the code was indeed writing out data via SPI and got the expected byte sequence:

I will try next on Jetpack 6.2.

Extra: Deploying a Kernel image to multiple devices

  • It’s also possible to reuse the new kernel image in other Jetson Orin devices running the same Jetpack and L4T versions. On the target device:

    • copy the newly created kernel image to /boot
    • if the source device is currently running the new kernel image, get (maybe zip) the contents of ls /lib/modules/$(uname -r)/ from the source device and copy (and unzip) to /lib/modules/ on the target device
    • edit /boot/extlinux/extlinux.conf accordingly as explained above
    • reboot the target device
1 Like

Glad to hear that, and thank you for providing the detailed verification steps regarding the current issue.

For completeness’ sake, here are some instructions that worked for me with JetPack 6.2 (L4T 36.4.3):

How to Patch the SPI Driver on JetPack 6.2 (L4T 36.4.3)

Kernel Build & Patch Process

# Set up the kernel source directory
mkdir custom-kernel
cd custom-kernel

# Download the L4T BSP
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v4.3/release/Jetson_Linux_r36.4.3_aarch64.tbz2
tar xfv Jetson_Linux_r36.4.3_aarch64.tbz2

cd Linux_for_Tegra

# Apply your patch to the SPI driver
vim source/kernel/kernel-jammy-src/drivers/spi/spi-tegra114.c

# Download and extract the root filesystem
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v4.3/release/Tegra_Linux_Sample-Root-Filesystem_r36.4.3_aarch64.tbz2
tar -xvf Tegra_Linux_Sample-Root-Filesystem_R36.4.3_aarch64.tbz2 -C rootfs/

# Download and extract Ubuntu Jammy sources
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v4.3/sources/ubuntu_jammy-l4t_aarch64_src.tbz2
tar -xvf ubuntu_jammy-l4t_aarch64_src.tbz2 -C rootfs/

# Compile the kernel
cd source
sudo make -C kernel

# Set module install path
export INSTALL_MOD_PATH=../rootfs/

# Install kernel and modules
sudo -E make install -C kernel

cd kernel/kernel-jammy-src/
sudo make modules
sudo make modules_prepare
sudo -E make modules_install

# Update initrd
sudo nv-update-initrd

# Copy the new kernel image
cp arch/arm64/boot/Image /boot/ImageSPIPatch

# Update extlinux.conf to use the new kernel
sudo vim /boot/extlinux/extlinux.conf

# Reboot into the new kernel
sudo reboot

Results

  • The SPI write times observed when using the new patched kernel image, are in line with what was already mentioned in my previous post regarding Jetpack 5.1 (~60 µs per write instead of ~300 µs), jitter spikes still occur as shown in this figure:

Average: 68.91 µs, Median: 68.00 µs, Std Dev: 57.47 µs

Alternate Method: Patch Without Rebuilding the Kernel (applies to JP5.1 and JP6.2):

  • At the end of previous section, a new kernel object file for the spi-tegra driver would have been created in /lib/modules/$(uname -r)/kernel/drivers/spi/spi-tegra114.ko. It’s likely that not all steps above are required to create that file, I didn’t make a note of exactly which one does it.

  • As far as I could understand, the SPI driver is compiled as a module (m), and in that case you can simply replace the .ko file with the one created by the steps in the previous section (or download it below from the Artefacts section), without rebuilding the kernel. I confirmed that that was the case by running:

    zcat /proc/config.gz | grep CONFIG_SPI_TEGRA114
    
    # Output should be: CONFIG_SPI_TEGRA114=m
    

Patch Steps (JetPack 5.1 & 6.2):

  1. On the target device, replace the existing .ko file:

     sudo cp spi-tegra114-with-patch.ko /lib/modules/$(uname -r)/kernel/drivers/spi/spi-tegra114.ko
    
  2. Regenerate module dependencies:

    sudo depmod -a
    
  3. Reboot:

    sudo reboot
    

Artefacts

Final Remarks:

  • This solution was reached through a combination of interpreting NVIDIA documentation, trial and error, and informed experimentation. If there’s a better way to achieve the same result—or if this approach introduces issues I haven’t yet encountered—feedback is very welcome and encouraged.
    Uploading: spi_latency_patch_JP6.2.png…
1 Like

The Jetpack 5.1 SPI driver patch I uploaded above is incorrect. The CS patch from Jetson orin nano SPI Speed not changing - #9 by KevinFFF suggests: tspi->use_hw_based_cs = true; but it seems like that is incorrect since on compiling we get an error suggesting that it should be tspi->is_hw_based_cs = true; instead. The file below reflects that and should be used as the SPI driver patch for JP 5.1:

spi-tegra114-patch-JP5.1-l4t35.2.1.v2.0.c.txt (57.5 KB)

For JP5.1, I think you only need to apply the following patch.

You don’t need to apply SPI CS related patch for SPI driver of JP5.x

In that case, for Jetpack5.1. removing the CS related patch, results in:
spi-tegra114-patch-JP5.1-l4t35.2.1.v4.0.c.txt (57.4 KB)

which creates this kernel object file (zipped for upload to work) after compilation:
spi-tegra114.ko.zip (196.6 KB)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.