Mapping QSPI flash as a block device in Linux

Is it possible to map QSPI flash as a Linux block device? Final goal is to read partitions that are not present on emmc, like EKS.

Writing could be useful too, but reading is absolutely necessary.

hello initrd.img

why you said EKS is not present on eMMC? there’re A_eks and B_eks partitions that flashed eks image to AGX Orin DevKit.
did you meant QSPI flash to external storage?

There are two flash devices on Orin - QSPI and EMMC.

QSPI holds most of boot partitions - EKS, TOS, UEFI image, MB1, MB2, etc.
EMMC holds ESP, kernel, dtb and rootfs.

By default only EMMC is visible on Linux:

root@orin:~# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0          7:0    0   16M  1 loop 
mmcblk0      179:0    0 59.3G  0 disk 
├─mmcblk0p1  179:1    0 57.8G  0 part /
├─mmcblk0p2  179:2    0  128M  0 part 
├─mmcblk0p3  179:3    0  768K  0 part 
├─mmcblk0p4  179:4    0 31.6M  0 part 
├─mmcblk0p5  179:5    0  128M  0 part 
├─mmcblk0p6  179:6    0  768K  0 part 
├─mmcblk0p7  179:7    0 31.6M  0 part 
├─mmcblk0p8  179:8    0   80M  0 part 
├─mmcblk0p9  179:9    0  512K  0 part 
├─mmcblk0p10 179:10   0   64M  0 part 
└─mmcblk0p11 179:11   0    1G  0 part 

root@orin:~# parted /dev/mmcblk0
GNU Parted 3.3
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Warning: Not all of the space available to /dev/mmcblk0 appears to be used, you
can fix the GPT to use all of the space (an extra 6 blocks) or continue with the
current setting? 
Fix/Ignore? i                                                             
Model: MMC G1M15M (sd/mmc)
Disk /dev/mmcblk0: 63.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name                Flags
 2      20.5kB  134MB   134MB                A_kernel            msftdata
 3      134MB   135MB   786kB                A_kernel-dtb        msftdata
 4      135MB   168MB   33.2MB               A_reserved_on_user  msftdata
 5      168MB   302MB   134MB                B_kernel            msftdata
 6      302MB   303MB   786kB                B_kernel-dtb        msftdata
 7      303MB   336MB   33.2MB               B_reserved_on_user  msftdata
 8      336MB   420MB   83.9MB               recovery            msftdata
 9      420MB   421MB   524kB                recovery-dtb        msftdata
10      421MB   488MB   67.1MB  fat32        esp                 boot, esp
11      488MB   1562MB  1074MB               reserved            msftdata
 1      1562MB  63.7GB  62.1GB  ext4         APP                 msftdata

I’m looking for a way to access the bootloader partitions, both to read and to do OTA update.

hello makkarpov,

may I have more details for doing this by yourself, since there’s bootloader update tool, for example, Update and Redundancy.

The most of the problems with many jetpack solutions is that they are designed for a very specific narrow use case, and mostly undocumented/unsupported if you want to use them even slightly outside of these cases. Usually it’s easier to write a solution from scratch than to adapt existing ones.

E.g. for mentioned update engine:

  • The current Update Engine only supports updates to slots A and B at the same time.

    What if I want to do A/B update without touching current slot?

  • What is the format of update payload? How to generate it without using megabytes of bash scripts present in SDK?

  • (for overall OTA update engine, not the bootloader part) What if OTA payload does not fit in the remaining flash space and streaming is required?

I’m trying to avoid using nVidia-provided tooling at all costs, with exception of very low-level utilities like tegrarcm, tegrabct, tegrahost and similar. Most of the jetpack tooling code is very rigid and has a litte space for adjustments. Moreover, it’s tightly coupled and written in a very cryptic and obscure manner (these bash scripts, yes). That also makes it very difficult to debug and diagnose. It’s good for flashing jetsons to factory-default config, but not for anything else.


Reasons for reading EKS are more simple. I store a device config archive (~100kb) in EKS alongside with keys. It is too heavy to be stored in EKB directly, and is used in userspace instead of trusted OS. So the xavier solution was to query TOS for keys and then read and decrypt the userspace config. Config contains authentication certificates and private keys, so EKS is a rather logical place to hold it.

I could probably survive without bootloader updates (or updating both slots on same time), but ~100kb of EKS data is just too much to handle in trusted OS.

1 Like

hello makkarpov,

please check developer guide, Tool for EKB Generation, it’s key assignments to create the eks image for flashing to the target. also, there’s crypto service, that TOS code parses EKS.

Well, that’s not funny anymore. It’s third message roundtrip mentioning “try X, try Y” and not getting any closer to the topic question.

I had read all these manuals, they are not too long, you know. And I also tried passing 100 kb EKB to TOS, and it won’t even start. On Xaviers and JP 4.6 it will emit some weird error to UART and skip TOS boot completely (IIRC, this was few months ago). Probably the content didn’t fit in TZRAM, idk. I don’t see any section “TOS does not boot with weird bootloader error” in developer guide. Given that I don’t need that data in TZRAM at all, I considered cutting EKB to ~1 kb of TOS keys and decrypting the rest from userspace. That worked. And this is all relating to EKB. It already works as is, but requires userspace to read EKS partition.

I don’t ask messages on forums before searching for manuals and reading them. I could discuss some implementation details on existing solution, but please don’t steal Google’s job by pointing me to the document which don’t seem to be changed from Xavier ages.

The original topic of the question is “How to access QSPI flash from Linux userspace”. Ok, not as block device (but Linux almost definitely has corresponding driver). Probably as /dev/spi node and bunch of ioctl’s. But how? QSPI probably isn’t mounted on carrier board — carrier board schematics are useless here. Which bus? Which pin is CS pin? Are there some DTB snippets?

1 Like

QSPI flashed partitions are not available via Linux user-space.

QSPI flashed partitions are not available via Linux user-space.

This turned out to be false. They are available right from the start, without any further config, but via MTD subsystem of Linux. It’s mapped as /dev/mtd0 and can be accessed as an ordinary device, albeit with specific set of ioctl’s and interaction patterns when writing.

You can do modprobe mtdblock to use that as a block device, but this is of little use as QSPI does not contain primary GPT. You will see 64M /dev/mtdblock0, but you won’t see any partitions on it. Probably it could be done via mtdpart to slice up partitions, but it fails for some reason (probably kernel isn’t configured for that).

The primary issue is that you have to parse secondary GPT by hand. Linux won’t help you with that. This is not that complicated, but you still have to do it. You have to read secondary GPT header from the last block (512b) of flash device, parse it, walk the partition table and determine partition offsets. After that you can finally read or write the partition.

For anyone else searching for this, this is how to access “not available” flash partitions (simplified for brevity):

#include <string.h>
#include <stdio.h>
#include <stdint.h>

#include <fcntl.h>
#include <unistd.h>
#include <mtd/mtd-user.h>
#include <sys/ioctl.h>
#include <malloc.h>

typedef struct {
    uint8_t  _reserved0[72];
    uint64_t partitionsOffset;
    uint32_t partitionsCount;
    uint32_t partitionSize;
    uint8_t  _reserved1[424];
} __attribute__((packed)) gpt_header_t;

typedef struct {
    uint8_t _reserved0[32];
    uint64_t lbaStart;
    uint64_t lbaEnd;
    uint64_t flags;
    uint16_t name[36];
} __attribute__((packed)) gpt_entry_t;

enum { SECTOR_SIZE = 512 };

_Static_assert(sizeof(gpt_header_t) == 512, "misaligned gpt_header_t");

static int readBlock(int fd, uint64_t offset, void *target, size_t length) {
    if (lseek(fd, (int64_t) offset, SEEK_SET) < 0) {
        perror("readBlock(seek)");
        return 1;
    }

    off_t ofs = 0;
    while (ofs < length) {
        ssize_t rd = read(fd, (uint8_t*) target + ofs, length - ofs);
        if (rd <= 0) {
            perror("readBlock(read)");
            return 1;
        }

        ofs += rd;
    }

    return 0;
}

static char strcmp_w(uint16_t *a, char *b) {
    while (*a == *b && *a != 0) { a++; b++; }
    return *a == 0 && *b == 0;
}

static void hexDump(const uint8_t *data, size_t size) {
    uint64_t ofs = 0;
    uint32_t lineSize = 16;
    while (ofs < size) {
        printf("%08lx:", ofs);

        for (uint32_t i = 0; i < lineSize; i++) printf(" %02x", data[ofs + i]);
        printf(" |");

        for (uint32_t i = 0; i < lineSize; i++) {
            char c = data[ofs + i];
            if (c < 32 || c >= 127) c = '.';
            printf("%c", c);
        }

        printf("|\n");
        ofs += lineSize;
    }


    printf("%08lx total", ofs);
}

int main() {
    int fd = open("/dev/mtd0", O_RDWR);
    if (fd == -1) {
        perror("open(mtd0)");
        return 1;
    }

    mtd_info_t info;
    if (ioctl(fd, MEMGETINFO, &info) != 0) {
        perror("ioctl(MEMGETINFO)");
        return 1;
    }

    // Seek to secondary GPT header:
    gpt_header_t header;
    if (readBlock(fd, info.size - SECTOR_SIZE, &header, sizeof(header)) != 0) {
        fprintf(stderr, "readBlock(gptHeader) failed\n");
        return 1;
    }

    // Skipped verification of GPT header magic, CRC32, etc.
    size_t partitionsSize = header.partitionSize * header.partitionsCount;
    uint8_t *partitions = malloc(partitionsSize);
    if (readBlock(fd, SECTOR_SIZE * header.partitionsOffset, partitions, partitionsSize) != 0) {
        fprintf(stderr, "readBlock(gptTable) failed\n");
        return 1;
    }

    for (int i = 0; i < header.partitionsCount; i++) {
        gpt_entry_t *entry = (gpt_entry_t*) (partitions + i * header.partitionSize);
        if (entry->lbaStart == 0 || !strcmp_w(entry->name, "A_eks")) {
            continue;
        }

        // lbaEnd is inclusive, so total size is +1.
        size_t dataSize = (entry->lbaEnd - entry->lbaStart + 1) * SECTOR_SIZE;
        if (dataSize > 0x4000) dataSize = 0x4000; // limit output to reasonable length for readability

        uint8_t *data = malloc(dataSize);
        if (readBlock(fd, SECTOR_SIZE * entry->lbaStart, data, dataSize) != 0) {
            fprintf(stderr, "readBlock(data) failed\n");
            return 1;
        }

        hexDump(data, dataSize);
        free(data);
    }

    free(partitions);
    return 0;
}


@JerryChang If you don’t know what to anwser - no answer (or plain “IDK”) is much better than false and misleading answers. I’m not that deep into embedded Linux development (I usually do much more high-level things), so I didn’t knew that Linux has separate subsystem for raw flash devices. Pointing to MTD subsystem would be much more helpful than giving two irrelevant links and saying that it’s impossible.

which release you’re using? the access was there for r34.x release but it’s removed for r35.x release.

JP 5.0.2 on Orin devboard.

# R35 (release), REVISION: 1.0, GCID: 31250864, BOARD: t186ref, EABI: aarch64, DATE: Thu Aug 11 03:40:29 UTC 2022

Why ‘nvbugs’ tag, by the way? Availability is a bug? It looks like a feature, because otherwise no OTA updaters would work - they are by no way special from any other userspace software.

I don’t see any way to make that access easier, due to missing primary GPT. And primary GPT is missing probably because BCT must start right from the first byte.

hello makkarpov,

it has removal of CCPLEX access to QSPI in L4T. it’s not disable for r35.1 completely.
eventually, QSPI access would be locked, and OS won’t be able to access anything on QSPI.

Just curious, what is the motivation behind this decision? Why nVidia is explicitly locking QSPI access from the OS? I don’t see any benefit from that, but I do see considerable amount of troubles caused by it.

As this lockout is a purely software-based (since now everything works fine), will it be possible to disable it somehow via configs? Having QSPI flash access is really convenient for many things.

it should make the product as much robust as possible, having CCPLEX access to QSPI violates platform security.

You mean safety-oriented applications? As content on QSPI is already signed and encrypted by keys unknown to OS, I don’t see any security violations, only safety maybe.

And again, will there be any way to explicitly disable that limitation?

No, as mentioned previously, this is not in the plan to do so.
In coming release, QSPI access would be locked, and OS won’t be able to access anything on QSPI.

1 - What component will be able to access it after lock? Trusted OS? UEFI? Or everything running on main CPU will be locked out?

2 - How OTA update will run afterwards? What will be the “magic component” responsible for transferring OTA image from filesystem to QSPI? Currently nv_update_engine just uses /dev/mtdblock0 to write, and so it will stop working too.

1 Like

hello makkarpov,

>> Q1
it’s UEFI, Trusted-OS, RCE firmware,…etc.

>> Q2
for r35.1, you may use update engine to update QSPI. (because UEFI capsule update service has not yet complete)

OTA tool package can be used to create OTA payload package for updating system.
you may have Bootloader Update Payload (BUP) to update bootloader. please see-also Generating the Bootloader Update Payload.
note, Image-based OTA also supports to do cross version update, for example, from r32.7 to r35.1 upgrade.
currently, Image-based OTA update does not support update bootloader only or filesystem only.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.