Question about memory allocation among different applications in Trusty


I am developing Trust Applications in Trusty on Jetson Xavier Device. There was one discussion about max size of port buffer in TIPC connection, and we agree that the max msg size is the same as the PAGE_SIZE which is 4KB in the kernel.

My question is what if I allocate some data with large size (> 4KB) to a structure and send only its memory address and size information as the message through the TIPC protocol, does it work legally?

This may be associated with Trusty memory allocation mechanism, I assume that the memory is not sharable among different applications in Trusty, am I right ?

Thanks in advance !

hello tianchi.yu,

it’s the bootloader allocates a dedicated carveout, TZ-DRAM, to run a secure OS. All secure operations are initiated by a client application running in the non-secure environment. A trusted application, in the secure world, never initiates contact with the non-secure environment.
please see developer guide for more details, Trusty, a Trusted Execution Environment.

Hello again,
I want to extend the question to a the specific case, which is I want to transfer a data with large size (>5MB) between CA and TA. Because of the limitation of port size through TIPC protocol, does it mean that my only choice is to split the data to small pieces and transfer them many times.

Thanks very much !

yes, your understanding is correct.

1 Like

Thanks !

Regarding allocating a dedicated carveout to run the secure OS, it seems to be possible to expand its memory by modifying value of TZDRAM, as suggested in this answer. However I didn’t find a proper path of this configuration.
The device I am using is Jetson AGX Xavier. Could you give me some helps about in which files we need to set this value, and the risk we may take if we set a very large value of memory.

Thank you !


hello tianchi.yu,

you may check carve_out_type_t for the details.
please access Jetson Linux R32.6.1 Release Page | NVIDIA Developer to download the cboot sources package, (i.e. Cboot Sources T194), and please checking header file, tegrabl_carveout_id.h for details.

you’re able to increase the TZDRAM size, please check the JetPack installation folder for the flash configuration file.
for TX2 series…
please modify tegra186-mb1-bct-misc-si-l4t.cfg to increase the tzdram size (TOS memory region). i.e. sw_carveout.tzdram_carveout_size = 0x00C00000;
for Xavier series…
you should check tegra194-mb1-bct-misc-l4t.cfg and update the settings of carveout.tzdram.size = 0x01000000; for your use-case.

Hi JerryChang,

Thanks again for your help. I find the configuration file as you said. Because I am a beginner of Xavier device, I still have some questions based on the procedure of increasing TOS memory region.

  1. After update the setting of carveout.tzdram.size, what are the next steps of increasing the TOS memory region ? And is it reversible, that I could roll back if something goes wrong?
  2. Am I supposed to flash the ubuntu OS and all packages of JetPack through sdkmanager ? Does it mean that the Xavier would be reset and I will lose all resources ? Since I have developed on the device for some time, I want to avoid ruining all dependencies and environment built already.


hello tianchi.yu,

you’ll need to re-flash the target in order to update the cfg settings.

you may see developer guide, To clone a Jetson device and flash, which backup your filesystem (i.e. system.img) to reserve your user-space contents. after that, please use the command-line to restore your backup image for flashing the target.
for example, $ sudo ./ -r jetson-xavier-nx-devkit mmcblk0p1

there’re some discussion threads you may check for reference, i.e. Topic 169760, and Topic 205693.

1 Like

It helps a lot. Could I only flash the secure-os partition ?
Command is like sudo ./ -k secure-os jetson-xavier mmcblk0p1.
Or I have to flash partition APP ?

And if I have to flash APP partition, which means it would generate a new system.img. The bad news is that I lost the original system.img but only have system.img.raw locally.
How could I backup current target device without flashing/generating a new system.img ?

hello tianchi.yu,

please check the flash configuration file, for example, jetson-xavier-nx-devkit.conf, and you’ll see the definition here… EMMC_CFG=flash_l4t_t194_spi_sd_p3668.xml;
please examine flash_l4t_t194_spi_sd_p3668.xml for all of the partition names to flash Xavier-NX DevKit.
once you assign a correct partition name to the command-line, you may use the -k options in flash script to update that partition.
so, you’re able to execute $ sudo ./ -k secure-os ... to update the Trusted OS partition.

please check my previous comments, you may see developer guide, To clone a Jetson device and flash , which backup your filesystem. i.e. APP partition.


My questions were not clear. Firsty, I want to ask if it is sufficient to increase the Trusty Memory (after configging carveout.tzdram.size) by only flashing secure-os partition.

And about backup filesystems, command sudo ./ -r -k APP -G <clone> <board> mmcblk0p1 is clear for me. The only problem is that, the current system.img file is deleted by mistake, and I don’t want to flash a new system.img which would overwrite the target device, because I did some modifications directly in device. So could I run command like sudo ./ -k APP -G <clone> <board> mmcblk0p1 without -r option, in order to generate the correct image and clone it but not flash it into the device ?

Last question, if I understand correctly, the images would store the modifications/installations that I did on the device directly. Am I right ?


hello tianchi.yu,

it’s using configuration file to update the tzdram size (TOS memory region), so you’ll need to perform full flash to update the settings.

this should works, since it’s communicate with the target to create the backup image to your host machine.

Hi JerryChang,

Thanks for your helps.
We are back with some questions on the overall process, we were not able to expand Trusty available memory space. (see more detail below)

You can find detailed and recorded process below, and our tests to validate memory expansion.

Could let us know if we’ve most probably missed something important along the process?

We also wonder how configuration changes are applied to the newly flashed TOS?

Thanks again for your support !

Trusty available RAM expansion

As previously discussed, we want to increase available memory space in Trusty Trusted Application.

After applying the suggested previously discussed modifications, we’ve followed your instructions by

Modifying the carveout.tzdram.size configuration variable


We’ve applied the same changes to the configuration files in both Linux_for_Tegra/bootloader/ and Linux_for_Tegra/bootloader/t184ref/BCT/.

Reflashing the secure-os partition


Trusty memory expansion validation

In order to check the Trusty TA memory expansion, we implemented two TAs, ECHO-SRV and SMALL-ECHO-SRV.
Both them simply collect the big float array data from Client Application (and store them, manipulate them or not) and send them back to CA.
Since the message port size is limited to 4KB, we need to send multiple rounds of data from CA to TA.

The only difference between two TAs is the maximal size of float array that could be stored in the TA.
For ECHO-SRV, the length of wanted array is 64000, for SMALL-ECHO-SRV, it is 20480.

Allocates a float array of size 20480/64000 at bootloading

all_data = (float*)malloc(MAX_SIZE*sizeof(float));

Echoes data from a CA.

We implement one CA named echo-srv-app which could connect the TIPC channel with either ECHO-SRV or SMALL-ECHO-SRV.

It is able to send a big float array with different sizes to TA, we could compare the different results from two TAs.

The CA splits the big float array to small payloads that are able to be sent to TA.

    memcpy(msg->data, data+(BLOCK_SIZE*block_nums),BLOCK_SIZE);
    /* Send msg to TA */
    printf("This is the %d round of sending data with size %d.\n",block_nums++,BLOCK_SIZE);
    write(send_srv_fd, msg, msg_size);
    read(send_srv_fd, msg, msg_size);

You can find attached TAs and CA source code. (14.2 KB) (9.0 KB) (9.1 KB)

Float array allocation(by malloc in C) seems work well, as suggested by the bootloader log.

ECHO-srv: 45: Welcome to ECHO Service(for a big float array) !
ECHO-srv: 164: Init echo-srv IPC services!!
ECHO-srv: 57: All Data has been initialized with size: 64000
SAMLL-ECHO-srv: 45: Welcome to SMALL ECHO Service(for a big float array) !
SAMLL-ECHO-srv: 164: Init echo-srv IPC services!!
SAMLL-ECHO-srv: 57: All Data has been initialized with size: 20480

But for ECHO-SRV TA applications (which echoes array with size of 64000), it crashes at reception of very first CA payload.
No matter the size of data that CA sends(e.g. 1000, 2048 or 64000).

For your information, the SMALL-ECHO-SRV TA works well for a float array of smaller size (20480).

For clarification, we’ve recorded the full process below.

In addition, we can find below timed details on the several steps.

Recorded process

The recording is available here.

TOS_Memory_Expanding.mkv (37.0 MB)

Before the recording, the target device(Jetson AGX Xavier) is in Force Recovery Mode, and both TAs have been compiled successfully.
USB Type C and USB Micro B of target device are connected with host computer.

0:00 - 1:12 : config carveout.tzdram.size in bootloader files tegra194-mb1-bct-misc-l4t.cfg and tegra194-mb1-bct-misc-flash.cfg

1:13 - 1:30 : check the flash command and its options (e.g. secure-os partition)

1:38 - 1:39 (not recorded) : flash with only secure-os partition (According to that you have commented before, this step should be flash all partitions but not only secure-os partition, are we right?)

2:00 - 2:30 : ssh connect Xavier Device and compile CA for SMALL-ECHO-SRV

2:40 - 2:50 & 4:55 - 5:10: explain how to switch CA between two TA targets.

2:51 - 4:30 : execute CA and do tests by sending different sizes of data arrays to SMALL-ECHO-SRV.

4:31 - 4:54 : modify the CA target

5:11 - 5:50 : recompile CA and execute it by sending testing data to ECHO-SRV. Note that,
the testing array sent has lenght of only 2048, and Trusy OS crashed after receiving the first payload.

5:51 - end : check the crashed reason

this topic has been marked as solved. this thread is filed for asking the carveout modification, and it’s using cfg file to increase the TZDRAM size in comment #9.
for a better support and for others to scope the solution easily, please go ahead to create a new discussion thread with your latest update, we may using new thread to follow-up.

1 Like

Thanks, please check this new thread.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.