Carveout TZDRAM Modification and TOS memory region expansion

Hi ,

We want to expand Trusty available memory space in Jetson AGX Xavier Device, there were some discussions in another topic here.

We show more details below based on the overall process, where we were not able to expand Trusty memory space.

You can find detailed and recorded process below, and our tests to validate memory expansion.

Could let us know if we’ve most probably missed something important along the process?

We also wonder how configuration changes are applied to the newly flashed TOS?

Thanks for your support !

Trusty available RAM expansion

As previously discussed, we want to increase available memory space in Trusty Trusted Application.

After applying the suggested previously discussed modifications, we’ve followed your instructions by

Modifying the carveout.tzdram.size configuration variable


We’ve applied the same changes to the configuration files in both Linux_for_Tegra/bootloader/ and Linux_for_Tegra/bootloader/t184ref/BCT/.

Reflashing the secure-os partition


Trusty memory expansion validation

In order to check the Trusty TA memory expansion, we implemented two TAs, ECHO-SRV and SMALL-ECHO-SRV.
Both them simply collect the big float array data from Client Application (and store them, manipulate them or not) and send them back to CA.
Since the message port size is limited to 4KB, we need to send multiple rounds of data from CA to TA.

The only difference between two TAs is the maximal size of float array that could be stored in the TA.
For ECHO-SRV, the length of wanted array is 64000, for SMALL-ECHO-SRV, it is 20480.

Allocates a float array of size 20480/64000 at bootloading

all_data = (float*)malloc(MAX_SIZE*sizeof(float));

Echoes data from a CA.

We implement one CA named echo-srv-app which could connect the TIPC channel with either ECHO-SRV or SMALL-ECHO-SRV.

It is able to send a big float array with different sizes to TA, we could compare the different results from two TAs.

The CA splits the big float array to small payloads that are able to be sent to TA.

    memcpy(msg->data, data+(BLOCK_SIZE*block_nums),BLOCK_SIZE);
    /* Send msg to TA */
    printf("This is the %d round of sending data with size %d.\n",block_nums++,BLOCK_SIZE);
    write(send_srv_fd, msg, msg_size);
    read(send_srv_fd, msg, msg_size);

You can find attached TAs and CA source code. (14.2 KB) (9.0 KB) (9.1 KB)

Float array allocation(by malloc in C) seems work well, as suggested by the bootloader log.

ECHO-srv: 45: Welcome to ECHO Service(for a big float array) !
ECHO-srv: 164: Init echo-srv IPC services!!
ECHO-srv: 57: All Data has been initialized with size: 64000
SAMLL-ECHO-srv: 45: Welcome to SMALL ECHO Service(for a big float array) !
SAMLL-ECHO-srv: 164: Init echo-srv IPC services!!
SAMLL-ECHO-srv: 57: All Data has been initialized with size: 20480

But for ECHO-SRV TA applications (which echoes array with size of 64000), it crashes at reception of very first CA payload.
No matter the size of data that CA sends(e.g. 1000, 2048 or 64000).

For your information, the SMALL-ECHO-SRV TA works well for a float array of smaller size (20480).

For clarification, we’ve recorded the full process below.

In addition, we can find below timed details on the several steps.

Recorded process

The recording is available here.

TOS_Memory_Expanding.mkv (37.0 MB)

Before the recording, the target device(Jetson AGX Xavier) is in Force Recovery Mode, and both TAs have been compiled successfully.
USB Type C and USB Micro B of target device are connected with host computer.

0:00 - 1:12 : config carveout.tzdram.size in bootloader files tegra194-mb1-bct-misc-l4t.cfg and tegra194-mb1-bct-misc-flash.cfg

1:13 - 1:30 : check the flash command and its options (e.g. secure-os partition)

1:38 - 1:39 (not recorded) : flash with only secure-os partition (According to that you have commented before, this step should be flash all partitions but not only secure-os partition, are we right?)

2:00 - 2:30 : ssh connect Xavier Device and compile CA for SMALL-ECHO-SRV

2:40 - 2:50 & 4:55 - 5:10: explain how to switch CA between two TA targets.

2:51 - 4:30 : execute CA and do tests by sending different sizes of data arrays to SMALL-ECHO-SRV.

4:31 - 4:54 : modify the CA target

5:11 - 5:50 : recompile CA and execute it by sending testing data to ECHO-SRV. Note that,
the testing array sent has lenght of only 2048, and Trusy OS crashed after receiving the first payload.

5:51 - end : check the crashed reason

hello tianchi.yu,

could you please review the TA stack size allocation,
for example,


could you please enlarge the stack size and try again,

1 Like

Hi, JerryChang,

Thanks, it helps! While I found that enlarging the heap size by TRUSTY_APP_CONFIG_MIN_HEAP_SIZE helps the program to store larger data allocated, but TRUSTY_APP_CONFIG_KEY_MIN_STACK_SIZE does not. It makes sense because ‘malloced’ data is stored in heap.

Follow up the process, there is a new question:
I enlarged TRUSTY_APP_CONFIG_MIN_HEAP_SIZE to 200*MIN_HEAP_SIZE which is 200*4*4KB = 3.2MB, it works well(or even 300 times MIN_HEAP_SIZE). While I keep enlarging it, the Trusty(secure-os) may be unable to flashed, all CPUs cannot be allocated properly.
What is the limitation for TRUSTY HEAP SIZE ? Is it defined also by carveout.tzdram.size?


it’s TOS carveout region to define the TZDRAM, which is a software carveout. this is defined by carveout.tzdram.size,

If I understand correctly, we only should keep the sum of allocated HEAP_SIZE and STACK_SIZE smaller that carveout.tzdram.size and re-size each of them as we wish, is that correct?

If so, that would be the solution I would take. Thanks !

hello tianchi.yu

yes, please have implementation with that, you may open a new discussion thread for further supports.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.