Add Another Carveout Region (TX2)

Has anyone ever added another region into the carveout? I want to add another security aperture which only allows TrustZone access (prevents normal-world access). Is the number of carveout regions fixed at 44?

The only registers that I see on the TX2 are MC_SECURITY_CFG0, MC_SECURITY_CFG1, and MC_SECURITY_CFG3 which are specific to the TZDRAM, not generic to any aperture. How is the Memory controller programed for the other apertures?


hello richard.habeeb,

it’s by default to have 44 regions, as you can see for the carveout types,
for example,

	CARVEOUT_RESERVED1 = 42,				/* 0x2A */
	CARVEOUT_RESERVED2 = 43,				/* 0x2B */
	CARVEOUT_NUM = 44,						/* 0x2C */
	CARVEOUT_FORCE32 = 2147483647ULL		/* 0x7FFFFFFF */
} carve_out_type_t;

you should also check CBoot logs for the details of allocations.
actually, those two reserved regions were not used.

[0001.642] I> 42) Base:0x00000000 Size:0x00000000
[0001.646] I> 43) Base:0x00000000 Size:0x00000000
1 Like

Oh, the reserved regions are unused! That is very helpful.

So I know you could potentially modify the carveout list data structure in the scratch space to configure one of those reserved regions before CBboot runs. But my expectation is that the memory itself won’t be protected, that’s what the BPMP MB1/MB2 do, right?

How does the BPMP tell the memory controller to protect the carveout regions?

The MC has these relevant registers that I can find:

TZDRAM Carveout Regs

  • MC_SECURITY_CFG0_0: base address bits 0-31
  • MC_SECURITY_CFG3_0: base address bits 32-33
  • MC_SECURITY_CFG1_0: size in MB

Microcode(?) TZRAM? Carveout Regs:

  • MC_SEC_CARVEOUT_BOM_0: base address bits 0-31
  • MC_SEC_CARVEOUT_ADR_HI_0: base address bits 32-33
  • MC_SEC_CARVEOUT_REG_CTRL_0: sticky bit preventing access to security aperture registers

There are 3 additional registers associated with the VPR carveout. These three sets are probably the so-called “fixed-function” security apertures. Which the manual says this:

Note that the security checks for CCPLEX accesses will be check in the SCF. These checks match the functionality of the checks in MC, but there are slight differences in the implementation of the control registers. At boot (and whenever carveouts are changed dynamically) the values of the MC control registers need to be copied over to their counterparts in SCF.

What does this paragraph mean? It seems like the other carveout regions are not setup in the MC but instead in the SCF (System Coherence Fabric). Which I believe is another term for the Control Fabric (right?). If that’s true, then I would guess that the security apertures are programmed somehow using the SCR registers, but this is where I get lost.

hello richard.habeeb,

you could modify tegra186-mb1-bct-misc-si-l4t.cfg for TX2 to increase the tzdram size (TOS memory region).
for example, sw_carveout.tzdram_carveout_size = 0x00C00000;

it should works in theory, please validate and share the results.

Right, I’ve been curious about this before (Increase the size of the TZDRAM on TX2 - #4 by richard.habeeb). I didn’t end up testing this approach, but I might in the future when I get a chance. (I didn’t originally try this because the TX2 doesn’t have this config option, but the Xavier does according to this doc Tegra Linux Driver)

Unfortunately I want to protect devices from normal world access, and their MMIO address ranges are not adjacent to the TZDRAM, so I need a new carveout entry.

I am currently doing some digging in the L4T boolloader flashing tools to see if there are potentially options for this. That cfg file seems to be ultimately parsed by a binary called tegrabct_v2 which creates the boot_config_table and mb1_boot_config_table. I would hazard a guess that the actual carveout list lives in there somewhere.

I also noticed a perl script called which has some interesting strings like:

McGeneralizedCarveout1Bom = 0
McGeneralizedCarveout1BomHi = 0
McGeneralizedCarveout1Size128kb = 0

It looks like there are some sort of configuration options for 31 different generalized carveouts! I will continue to investigate, and report if I can find a way to configure them.

So that perl script maps McGeneralizedCarveout1Bom to MC_SECURITY_CARVEOUT1_BOM_0 which looks like a register name, but not one that i’ve seen before. I checked the CBoot sources to find this:

#define MC_SECURITY_CARVEOUT1_BOM_0                     _MK_ADDR_CONST(0xc0c)

Which is defined in hwinc-t18x/armc.h and referenced in hwinc-t18x/sdram_scratch_list_secure_generated.h

So this could be a register offset for the MC… for a register that doesn’t show up in the TRM. I bet these registers are what the MB1 uses to setup the carveout in the MC. I will try to program it in the same way that I did before.

Great news!

So it turns out that these are indeed undocumented registers for the memory controller for the generalized carveouts!! Also there are actually 29 of them which matches the number listed in the TRM. I dumped their default values on my machine:

NOTICE:  BL31: Reading Generalized Carveout Registers
NOTICE:         0: 0x277f00000
NOTICE:         1: 0x277e00000
NOTICE:         2: 0x277d00000
NOTICE:         3: 0x277c00000
NOTICE:         4: 0x277b00000
NOTICE:         5: 0x277800000
NOTICE:         6: 0x277400000
NOTICE:         7: 0x277a00000
NOTICE:         8: 0x277300000
NOTICE:         9: 0x276800000
NOTICE:         10: 0x30000000
NOTICE:         11: 0xf0000000
NOTICE:         12: 0x30040000
NOTICE:         13: 0x30048000
NOTICE:         14: 0x30049000
NOTICE:         15: 0x3004a000
NOTICE:         16: 0x3004b000
NOTICE:         17: 0x3004c000
NOTICE:         18: 0x3004d000
NOTICE:         19: 0x3004e000
NOTICE:         20: 0x3004f000
NOTICE:         21: 0
NOTICE:         22: 0xf0100000
NOTICE:         23: 0
NOTICE:         24: 0
NOTICE:         25: 0
NOTICE:         26: 0
NOTICE:         27: 0x84400000
NOTICE:         28: 0x30000000

Which matches my CBoot dump of the carveouts from the scratch:

[0030.584] I>  0) Base:0x00000000 Size:0x00000000
[0030.588] I>  1) Base:0x277f00000 Size:0x00100000
[0030.593] I>  2) Base:0x277e00000 Size:0x00100000
[0030.597] I>  3) Base:0x277d00000 Size:0x00100000
[0030.602] I>  4) Base:0x277c00000 Size:0x00100000
[0030.606] I>  5) Base:0x277b00000 Size:0x00100000
[0030.611] I>  6) Base:0x277800000 Size:0x00200000
[0030.615] I>  7) Base:0x277400000 Size:0x00400000
[0030.620] I>  8) Base:0x277a00000 Size:0x00100000
[0030.624] I>  9) Base:0x277300000 Size:0x00100000
[0030.629] I> 10) Base:0x276800000 Size:0x00800000
[0030.633] I> 11) Base:0x30000000 Size:0x00040000
[0030.638] I> 12) Base:0xf0000000 Size:0x00100000
[0030.642] I> 13) Base:0x30040000 Size:0x00001000
[0030.647] I> 14) Base:0x30048000 Size:0x00001000
[0030.651] I> 15) Base:0x30049000 Size:0x00001000
[0030.656] I> 16) Base:0x3004a000 Size:0x00001000
[0030.660] I> 17) Base:0x3004b000 Size:0x00001000
[0030.664] I> 18) Base:0x3004c000 Size:0x00001000
[0030.669] I> 19) Base:0x3004d000 Size:0x00001000
[0030.673] I> 20) Base:0x3004e000 Size:0x00001000
[0030.678] I> 21) Base:0x3004f000 Size:0x00001000
[0030.682] I> 22) Base:0x00000000 Size:0x00000000
[0030.687] I> 23) Base:0xf0100000 Size:0x00010000
[0030.691] I> 24) Base:0x00000000 Size:0x00000000
[0030.695] I> 25) Base:0x00000000 Size:0x00000000
[0030.700] I> 26) Base:0x00000000 Size:0x00000000
[0030.704] I> 27) Base:0x00000000 Size:0x00000000
[0030.709] I> 28) Base:0x84400000 Size:0x00400000
[0030.713] I> 29) Base:0x30000000 Size:0x00010000
[0030.718] I> 30) Base:0x278000000 Size:0x08000000
[0030.722] I> 31) Base:0x00000000 Size:0x00000000
[0030.727] I> 32) Base:0x276000000 Size:0x00600000
[0030.731] I> 33) Base:0x80000000 Size:0x70000000
[0030.736] I> 34) Base:0xf0110000 Size:0x1856f0000
[0030.740] I> 35) Base:0x00000000 Size:0x00000000
[0030.744] I> 36) Base:0x00000000 Size:0x00000000
[0030.749] I> 37) Base:0x2772e0000 Size:0x00020000
[0030.753] I> 38) Base:0x84000000 Size:0x00400000
[0030.758] I> 39) Base:0x96000000 Size:0x02000000
[0030.762] I> 40) Base:0x85000000 Size:0x01200000
[0030.767] I> 41) Base:0x275800000 Size:0x00500000
[0030.771] I> 42) Base:0x00000000 Size:0x00000000
[0030.776] I> 43) Base:0x00000000 Size:0x00000000

The indexes are off by one for some reason, but that’s not a big deal. I have no idea how you program all of the last 14 carveout registers, One of them is the TZDRAM, but not sure how to configure the others. This shouldn’t be an issue in theory. There seem to be 5 unused generalized carveouts that will hopefully work.

To anyone curious I am attempting to write to these registers in the ARM-TF BL31 using tegra_mc_write_32 with the offsets defined in the cboot source’s hwinc-t18x/armc.h.

The base address is split into two registers:

The size register is for bits 0-26 are X * 128KB, and for bits 27-31 are Y * 4096. You can infer this from reading the sizes and comparing to the Cboot list.

Update: writing to the generalized security carveout (GSC) registers from EL3 (!?!) doesn’t seem update the values. I can think of a few reasons:

  • these registers might have a locking feature preventing updates
  • the CCPLEX just doesn’t have access to the GSCs outside of the TZDRAM one
  • There is something going on with the ARI request section of the TRM. E.g. there is a request called UPDATE_CCPLEX_GSC, which seems highly relevant.

I will continue to investigate.

Update 2: After reading from MC_SECURITY_CARVEOUTn_CFG0 it seems that ALL of the GSCs have a lock bit set by the time the BL31 runs. I am infering that the second bit is the lock bit from these definitions found in the BL31 source (probably an artifact from something deprecated):

149 /* General Security Carveout register macros */
151 #define MC_GSC_LOCK_CFG_SETTINGS_BIT    (U(1) << 1)
152 #define MC_GSC_ENABLE_TZ_LOCK_BIT   (ULL(1) << 0)
153 #define MC_GSC_SIZE_RANGE_4KB_SHIFT U(27)
158 #define MC_GSC_ENABLE_CPU_SECURE_BIT    (U(1) << 31)

If this is true, this is a huge shame. Nvidia maintaining control of these core security features is definitely a limitation to the TX2.

So the last hope is that its feasible to setup the BCT at compile/flash time with a custom GSC. Will continue to investigate. I reaaaaly don’t want to have to reverse engineer the MB1 to do this.

Another update:

So the BCT config files seem to be the places that define some defaults for the MC generalized carveout registers (my board uses bootloader/t186ref/BCT/P3310_A00_8GB_lpddr4_A02_l4t.cfg). However, these don’t define the values that are ultimately programmed to setup the final carveout list that we see in Cboot.

SDRAM[0].McGeneralizedCarveout1Bom                = 0x00000000;
SDRAM[0].McGeneralizedCarveout1BomHi              = 0x00000000;
SDRAM[0].McGeneralizedCarveout1Size128kb          = 0x00000008;
SDRAM[0].McGeneralizedCarveout2Bom                = 0x00000000;
SDRAM[0].McGeneralizedCarveout2BomHi              = 0x00000000;
SDRAM[0].McGeneralizedCarveout2Size128kb          = 0x00000008;

I am attempting to see if changing these values for the unused carveouts has any effect. I am running into issues with the flashing tool though: TX2 Flashing Tools Bugs (using '-k BCT' option)

I still haven’t figured out if the GSCs are configurable or whether they are compiled into the MB1/MB2 binaries.

Okay, I have some new things I’ve learned on this topic. I know this topic is getting long, but this needs to be figured out.

The keywords in the file tegra186-mb1-bct-misc-si-l4t.cfg actually perfectly match the struct tegrabl_mb1_bct data structure defined in the CBoot sources. This means that this config file is able to change any of the fields in this struct (which seems obvious now looking back), by referencing their C names.

However, the MB1 BCT doesn’t contain the generalized security carveouts, only these ones:

struct tegrabl_mb1bct_sw_carveout_data {
    uint32_t version;
    struct tegrabl_mb1bct_sw_carveout_info cpubl;
    struct tegrabl_mb1bct_sw_carveout_info mb2;
    struct tegrabl_mb1bct_sw_carveout_info tzdram;
    struct tegrabl_mb1bct_sw_carveout_info ramdump;
    struct tegrabl_mb1bct_sw_carveout_info mb2_heap;
    struct tegrabl_mb1bct_sw_carveout_info cpubl_params;
    /* Resered space to store info of one more carveout. */
    uint8_t reserved[20];

So using this cfg file, you cannot configure the GSCs. The GSC data isn’t in the MB2 BCT either; however, the MB2/MB1 setup the struct tboot_cpubl_params which contain the final GSC carveouts in the field struct tegrabl_global_data.

I still haven’t tracked down if the generalized carveouts are hardcoded or whether or not they are configurable in another way, as it seems that the BCT was a red herring.