Secure OS / Rootfs Encryption

[greg2]

Hi,

We would like to ship the platform & SW we’ve built on our Jetson in portions of the world frequently known for IP theft. As such, we want to lower the risk of a 3rd party accessing the binaries released on our Jetsons & reverse engineering them to steal algorithmic IP. The applications we have mostly reside in user space, in the rootfs, and we add a few kernel drivers after boot.

In an ideal world we’d also like to protect against the imaging of the Jetson’s eMMC (by decrypting with an unreadable fuse in SBK or similar stored on the Jetson) so the eMMC will not work if placed in another Jetson. The reason for that is somebody blackbox copying our firmware.

After digging around it seems the answer to this (at least the first part) is “Secure OS”, but I can’t find many references to it, time table, documentation, etc.

I see in the thread below as of late March of 2018 there was no firm schedule yet. We were wondering if the status of the schedule has changed and/or what the status of Secure OS is.
https://devtalk.nvidia.com/default/topic/1002938/jetson-tx2/linux-support-of-secure-os/post/5247116/#5247116

If the status of Secure OS is not near completion or completed; or if I’m misunderstanding the use-case and that’s not what we’re actually looking for, is there an alternative supported by the Jetson for encrypting the rootfs and protecting our binaries?

Thanks

[linuxdev]

I can’t answer much on Secure OS, but can probably help reduce some of the confusion.

When you have Secure OS, then your boot environment will refuse to boot if it has been modified. This in no way (by itself) helps in stopping copy of rootfs. Further down there is some explanation of why you don’t want the boot environment to be allowed modification.

You can encrypt the rootfs partition (or at least partitions your software uses) and prevent meaningful copy of the unmounted partition. This would for example require a key or password to mount. If you didn’t use Secure OS to protect the boot environment, then someone could easily modify boot to publish the password or key content, and so the encryption would not matter (the modifications could just echo the steps to mount for anyone to see). Secure OS can protect any partition encryption’s key/password. The partition encryption bypasses the ability to use cloning or dd methods to extract content.

Unfortunately, even if your partition is encrypted, then once it is mounted the content itself becomes unencrypted. You’d need to then think about how to deal with that, and this is the hardest part.

Now imagine you enabled SElinux (and this is not used normally on L4T, but the Linux kernel can easily support this). It might be possible to add an SElinux context which is accessible only by some particular special user (or some special circumstance to read). Someone with root access could normally change the context. On the other hand, if the context rules were themselves somehow protected and read-only for that partition, then even a root user could not change the SElinux context. This is beyond me to say how to do this, but what I am trying to point out is that saving data from theft requires several combined steps. The ability to protect the boot environment is first because it can protect any special methods you provide for boot, e.g., protect how an encrypted partition is granted access. SElinux is quite complicated, but I suspect you would benefit by studying how it works and what you can do with it.

Right now one of the biggest problems I see is that I have not successfully seen a way to build a useful initrd, and if you want your rootfs to be something like FuseFS or encrypted, then you might need an initrd. The easy workaround would be that your content of the rootfs be quite minimal, and after Linux runs, mount something else over the top of some subset of rootfs (e.g., once booted mount an encrypted smaller partition over “/lib/modules/$(uname -r)/”). By doing this the bootloader does not need to understand an encrypted partition. Assuming not everything needs to be encrypted simplifies some of the boot issues while complicating the overall scheme. The boot scheme could in fact demand a checksum of the unencrypted “simple” initial rootfs before allowing boot…this wouldn’t stop someone from modifying the unencrypted parts, but it could prevent modified code from becoming part of boot (in which case Secure OS would require the checksum test to not be modified or viewed for checksum value or algorithm).

Consider the “/boot/Image” file…it can’t be encrypted because U-Boot needs to load it. You could modify the boot environment to demand Image has a particular checksum.

[greg2]

I can’t answer much on Secure OS, but can probably help reduce some of the confusion.

When you have Secure OS, then your boot environment will refuse to boot if it has been modified. This in no way (by itself) helps in stopping copy of rootfs. Further down there is some explanation of why you don’t want the boot environment to be allowed modification.

To me it sounds like you’re describing Secure Boot, not Secure OS. Or am I mistaken?

You can encrypt the rootfs partition (or at least partitions your software uses) and prevent meaningful copy of the unmounted partition. This would for example require a key or password to mount. If you didn’t use Secure OS to protect the boot environment, then someone could easily modify boot to publish the password or key content, and so the encryption would not matter (the modifications could just echo the steps to mount for anyone to see). Secure OS can protect any partition encryption’s key/password. The partition encryption bypasses the ability to use cloning or dd methods to extract content.

Well it seems with the secure boot fuses there’s some unreadable write-only write-once fuses for storing the keys for Secure Boot. These should be able to also store the decryption keys for Secure OS, as they can’t be replaced and can’t be read. We’d realistically need hardware support like this in order to do what we want to do correctly & not have the decryption key available, to the best of my knowledge.

Unfortunately, even if your partition is encrypted, then once it is mounted the content itself becomes unencrypted. You’d need to then think about how to deal with that, and this is the hardest part.

Yeah, I feel like once it’s mounted there’s all sorts of avenues possible. Should be able to disable a lot of the debugging facilities through the Secure Boot fuses, which should [hopefully] prevent it from being mountable, debugger access, etc. I’m not entirely sure how to deal with it when it’s mounted. I guess it depends on a variety of factors with the behavior including what happens if it’s not unmounted properly, or if there’s any other ways to get it to “not encrypt” when unmounted.

Now imagine you enabled SElinux (and this is not used normally on L4T, but the Linux kernel can easily support this). It might be possible to add an SElinux context which is accessible only by some particular special user (or some special circumstance to read). Someone with root access could normally change the context. On the other hand, if the context rules were themselves somehow protected and read-only for that partition, then even a root user could not change the SElinux context. This is beyond me to say how to do this, but what I am trying to point out is that saving data from theft requires several combined steps. The ability to protect the boot environment is first because it can protect any special methods you provide for boot, e.g., protect how an encrypted partition is granted access. SElinux is quite complicated, but I suspect you would benefit by studying how it works and what you can do with it.

Yeah but I think root will always be able to remount with write option, unless it’s read-only in HW.

Right now one of the biggest problems I see is that I have not successfully seen a way to build a useful initrd, and if you want your rootfs to be something like FuseFS or encrypted, then you might need an initrd. The easy workaround would be that your content of the rootfs be quite minimal, and after Linux runs, mount something else over the top of some subset of rootfs (e.g., once booted mount an encrypted smaller partition over “/lib/modules/$(uname -r)/”). By doing this the bootloader does not need to understand an encrypted partition. Assuming not everything needs to be encrypted simplifies some of the boot issues while complicating the overall scheme. The boot scheme could in fact demand a checksum of the unencrypted “simple” initial rootfs before allowing boot…this wouldn’t stop someone from modifying the unencrypted parts, but it could prevent modified code from becoming part of boot (in which case Secure OS would require the checksum test to not be modified or viewed for checksum value or algorithm).

Consider the “/boot/Image” file…it can’t be encrypted because U-Boot needs to load it. You could modify the boot environment to demand Image has a particular checksum.

Yeah, we can definitely make the boot sequence (in terms of the bootloader & kernel) simpler with a 2-stage boot style like that, but I think the problem is that in order to mount the encrypted partition post boot the OS still needs to have the key stored somewhere in the open.

Traditionally full disk encryption relies on a user password to decrypt, but the idea with this is to avoid that.

I think at the end of the day these are all SW approaches which require user input/action; and to do this correctly we need HW support. I believe we need HW support that uses one of the write-only write once fuses (or similar) as a decryption key for the whole root partition, so that everything including the kernel, is encrypted & decrypted in HW.

[linuxdev]

To reiterate, I can’t say much about Secure OS versus Secure Boot, so on. I’m simply describing how the stages of boot can be made to do what you want…how to achieve each step is something I’m not particularly good at describing (I don’t know enough about the specifics of a TX2). However, you cannot have a secure o/s without the boot being secure because a non-secure boot is essentially allowing a “man in the middle” attack. Someone else with detailed knowledge of the evolving secure boot and/or secure o/s could perhaps provide more detail.

Fuses are indeed able to force certain media to be signed or denied if not signed properly. This is the part for preventing a modified boot environment. Keep in mind that if such fuses did not prevent an end user from tampering with the boot loader, then the fuse settings themselves could be monitored and a replacement boot media added to allow man in the middle (malware could search for a useful man in the middle boot modification and then pick some method of getting a hash collision with the original checksum if the fuse did not make some of that information private). However, there are further stages to boot, so it isn’t enough to secure boot itself.

The beauty of enforced SElinux is that even root can be prevented from doing specific tasks, e.g., remount. If you have a means of guaranteeing the kernel is not modified during boot, then SElinux can guarantee access to change the kernel or various partitions is denied…even to root. It is a case of setting up SElinux labels and assigning roles. An example would be that perhaps there is a web admin, and the web admin has sudo access. Normally sudo would allow manipulating partitions or adding software to one of the bin directories, but SElinux could intervene and stop this; or the other way around, an actual root login might not be expected to modify web server content, and only the web admin should do so…SElinux can deny root logins from modifying content. This is nice because someone getting access to an account which can escalate to root access will still be denied certain access. SElinux is complicated (and considers roles, not just users), but you might want to read up on use-case examples to see what it can do prior to trying to learn it. For example, this Google search:
[url]"SElinux" use cases - Google Search

If the boot media is designed to not tell the end user what any key or password is, and if the boot media must be signed such that only your version can be used, then it would become quite difficult to get that information through any “normal” means (I guess it depends on the motivation and resources of the person who has access) even if boot media contains a plain text password or key. On the other hand, if SElinux has removed the ability to remount, then even the password won’t be useful without some means of cloning the rootfs (SElinux would refuse to do anything useful with the password). It is only a thought since I couldn’t tell you how to actually achieve this…but the point is to describe ideas behind what an attacker might do, and ideas on what to look at to prevent this.

Hardware alone might do the job, but at this point I don’t know if the TX2 is able to do that, e.g., such as via the built in support of a hard drive for total encryption. Someone else will need to answer that.

[greg2]

To reiterate, I can’t say much about Secure OS versus Secure Boot, so on. I’m simply describing how the stages of boot can be made to do what you want…how to achieve each step is something I’m not particularly good at describing (I don’t know enough about the specifics of a TX2). However, you cannot have a secure o/s without the boot being secure because a non-secure boot is essentially allowing a “man in the middle” attack. Someone else with detailed knowledge of the evolving secure boot and/or secure o/s could perhaps provide more detail.

Fuses are indeed able to force certain media to be signed or denied if not signed properly. This is the part for preventing a modified boot environment. Keep in mind that if such fuses did not prevent an end user from tampering with the boot loader, then the fuse settings themselves could be monitored and a replacement boot media added to allow man in the middle (malware could search for a useful man in the middle boot modification and then pick some method of getting a hash collision with the original checksum if the fuse did not make some of that information private). However, there are further stages to boot, so it isn’t enough to secure boot itself.

The beauty of enforced SElinux is that even root can be prevented from doing specific tasks, e.g., remount. If you have a means of guaranteeing the kernel is not modified during boot, then SElinux can guarantee access to change the kernel or various partitions is denied…even to root. It is a case of setting up SElinux labels and assigning roles. An example would be that perhaps there is a web admin, and the web admin has sudo access. Normally sudo would allow manipulating partitions or adding software to one of the bin directories, but SElinux could intervene and stop this; or the other way around, an actual root login might not be expected to modify web server content, and only the web admin should do so…SElinux can deny root logins from modifying content. This is nice because someone getting access to an account which can escalate to root access will still be denied certain access. SElinux is complicated (and considers roles, not just users), but you might want to read up on use-case examples to see what it can do prior to trying to learn it. For example, this Google search:
https://www.google.com/search?source=hp&ei=1GAFXIzDGJfA0PEPzb6fkA0&q=%22SElinux%22+use+cases&btnK=Google+Search&oq=%22SElinux%22+use+cases&gs_l=psy-ab.3…0.869.5187…5657…0.0…0.119.1989.5j14…2…0…1…gws-wiz…0…0i131j0i10j0i22i30.Q0fGxA621do

If the boot media is designed to not tell the end user what any key or password is, and if the boot media must be signed such that only your version can be used, then it would become quite difficult to get that information through any “normal” means (I guess it depends on the motivation and resources of the person who has access) even if boot media contains a plain text password or key. On the other hand, if SElinux has removed the ability to remount, then even the password won’t be useful without some means of cloning the rootfs (SElinux would refuse to do anything useful with the password). It is only a thought since I couldn’t tell you how to actually achieve this…but the point is to describe ideas behind what an attacker might do, and ideas on what to look at to prevent this.

Hardware alone might do the job, but at this point I don’t know if the TX2 is able to do that, e.g., such as via the built in support of a hard drive for total encryption. Someone else will need to answer that.

Yeah, I’m not sure if the TX2 is capable either, I don’t know enough about it and what facilities it may or may not have. I see what you’re saying about SELinux, it may prove to be a decent deterrent. Unfortunately though I don’t think there’s anything magical about the MMC, I think it can just be removed from the Jetson & used as a slave somewhere else. The only mitigation to this is full system encryption.

You’re right, even with proper hardware support some “man in the middle” attack may be possible, in order to be perfect it needs to encrypt on the lines. Though this goes back to how much we do (or don’t) know about the system.

Hopefully an Nvidia mod out there can see this and shed some light on Secure OS status, or other verified encrypted rootfs methods.

[linuxdev]

If the eMMC is encrypted, then you are safe only if the key is not obtained (well, this assumes it can’t be brute force cracked). One of the routes of an attacker is to find the keys (which is the only route if this is strong encryption and extreme cracking power isn’t available). A point of vulnerability is that at some stage software must decrypt or pass the key…this is where preventing boot environment changes are important…it means they can’t modify the boot environment during the passing of a key.

The fuse with signed content works at a stage even before that.

Having SElinux enforced can prevent certain exploits from a running system, which is the final stage of boot. For example, reading memory directly with an escalation of privilege or even from a privileged account which shouldn’t be doing something. To disable this the system must be shut down and the kernel edited or replaced (or kernel modules)…but between fuses and keys being protective of boot environment, then barring other flaws this can’t be done.

I do not know what the current state is for Secure OS on L4T.