Most CHERIoT work to date has been done on software or FPGA simulations. We have several such implementations: The executable model built from our formal ISA specification, the MPact simulator from Google, Microsoft’s CHERIoT SAFE FPGA target for the Arty A7, and of course lowRISC’s beautiful Sonata FPGA board, which is designed to simulate CHERIoT systems. These were always intended to be developing and prototyping systems, so I’m delighted to announce that SCI Semiconductor has the first silicon CHERIoT implementation.

[ Conflict disclaimer: I am a co-founder of SCI Semiconductor. ]

The dev board pictured above contains one of the first batch of ICENI chips to come back from the fab. This is a complete CHERIoT system, with all of the core CHERI properties (spatial memory safety, no pointer injection, and so on) along with all of the CHERIoT extensions that provide deterministic use-after-free protection, auditable control over interrupt state, and everything that we need for an aggressively compartmentalised RTOS.

This chip uses the CHERIoT Ibex core, running at up to 250 MHz, and includes a few feature that accelerate temporal safety, improve interrupt determinism, and so on. These build on top of all of the benefits of any CHERIoT implementation: deterministic mitigation of memory safety bugs from simple buffer overflows up to use-after-free, fine-grained compartmentalisation, and a programming model co-designed with both the ISA and the software stack to provide a tiny TCB. Anything that works on CHERIoT SAFE or Sonata should be very easy to port to ICENI for production use. Anything that runs on the software simulators should just work.

We’ll be showing the chips at Embedded World (Stand 4A - 131) next week and at CHERI Blossoms a couple of weeks later. From tomorrow, one will also be on display in the CHERI 15th anniversary exhibit in the Cambridge Computer Laboratory.

Aside: The Iceni tribe were one of the pre-Roman tribes in Britain and are famous for their chariots (though more due to this statue than historical fact). I am only partially to blame for the bad puns in the naming.

The top three (CWEs 79, 89, and 352) are not typically applicable on embedded platforms. Two are cross-site things that apply to web applications, one is SQL injection.

At position 4, we have CWE-862, missing authorisation. This is not something that’s automatically mitigated by CHERIoT, but the design of CHERIoT RTOS and the programming model that we expose makes it easy to write code that avoids this kind of issue. The CHERIoT pattern for any operation that you do on behalf of another compartment is to require an authorising capability. For example, if you allocate memory, you must present an allocation capability that encapsulates your right to allocate memory (and your quota). If you want to create a socket, you must present a capability that authorises you to bind to a specific port (for server sockets) or connect to a specific remote host. The same applies for dynamically created things, such as sockets themselves, message-queue endpoints, and so on. If you forget to authorise something, it will not have the capability to perform the action and the action will fail. This is a general property of capability systems and not something specific to CHERIoT.

Position 5 is CWE-787, out-of-bounds write, also known as a buffer overflow. This one is deterministically mitigated by any CHERI platform.

Technically, CHERIoT is not vulnerable to the path-traversal bugs in position 6 (CWE-22), but only because we don’t yet ship a filesystem. But, again, this kind of issue is a solved problem in capability systems. Capsicum, for example, eliminates this kind of vulnerability and I’d expect our filesystem APIs to follow a similar shape. There’s no excuse for writing APIs that are vulnerable to path traversal in the 2020s.

The next two are good old-fashioned memory-safety vulnerabilities. Position 7 is use after free (CWE-416), and 8 is out-of-bounds read (CWE-125). The latter is mitigated by any CHERI platform. The former is usually made unexploitable by CHERI systems, and is deterministically mitigated by CHERIoT.

The next two are unlikely to apply to embedded platforms. CWE-78 at position 9 is largely to do with failing to validate dynamically created command lines that you pass to a shell. Then CWE-94 (Improper Control of Generation of Code) at position 10 is typically introduced with scripting languages producing output that can be influenced by an attacker and is then executed by another interpreter, a rare situation on embedded devices.

Position 11 (CWE-120) is a ‘Classic Buffer Overflow’, i.e. something that CHERI deterministically mitigates. Not to be confused with the buffer overflow we had at position 5.

The 12th entry is another that’s rare on embedded devices. CWE-434 relates to unrestricted uploads of dangerous file types, something that matters a lot to web apps and far less to other classes of program.

Next, position 13, is a null-pointer dereference where a valid pointer was expected (CWE-476). CHERI guarantees that this will trap (even if an attacker can provide arbitrary offsets to the null pointer) and CHERIoT makes this a recoverable error either via our scoped error handlers or by simply unwinding the compartment to the caller.

Buffer overflows seem to be popular and position 14 is the third instance of this kind to make the list, this time on the stack (CWE-121). Again, this will deterministically trap on any CHERI platform.

The next entry is more interesting. Unsafe deserialisation of untrusted data (CWE-502) is something a lot of people get wrong. Phil Day wrote about how to do this safely a couple of years ago. Lightweight compartmentalisation makes it easy to limit the scope of damage that this kind of bug can do, to almost nothing.

Did I mention that buffer overflows are a recurring theme on this list? Position 16 (CWE-122) is yet another buffer overflow, this time on the heap. One more that any CHERI platform deterministically mitigates.

Positions 19–21 all relate to incorrect access control at the application layer and, sadly, are not mitigated by CHERI. Position 24 is similar.

In between these, we have another web app problem (CWE-918, server-side request forgery) and another command injection (CWE-77). These are unlikely to be present on embedded devices.

Finally, at position 25, we have a fairly broad category of availability issues that arise from not constraining resource allocation (CWE-770). These are normally mitigated by the software capability layer on CHERIoT. For example, a compartment can’t allocate memory unless it has a capability that authorises it to do so. That capability encapsulates a quota and so provides a limit to the total amount of allocation. Other resources that can be dynamically allocated are normally managed in the same way.

So, what’s the final score card?

Not applicable in embedded contexts: 1, 2, 3, 9, 10, 12, 22, and 23

Deterministically mitigated with just a recompile: 5, 7, 8, 11, 14, and 16.

Mitigated by CHERIoT design patterns and software model: 4, 6, 13, 15, 25.

That still leaves six that we don’t mitigate (17, 18, 19, 20, 21, and 24), but now hopefully the cognitive load is much lower from not having to think about the eleven that we do prevent and you can avoid some of these as well!

What are ML-KEM and ML-DSA for?

I am not a mathematician and so I’m not going to try to explain how these algorithms work, but I am going to explain what they’re for.

Module-Lattice-Based Key-Encapsulation Mechanism (ML-KEM) is, as the name suggests, an algorithm for key encapsulation. One side holds a public key and uses it (plus some entropy source) to generate a secret in both plain and encapsulated forms. The encapsulated secret can be sent to a remote party who holds the corresponding private key. The receiver can then recover unencrypted version of the secret (and detect tampering). Now, both parties have the same secret and can use it with some key-derivation function to produce something like an AES key for future communication.

Note that this is somewhat more restrictive than traditional key-exchange protocols. You don’t get to exchange an arbitrary value, the generation step is part of encapsulation. This also means that it’s a fixed size, defined by the algorithm, which is why you typically feed it into a key-derivation function rather than using it directly.

Module-Lattice Digital Signature Algorithm (ML-DSA) has a similarly informative name. It is intended for providing and validating digital signatures. It takes a private key, an arbitrary-sized document and context, and produces a signature. A holder of the associated public key can then validate that the document matches the version signed with the private key and context.

These are both quite low-level building blocks for higher-level protocols. For example, TLS can use ML-KEM for key exchange and ML-DSA for certificate validation, but also incorporates traditional algorithms in case the PQC algorithms have unexpected weaknesses against classical computers.

Initial porting

As is usually the case for CHERIoT, porting the C implementations of ML-KEM and ML-DSA required no code changes. I worked with upstream to slightly simplify the platform-integration layer, so we just provide a single header describing the port. For example, the port header for ML-DSA configures the build to produce ML-DSA44 support, defines a custom function for zeroing memory and getting entropy, and adds the __cheriot_libcall attribute to the all exported APIs (so we can build them as shared libraries, rather than embedded in a single compartment). The file for ML-KEM is almost identical.

With these defined, it is possible to build both libraries as CHERIoT shared libraries. This motivated a bit of cleanup. We have a device interface for entropy sources, but it wasn’t implemented on the Sail model (which doesn’t have an entropy source). It has a way of exposing the fact that entropy is insecure, so that wasn’t a problem, it just needed doing, so I refactored all of the insecure entropy-source drivers to use a common base. Most encryption algorithms want an API that fills a buffer with entropy. It’s nice if these don’t all need to touch the driver directly, so I created a compartment that provides this API and exposes it. Now, both libraries are simply consumers of this API. This also makes it easier to add stateful whitening for entropy drivers for hardware entropy sources that don’t do the whitening in hardware.

Most CHERIoT stacks are on the order of 1-2 KiBs. The PQC algorithms use much more space. More, in fact, than we permitted.

The previous limitation was based on the precision of bounds rounding. A CHERI capability compresses the bounds representation by taking advantage of the fact that, for a pointer to an allocation, there is a lot of redundancy between the address of the pointer, the address of the end of the allocation (the top), and the address of the start of the allocation (the base). The distance from the address to base and top are stored as floating-point values with a shared exponent. In practical terms, this means that the larger an allocation is, the more strongly aligned its start and end addresses must be. The same restrictions apply for any capability that grants access to less than an entire object.

When you call a function in another compartment, the switcher will truncate the stack capability so that the callee sees only the bit of the stack that you weren’t using. The top and base of the stack must be 16-byte aligned (as an ABI requirement), but a very large stack may have hardware requirements for greater alignment and so may require a gap between the bottom of the caller’s stack and the top of the callee’s.

Fortunately, we’d added an instruction precisely for this kind of use case: CSetBoundsRoundDown. This takes a capability and a length and truncates it to at most that length. It was a fairly small tweak to the switcher to make it do this, and a much larger amount of time with SMT solvers to convince ourselves that this was a safe thing to do.

This also showed up a bug in our linker’s handling of the CAPALIGN directive, which rounds a section’s base and size up to the required alignment to be representable. This was not working for sections that followed an explicit alignment directive. Our stacks must be both at least 16-byte aligned and representable as capabilities. This is now fixed.

So now we support stacks up to almost 64 KiB, a limitation imposed by the current loader metadata format rather than anything intrinsic to how the system operates after booting. We could easily increase this limit but 64 KiB ought to be enough for anyone.

Performance on CHERIoT Ibex

The repository contains a simple benchmark example that tries each of the operations and reports both the cycle time and stack usage. The output on the CHERIoT Ibex verilator simulation is:

PQC benchmark: Starting: stack used: 224 bytes, cycles elapsed: 41
PQC benchmark: Generated ML-KEM key pair: stack used: 14304 bytes, cycles elapsed: 5143987
PQC benchmark: Encrypted secret pair with ML-KEM: stack used: 17440 bytes, cycles elapsed: 1773235
PQC benchmark: Decrypted secret pair with ML-KEM: stack used: 18464 bytes, cycles elapsed: 2176226
PQC benchmark: Compared results successfully for ML-KEM: stack used: 224 bytes, cycles elapsed: 414
PQC benchmark: Generated ML-DSA key pair: stack used: 46912 bytes, cycles elapsed: 3622132
PQC benchmark: Signed message with ML-DSA: stack used: 60544 bytes, cycles elapsed: 5391177
PQC benchmark: Verified message signature with ML-DSA: stack used: 44672 bytes, cycles elapsed: 3674071
PQC benchmark: Correctly failed to verify message signature with ML-DSA after tampering: stack used: 44672 bytes, cycles elapsed: 3673706

The ML-KEM encrypt (generate shared secret and encrypted version) and decrypt (recover shared secret from encrypted version) each use around 18 KiB of stack and run in around two million cycles. CHERIoT Ibex should scale up to 200-300 MHz (though may be clocked lower for power reasons in some deployments), but even at 100 MHz that’s 50 encryption or decryption operations per second. Remember that this is an operation that typically happens when you establish a connection, then you use a stream cypher such as AES with the exchanged key.

The ML-DSA operations are slower and use a lot more stack space (almost 60 KiB for signing!). But, even there, the performance is reasonable, under 4 M cycles. This means that you can do 20 signature-verification operations per second at 100 MHz.

Even using ML-KEM for key exchange and ML-DSA for certificate validation in a TLS flow is unlikely to add more than a few tens of milliseconds to the handshake time, which is perfectly acceptable for the common use case for embedded devices.

In terms of code size, both are small. The ML-KEM implementation is around 12 KiB, the ML-DSA implementation 18 KiB. These both include a SHA3 (FIPS 202) implementation, so there’s scope for code-size reduction on systems that need both, but 30 KiB of code isn’t too bad.

Future plans

The stack usage is very high. Upstream has some plans to allow pluggable allocators, which will allow us to move a lot of this to the heap. This is precisely the kind of use case that CHERIoT’s memory-safe heap is great for: something needs 60 KiB of RAM for 4,000,000 cycles, but then doesn’t need that RAM again for a long time. That memory can then be used for something else, even in a mutually distrusting compartment.

Currently, the library builds are very thin wrappers around the upstream projects. This is great as a building block, but we should make more use of CHERIoT features in the longer term.

Both ML-KEM and ML-DSA depend on SHA3 (FIPS 202). Ideally, we’d factor that out as some common code, rather than carrying a copy in each library. Similarly, the libraries provide an option to plug in your own SHA3 implementation. This is likely to be a common hardware operation even for chips that don’t have full PQC implementations, so we should expose this option in the build system.

Is it secure?

Security always depends on the threat model.

For signature validation, you don’t have any secret data, just a public key, a document, and a signature. The only concerns are whether there are weaknesses in the algorithm, or bugs, that would allow an attacker to substitute a different document for the same signature. CHERIoT prevents memory-safety bugs, so this is concerned solely with logic errors. The code upstream is checked against a set of test vectors that aim to trigger corner cases in the logic of the underlying implementation, so hopefully is secure in this way.

For signing or key exchange, you need to worry about the key leaking. On a CHERI system, it’s unlikely to leak explicitly, but may leak via side channels. The security section of the upstream projects discusses a number of techniques that they use to mitigate this kind of attack.

That’s typically sufficient. It’s been recommended practice for embedded devices to have per-device secrets for a long time. This means that leaking a key from one device doesn’t compromise the device class, only that specific device.

For some very high-assurance use cases, that secret may matter and need to be robust against an adversary with physical access to the device. Hardware encryption engines typically care about confidentiality breaches via power side channels and integrity breaches via glitch injection. Power side channels are difficult to mitigate in software: the power requirements of multiplying two numbers together may depend on the number of carry bits set, for example. They’re much easier to mitigate in hardware, by simply doing the same calculation twice in parallel, once with the original inputs and once with the inputs permuted to have the opposite power characteristics.

Glitch injection takes the chip out of its specified power or frequency (or thermal) envelope and attempts to introduce bit flips, which can corrupt state in such a way that tamper with signing or leak a key. These are also effectively impossible to mitigate in software because the software that’s attempting the mitigation is vulnerable to the same glitches. There are some compiler techniques that can make these harder, but they come with a high performance cost.

If power analysis and glitch injection are part of your threat model, the software implementations are not sufficient. In this case you may also need to worry about someone removing the top of the chip and using a scanning-tunnelling electron microscope to read bits from non-volatile memory. This used to require tens of thousands of dollars but is now much cheaper. Devices that need to worry about this often have tiny explosive charges in the package to destroy the chip in cases of tampering. If that’s your threat model, hardware PQC implementations may not be sufficient, at least alone.

But if you care about attackers on the network being unable to compromise the security of the class of devices, even if they have a magical and imaginary quantum computer, then these should be sufficient.

TockOS is implemented in Rust and has safety as a priority. I generally regard it as the gold standard for an RTOS that is forced to operate under the constraints of working with existing hardware. But note where the drivers are: In the kernel, right at the bottom.

Here’s a similar diagram from the Zephyr RTOS:

Here, there is some separation between drivers and the core of the kernel, but the default configuration runs with no privilege separation. Indeed, the Zephyr Security Overview says:

The security architecture is based on a monolithic design where the Zephyr kernel and all applications are compiled into a single static binary. System calls are implemented as function calls without requiring context switches.

In fact, the only exception I’ve seen to this recently is LionsOS, a multiserver system built on top of the formally verified seL4 microkernel:

This runs all of the device drivers in unprivileged contexts. Unfortunately, seL4 assumes an MMU and so is not feasible for small embedded devices (seL4 uses more memory to hold page tables than a lot of CHERIoT firmware images use in total).

Drivers are attack surface

Device drivers, at least for I/O devices, are code that interacts with the outside world. An attacker trying to compromise a device has a much easier job if they don’t need to take the chip apart. The easiest attacks to mount are ones that work from an across the network. The next easiest are ones that drive local I/O, which may be reachable remotely via other paths.

Drivers for I/O devices, by definition, make decisions based on the values that they read from the device. These operations include mapping error codes to some type-safe enumeration (which can fail if the hardware is buggy), or using the values to index into other structures. These are hard to get right, even in a memory-safe language, because they often sit below the language’s abstraction layer. Microsoft estimates that 70% of Windows crashes are caused by bugs in device drivers.

Any bug in a driver for an I/O device is a useful building block for an attacker. This problem is made much worse if the device driver is in a privileged component. For example, at the end of last month there were three bluetooth CVEs in Zephyr that all could lead to compromise and, if the Bluetooth stack is not privilege separated, can lead to arbitrary code execution by an attacker who gets within a few meters of the device (or compromises another Bluetooth-enabled device nearby).

Device drivers do abstraction and multiplexing

This design results from conflating the two functions of a device driver.

A device driver has to provide an abstraction over a particular device. Sometimes this happens in multiple layers. For example, an Ethernet device may have an abstraction for sending and receiving Ethernet frames, but this is then the foundation for a further abstraction layer for sending IP packets, which is then used to expose TCP streams and UDP datagrams.

A device driver often also has to provide some secure multiplexing. For example, two mutually distrusting components may be allowed to create sockets for different TCP connections that flow over the same Ethernet device. Or they may be allowed to talk to two different USB bus endpoints via the same USB controller.

The first of these requirements is a software engineering problem. The second is primarily a security problem, but typically the multiplexed abstractions need to be device independent and so it’s also a software-engineering problem.

In embedded development, there’s often a distinction between a hardware-abstraction layer (HAL) and a driver, with the former providing only the abstraction and the latter also providing multiplexing. This is a useful distinction because, in a lot of cases, embedded systems have a single consumer for a device. For example, you may have multiple SPI or I²C interfaces on a device, but each one is used for a single purpose. It is convenient to be able to write software to talk to a SPI device without having to know exactly which SPI controller this chip uses, but you don’t need to handle safely sharing those pins with other components.

CHERIoT RTOS distrusts drivers

In CHERIoT RTOS, the core platform provides device abstractions that meet the earlier definition of a HAL: they provide abstractions over classes of device, but do not attempt to provide security. The RTOS also provides a trivial way of auditing which compartments can access which devices, so that you can ensure that devices are not accessible to compartments that are not trusted to interface with them.

The platform’s device code runs within whatever compartment you instantiate it in. It has no elevated privileges except the MMIO region(s) that you explicitly pass it for talking to a particular device.

This makes it easy to support both bespoke and reusable security models. If you need to share a device between two compartments with some secure multiplexing based on a custom policy, you can do that by instantiating the driver in a compartment and exposing APIs to the two others.

Sometimes, the desired abstractions are reusable. For example, the CHERIoT network stack is assembled out of the following compartments:

graph TD
  Network
  subgraph Firewall["On-device firewall"]
    DeviceDriver["Device Driver"]
  end
  TCPIP["TCP/IP"]:::ThirdParty
  User["User Code "]
  NetAPI["Network API"]
  DNS["DNS Resolver"]
  SNTP:::ThirdParty
  TLS:::ThirdParty
  MQTT:::ThirdParty
  DeviceDriver <-- "Network traffic" --> Network
  TCPIP <-- "Send and receive Ethernet frames" --> Firewall
  DNS <-- "Send and receive Ethernet frames" --> Firewall
  NetAPI -- "Perform DNS lookups" --> DNS
  NetAPI -- "Add and remove rules" --> Firewall
  TLS -- "Request network connections" --> NetAPI
  TLS -- "Send and receive" --> TCPIP
  NetAPI -- "Create connections and perform DNS requests" --> TCPIP
  MQTT -- "Create TLS connections and exchange data" --> TLS
  User -- "Create connections to MQTT server and publish / subscribe" --> MQTT
  MQTT -- "Callbacks for acknowledgements and subscription notifications" --> User
  SNTP -- "Create UDP socket, authorise endpoints" --> NetAPI
  SNTP -- "Send and receive SNTP (UDP) packets" --> TCPIP
  TLS -- "Request wall-clock time for certificate checks" --> SNTP
  style User fill: #5b5
  classDef ThirdParty fill: #e44

Note that the driver for the Ethernet device is instantiated in the firewall compartment. What happens if an attacker gets arbitrary-code execution here? They could mount a denial of service attack (refuse to forward Ethernet frames in or out). They could tamper with Ethernet frames.

This sounds bad but the rest of a network stack already has to assume that things like this can happen. Packets coming over the network are intrinsically untrusted. The TCP/IP stack has to assume that they may be malicious. It isn’t always good at this. The FreeRTOS TCP/IP stack that we use has had 15 CVEs disclosed since it was released, but our compartmentalisation strategy mitigates all of them. By placing the parts of the system that are exposed to an attacker in the least, not most, trusted places, we make it easy to build secure systems.

The funding was awarded back in September, but the embargo was lifted last week and so we can talk about it publicly.

Rust and CHERIoT have complementary benefits

I’ve written previously about why safe languages and CHERI are complementary. Superficially, both Rust and CHERI provide similar benefits in terms of memory safety. That similarity goes away when you look at the details.

Rust provides a very rich set of type-system guarantees. If you are writing software, you can use Rust’s type system to enforce a range of properties that go beyond simple memory safety. The key part here is ‘if you are writing software’.

The industry learned some important lessons from Java and JavaScript attempts at language-level sandboxing: it’s very hard to write a compiler that assumes that the programmer is an adversary. Any soundness issue in the underlying type system or any bug in the compiler can be a security vulnerability if you assume that the person writing the software is malicious and actively trying to break the guarantees that the language aims to enforce.

The Rust compiler is currently tracking 107 bugs marked as soundness issues. A typical Rust programmer is unlikely to encounter these. Encountering these bugs typically require poking at corner cases of the language that you’re unlikely to hit by accident. In contrast, a malicious programmer wanting to insert a supply-chain vulnerability into something that you consume has a rich set of tools.

The CHERIoT compartmentalisation was designed with this kind of adversary in mind. It assumes that you may be incorporating arbitrarily buggy or malicious code into a device’s firmware and need to be able to protect against components that are compromised. The checks that CHERIoT does at run time are less rich than those that Rust enforces at compile time, but are not bypassable, even with unsafe code or inline assembly.

This means that you can use Rust’s type system to ensure that your code has strong confidentiality, integrity, and availability guarantees, while simultaneously using CHERIoT to ensure that supply-chain code cannot violate these guarantees (at least with respect to confidentiality and integrity, though availability is a bit harder). Rust’s guarantees make achieving high levels of confidence in your code much easier than if you used C/C++. Rust is one of the few languages that delivers this kind of guarantee and is also able to run on small embedded devices (such as CHERIoT implementations), making it a very exciting choice for future CHERIoT development.

CHERI and Rust give you the benefits of Rust faster

Rust gives rich guarantees to Rust code. Most software is not written in a single language, and especially not in a relatively young language. What if you have some legacy C/C++ component? You can use it from Rust, but in most systems this means bugs in the legacy code can completely undermine the security guarantees of the Rust code. A memory-safety bug in C code can corrupt any Rust object in the same process.

On a CHERI system, you get different levels of guarantee depending on how you mix languages. Calling C code from Rust within the same compartment requires the C code to follow the CHERI rules. If Rust code passes a pointer to C, the C code may tamper with objects reachable from there, but can’t arbitrarily affect the system unless it has a much narrower set of bugs. Stack uninitialised use bugs may cause C to access objects left on the stack, for example.

If you put the C code in another compartment, these guarantees become a lot stronger. C code can tamper with objects passed as arguments (or objects reachable from them) but has no access to the Rust compartment’s stack or globals. This makes it much easier to reason about the impact of bugs in C code when adopting Rust.

You can take this even further and restrict the permissions on pointers passed from Rust to C. Do you want C code to be able to read a Rust object or object graph? Or perhaps modify an object but not capture a pointer to it (similar to borrow semantics)? The hardware can enforce these properties.

This lets you get all of the great benefits of Rust from the very first function that you write in Rust, rather than having to wait until you’ve rewritten everything in Rust.

What are we trying to do?

Initially, we aim to make Rust work as a source language for targeting CHERIoT. CHERIoT is an embedded platform, so the no_std + alloc mode for Rust (you can dynamically allocate memory, but can’t use most of the standard library) makes sense as a target. This will make it easy to port existing embedded Rust code to CHERIoT and to write new Rust components. As the project runs, there will be a lot of quality-of-implementation work to ensure that it’s in a state to upstream and, until then, in a state where we can support it.

The next step in the project is to make sure that Rust is a first-class citizen of the CHERIoT platform. We have a set of C/C++ language extensions to provide rich compartmentalisation features. Rust will need equivalents of these. A direct port of the features is quite easy and works as a minimum viable product, but we’d also like to make sure that these work as idiomatic Rust: you shouldn’t have to write C-like Rust to use CHERIoT features.

Finally, there are several things in the Rust type system that can be dynamically enforced in CHERIoT. In current Rust, every call to non-Rust code must be unsafe. I hope that we’ll be able to relax this requirement and have the compiler enforce Rust properties by removing permissions from pointers before calling C.

There are a lot of subtleties here. The authors of Tock discovered that exposing Rust to untrusted code has some issues and provided solutions that worked in their specific context. I’m optimistic that the richer substrate of the CHERIoT ISA gives the compiler more tools to provide generic solutions to these issues.

The funded project is specifically scoped to CHERIoT, but we’re building on work from Morello and hope to make it easy to support other CHERI platforms.

Rust enables verification

The Verus project builds language-integrated formal verification tools in Rust. This is particularly interesting for core parts of CHERIoT RTOS. The lowest level parts of an operating system are tricky for safer systems languages because the things that they do are intrinsically unsafe. They must be able to use escape hatches that opt out of parts of the safety guarantees of a language like Rust because they are the things that implement those guarantees.

You can think of rich type systems as off-the-shelf verification tools (they define some generic properties and prove them for every program, and reject those for which they can’t prove them), whereas the lowest-level parts of systems code need bespoke verification to prove one-off sets of properties. There was some great work at SOSP this year showing how Verus can prove key aspects of a kernel and I’m excited to see how far we can get with this in CHERIoT.

Where will this project live?

The CHERIoT Rust project has its own web site which is where we’ll post more public information. The compiler repository is public, but is not yet ready for general use. We’ll post calls for testing when it reaches an early preview state. As with any other aspect of CHERIoT, contributions are welcome. Please join our public Signal chat if you want to discuss the project or help out!

CHERI is an abstract architecture

CHERI is a conceptual extension to conventional computing architectures define a capability model for accessing memory within an address space. There are a lot of concrete instantiations of this conceptual model. The original CHERI research was done on a 64-bit MIPS variant. Arm Morello extended ARMv8 with CHERI extensions. The University of Cambridge CHERI ISA v9 Technical Report contains a RISC-V CHERI adaptations and even a sketch of CHERI for x86. And there is an ongoing effort to standardise RISC-V CHERI base architectures.

These all provide the same core set of features. Most software isn’t written for a single target instruction-set architecture (ISA), it’s written assuming a set of language-level abstractions that can be represented in a particular target. Software written for CHERI in C (or any language that is higher-level than C) doesn’t care the bit patterns of capabilities, how bounds or permissions are encoded, and so on. It cares only that you have some capability registers, that you can store capabilities in memory, and that the C language notion of a pointer can be represented by a CHERI capability for the target ISA.

I like to use memory-management units (MMUs) as an analogy for CHERI. MIPS and SPARC had software-managed translation-lookaside buffers (TLBs). x86 and Arm had architectural page tables that hardware walked to fill the TLB automatically on miss. PowerPC had an MMU design that, in the words of Linus Torvalds, ‘can be used to frighten small children’. All of these implemented the same set of abstractions in terms of processes, shared memory, copy-on-write regions, and so on. All of these are MMU designs, just as there are many possible CHERI implementations.

There are lots of things like this in ISAs. Arm’s Neon, PowerPC’s AltiVec, and x86 SSE are all SIMD extensions, which provide similar programming models, but they all have quite different low-level details. It’s quite possible to write code that targets all of them.

CHERIoT is a concrete platform

CHERIoT is a complete ISA specification, co-designed with a software stack. In the same way that x86 page tables or MIPS’ software-managed TLB are both concrete instances of the abstract idea of a memory-management unit, CHERIoT is a concrete instantiation of CHERI.

This means that there is no ‘CHERI or CHERIoT’ choice. The real choices are between CHERI or not CHERI, or between CHERIoT and some other concrete instantiation of CHERI. Having worked on CHERI since 2012, I am obviously biased on the first of these: given a choice between CHERI and not-CHERI, you should definitely pick CHERI! The second choice (CHERIoT or some other CHERI variant) is more nuanced, as I’ll discuss more in the rest of this post. There are good reasons for CHERIoT but there are also domains where it is not the best choice.

CHERIoT is a rich CHERI platform

A minimal CHERI specification provides pointer integrity, bounds enforcement, and some permissions. Pointer integrity means (among other things) that you can precisely differentiate between pointers and other kinds of non-pointer data in memory and so can build temporal safety on top. Most of the CHERI temporal safety work has provided users with protection against use-after-reallocation issues. A pointer in a CHERI system may point to deallocated memory, but it will not point to reallocated memory. Something in the system will ensure that deallocated pointers are gone before the memory allocator reuses memory. This means that any use-after-free bug will either point to the old object, or will trap. It will never point to a new object and cause accidental aliasing.

CHERIoT builds on top of the core CHERI ideas with a couple of hardware features that guarantee deterministic trapping for use-after-free bugs. CHERIoT also provides a rich sealing design that works with our software model. It extends the sentry mechanism in CHERI systems to provide stronger control-flow integrity properties than most other CHERI platforms, and also uses them to enforce interrupt control.

The entire system is designed to allow fine-grained auditing of compartment rights, which lets us respect the principle of least privilege by having compartments as small as a single function and statically reasoning about what they can do in the presence of a compromise. This set of building blocks lets us do things like privilege separate the network stack and make callers respect the principle of intentional use, again with auditable guarantees that let you reason about which compartments can talk to which remote servers.

This adds up to an easy-to-use programmer model and richer security guarantees than, we believe, any other system (CHERI or otherwise).

CHERIoT is designed for resource-constrained systems

Some of the strengths of CHERIoT come from the fact that we were willing to tailor the system to small devices. Prior CHERI systems rely on the MMU to support concurrent revocation as their mechanism for temporal memory safety. CHERIoT does not include or need an MMU, it uses an alternative approach for temporal safety, inspired by our previous CHERI+MTE work

You don’t want an MMU on an embedded device. An MMU is typically larger than an entire microcontroller core! Page tables are large data structures and they enforce a minimum granule size for memory protection and accounting. A minimal process isolated with a typical MMU needs one page for the stack, one page of code, and one for read-write globals. On top of that, it needs a complete page table, which is at least two pages. This means you need a minimum of 20 KiB for an isolated component, of which 8 KiB is purely book-keeping overhead. On a CHERIoT system, a lot of our compartments are smaller than the page tables that would be necessary to isolate them. In addition, MMUs bring nondeterminism, which is not ideal in any system with even fairly soft realtime requirements.

The temporal memory system in CHERIoT is designed to scale across the range of microcontroller cores, including low-core-count multicore systems. We check whether capabilities are valid when you load them. Most pointers are used more than once, so this is more efficient than checking on every use (as an MTE system must do), but does not scale as well to large multicore systems or complex memory hierarchies. You would not want to build a temporal-safety system like ours for a large server system, for example.

CHERIoT also extends CHERI’s sealed function pointer (“sentry”) mechanism for interrupt control. This has some enormous benefits for the programmer model. It is quite easy to implement on microcontroller-class systems, even dual-issue designs such as CHERIoT-Kudu. It would be very hard to scale to big out-of-order cores.

And that’s all fine, for the same reason it’s fine to run different operating systems on different scales of core: different abstractions and different optimisations make sense at different scales. By the time you’ve built a system where you start to run into scalability challenges with CHERIoT, you have enough area that an MMU doesn’t add much overhead and enough RAM that you would benefit from running an OS designed for larger computers.

What about standard RISC-V CHERI?

The RISC-V standardisation process has an unenviable task of trying to create a standard base for RISC-V CHERI systems that will work for 2-stage microcontrollers, 25-stage superscalar out-of-order server chips, GPUs, AI accelerators, SmartNICs, and so on. This is likely to end up being a (hopefully small) family of bases and a set of extensions.

If all goes well, CHERIoT v2 will be one of those bases plus some extensions. RVB26 will be a RISC-V profile assembled from those bases and a set of extensions necessary to run a CHERI Linux or CheriBSD.

As with the rest of RISC-V, any useful chip implements a base architecture and a set of extensions. Profiles exist to corral a set of extensions that software can depend on in environments where binary compatibility is important, such as those running off-the-shelf operating systems and programs.

So when should I use CHERIoT vs some other CHERI?

If you are looking for a microcontroller-class system, CHERIoT is probably the right answer. It is co-designed with a compartmentalisation model and a rich set of software abstractions (implemented in CHERIoT RTOS), and provides temporal safety as a baseline feature. It is a mature and stable target, supported by multiple organisations.

If you are looking for an application core, CHERIoT is not for you. There are a few cases where people have moved to using Linux on an Arm A-profile system and could run the same workload on a cheaper CHERIoT system, but don’t expect to be able to run arbitrary Linux workloads on a CHERIoT system (ever). The upcoming CHERI RISC-V profile is a far better choice for these use cases. Capabilities Limited and lowRISC are implementing this in the open-source CVA6 core, so there will soon be a path to a useful open-source CHERI core in this part of the design space. Codasip’s X730 core is also available for commercial licenses and is targeting this spec (and pre-standard versions of it until it is ratified). Hopefully there will be more implementations available in coming years.

If you are looking for an ISA as a base for an accelerator or coprocessor, CHERIoT may be the right choice, depending on the rest of the system. Depending on your requirements, you may want to do something completely custom. As long as it provides the same underlying security guarantees, it can still be CHERI!

A quick intro to sealing

CHERI capabilities are most commonly used as pointers (they can also be used for very coarse-grained sandboxing, with a model similar to WebAssembly). Each capability is an address plus some metadata, protected by guarded manipulation. The CPU enforces properties on how capabilities can be manipulated (for example, their bounds can be shrunk, but not expanded). Importantly, it also checks the metadata before the capability can be used for any operation. For example, if you use a capability as the base for a load instruction then the core will check that the entire width of the load is within the bounds and that the capability has load permission.

The metadata for a CHERI capability has an object type (otype) field. If you’re just using capabilities to represent pointers in unmodified C/C++ code, then you will only ever see capabilities with zero in their otype field. This represents an unsealed capability.

The CSeal instruction (which has different spellings in some CHERI variants) combines two capabilities. One is a ‘normal’ pointer-like capability. The other, the sealing key, has the permit-seal permission and its address (sometimes referred to as ‘value’ or ‘cursor’) does not represent a memory location, but instead represents a value in the space of types. The result of sealing is a copy of the pointer-like input with its otype field set to the address of the sealing key, making it a sealed capability.

You can pass this around just like any other pointer, but you can’t use or modify it. If you try to use it as the base for loads or stores, it will trap. If you try to modify it (the address or the metadata), you will get an untagged (invalid) capability.

If you use that capability as one operand to CUnseal and provide an unsealing key (a capability with permit-unseal permission) that has the address from the sealed capability’s otype, you get back the original pointer-like capability. This lets you pass untrusted code a pointer to something that they can pass back but can’t tamper with.

Sealing gives type safety

The object type in a CHERI capability doesn’t necessarily have to have a 1:1 mapping to a language-level type. It’s quite common for a set of types to have some kind of internal type discriminator. For example, in our ASPLOS 2017 CHERI JNI paper, we used three otypes for everything passed from the JVM to native code. One each for field-ID and method-ID structures, and one for all Java objects. Each Java object starts with a pointer to its class, so we didn’t need one otype for each Java class type, just one for all Java objects. The virtualised sealing mechanism in CHERIoT uses this same approach to multiplex a huge number of possible object types onto two hardware otypes (one for statically allocated objects and one for dynamic, so the memory allocator is not in the TCB for statically allocated sealed objects). More on this later.

If you couple a sealing key type with a language-level type (or family of self-disambiguating types), then you can build type safety that works in the presence of an attacker. It’s common in C/C++ to expose an opaque type at API boundaries. This is usually implemented as a pointer to forward-declared structure type. C and C++ will not let you dereference this pointer unless you cast it to another type. CHERI sealing lets you enforce the no-dereference rule, even if malicious code does cast it to some other type. This means that you can hand sealed capabilities to other compartments and they must treat them as opaque types. When you get them back, you have a lightweight check that they really are the type that you expect.

After the 2015 paper, we separated the permit-seal and permit-unseal permissions. In most CHERIoT use cases, the entity that can seal and unseal pointers with a particular otype is the same. This isn’t universal. C++ provided one of our original use cases for separating them. If you seal C++ vtables with a well-known otype, and make the permit-unseal capability for it available anywhere, then you can have a C++ ABI where only the loader can forge vtables. This makes code reuse attacks harder.

There are other situations where an object type is used for integrity but not confidentiality. It is an attestation that some software trusted to seal with a particular otype has done so, even if anyone is then able to unseal the result. The sentry mechanism (discussed later) is a variant of this idea.

Virtualised sealing shares hardware object types

Morello had a 15-bit object-type field, which is sufficient for a lot of things. When we scaled CHERI down to 32-bit systems for CHERIoT, we ended up with only three bits of space. Of these, the zero value means not-sealed, so there were only seven bit patterns available. Even simple embedded software typically has more than seven types. CHERIoT systems usually have more than seven compartments, so seven types isn’t even enough for one type per compartment (and some compartments wish to offer more than one sealed type). Because they are (mostly) used for different purposes, we differentiated executable and data capabilities, so 3 bits actually gives 14 sealing types, seven for data and seven for sentries (see later).

The virtualised sealing mechanism was designed to work around this shortage of sealing types. We reserve two of the object types for objects that are instances of a structure with the following layout:

struct SealedObject
{
    /// The sealing type for this object.
    uint32_t type;
    /// Padding for alignment
    uint32_t padding;
    /// The real data for this.
    char data[];
};

The first 32-bit word is the type, followed by 32 bits of padding, and then the real object.

Our virtualised mechanism takes advantage of the fact that we can represent a lot more values in the address field (32 bits) of a capability than in the otype field (three bits). Any address in a permit-seal or permit-unseal capability that cannot fit into the otype field is available to authorise sealing or unsealing with the virtualised mechanism. This gives us almost four billion types that we can represent with the virtualised mechanism, at the expense of being able to seal only complete objects and of each object having a single type. This is fine for most use cases, though language VMs on bigger systems would benefit from at least a few hardware otypes. We also reserve two data otypes for future use, so have some flexibility going forward.

The CHERIoT token_unseal function unseals the object using the hardware sealing key and then compares the virtual otype in the header to the type of the permit-unseal capability passed as the key. If they match, it returns an unsealed capability to the object without the header, so the caller doesn’t ever have access to the header.

Sealed pointers using the software sealing mechanism always point to the end of the header. The header must be strongly aligned, which means that the low three bits are unused. We use these to implement three software permissions. As with the CHERI permissions, these can be cleared but not set.

Type safety gives easy-to-use handles

A lot of CHERIoT compartments use type-safe pointers as handles. For example, if you open a socket, you get back a sealed capability to that socket’s state. The same thing happens for message queues, TLS sessions, and so on.

In a conventional monolithic kernel, these would all be handles or file descriptors looked up in some table in the kernel. In a microkernel, they’d either be indexed per caller, or managed via some handle-manager service. In CHERIoT, these are simply opaque pointers. If these are allocated from the heap, they also benefit from the temporal safety mechanisms: Freeing the object that a handle refers to invalidates all handles, without requiring any further synchronisation.

Exactly the same abstractions that you use for data hiding in good API design work between trust domains, with a little bit of hardware help. This is great for programmers because you can easily retrofit a security boundary. If you have built an API around exposing opaque types, turning it into a robust security boundary simply means doing a few checks on the public APIs.

Type safety gives static software-defined capabilities

The same sealing mechanism that provides type safety for dynamic allocations also works for static objects. You can create a CHERIoT compartment that has access to a sealed object where another compartment owns the sealing key. The contents of these show up in the audit log.

We use these throughout the system to implement software capabilities. CHERI (hardware) capabilities are unforgeable (delegable) tokens of authority to perform some architectural action; they authorize operations carried out on your behalf by the hardware. CHERIoT software capabilities are unforgeable (delegable) tokens of authority to perform some software action; they authorise operations carried out on your behalf by some other compartment.

The first use of these most programmers will make is to allocate memory. If you call malloc, this is a thin compatibility wrapper around heap_allocate, which requires an allocator (software) capability as an argument. The allocator capability is a sealed object that contains a quota. It authorises you to allocate memory until your quota is exhausted. It also authorises you to free objects that were allocated from your quota and reclaim the quota. Similarly, when you create a connected TCP socket, you pass a capability that authorises you to connect to a specific host and port.

These static capabilities can be inspected with CHERIoT Audit so you can write policies that say things like ‘these four compartments, between them, can’t allocate more than 16 KiB of RAM’ or ‘this compartment may connect to my cloud back end, but nowhere else’.

Sealed handles give trivial flow isolation

Sealed capabilities are passed out and back as opaque values and can be unsealed without any global state beyond a constant (un)sealing key. A lot of compartments that use them have no global mutable state. If you pass a sealed capability to a TLS session into the TLS compartment, that compartment can unseal it and then see the state of your TLS session.

The thread operating on your TLS session, while within the TLS compartment, does not gain access to capabilities to other TLS sessions. If an attacker gains arbitrary code execution in the TLS compartment while operating on one TLS flow, that doesn’t give them the ability to attack another one.

The same applies to much simpler compartments. The message-queue compartment, which provides secure message queues and streams between compartments, has the same property. The only state that it operates over is reached via the sealed queue pointer or one of the arguments. If you dynamically compromise this compartment, you can tamper with a queue that you hold an endpoint handle for, but not any other queue.

Sentries build on sealing

CHERI also has a notion of a sealed entry (sentry) capability. These are sealed capabilities with a special otype that allows them to be used as jump targets. When you jump to a sentry, it is implicitly unsealed and installed as the program counter. PC-relative loads then let the jumped-to code retrieve capabilities inaccessible to the code that held the sentry.

In CHERIoT RTOS, all library functions, including the switcher (which handles cross-compartment calls) are provided as sentries.

Morello had several variants of sentries, including some designed for descriptors where the sentry was actually a pointer to a pair of a code and data capability. The jump would load the data capability and branch to the code capability. This underlying mechanism is very flexible and there is a lot more research to be done on how it can be used in the future on big systems.

CHERIoT provides rich sentries

CHERIoT uses sentries to control interrupts. Rather than the usual paradigm of software explicitly managing the interrupt enable status bit, we have sentry variants that explicitly enable or disable interrupts when you jump to them. This encourages a more structured style of code for interrupt management. Moreover, the interrupt status of a function is visible in the audit report, so it is possible to see which compartments are able to call which interrupt-disabling functions. This is one of the ways in which CHERIoT is very much designed for embedded systems. Implementing this feature is moderately easy on in-order pipelines, but would be much harder on out-of-order machines. This is also one of the strengths of RISC-V: we can add extensions that are desirable in part of the hardware design space, but not everywhere.

CHERIoT also differentiates forward and backwards-edge sentries. Function pointers and return addresses use different otypes. This means that you cannot trivially replace a return address on the stack with a function pointer for control-flow hijacking. This kind of attack is hard on a CHERI system anyway, but this provides some defence in depth. Credit to folks at the Microsoft Security Response Center for recommending this and to Murali Vijayaraghavan at Google for coming up with a way of adding it without invasive changes to the ISA.

When you jump to a forward-edge sentry with a jump-and-link instruction, the link register is a return sentry that captures the previous interrupt state (enabled or disabled). This means that, at least for leaf functions, we can enforce structured programming for control over interrupt state.

The forward-edge sentries are an attestation from the RTOS to the running software that this is a valid function pointer. The backwards-edge sentries are an attestation from the hardware that this is the result of executing a jump-and-link instruction.

CHERIoT uses sealing inside the RTOS

In addition to the otypes exposed directly to programmers, the RTOS reserves two for internal use. When you do a cross-compartment call, the compiler will insert a call to the switcher (via the switcher sentry). The function pointer for such a call is a sealed data capability to the target compartment’s export table. The base of this will point to the program counter and global pointer register values for the target. The address will point to the metadata describing this entry point. The switcher unseals this and so knows that this really is a cross-compartment entry point provided by the loader.

When the switcher takes an interrupt, it will spill the register file and pass the scheduler a sealed capability to the register-save area. The scheduler then returns a sealed capability of the same type and the switcher restores the register state from there. This ensures that the scheduler never sees the state of interrupted threads, it has only opaque tokens allowing it to choose the next thread to run.

Hopefully this short guided tour has given you some idea of both how powerful a mechanism sealing is, and how pervasive it is in the CHERIoT platform.

This specification defines the ISA, the CHERIoT language extensions, compilation model, relocations, and so on. The last change that we made to the ISA was in December 2024, so we are confident that this is a stable release that we can support in hardware for a long time. This specification was implemented by the 1.0 release of CHERIoT Ibex and by CHERIoT Kudu (which has not yet had an official release). These two implementations demonstrate that the ISA scales from three-stage single-issue pipelines to six-stage dual-issue pipelines, roughly the same range of microarchitectures supported by Arm’s M profile.

We at SCI have the first of our ICENI chips, which use the CHERIoT Ibex core, on the way back from the fab now and will be scaling up to mass production in the new year. I am not allowed to speak for other folks building CHERIoT silicon, but I expect 2026 to be an exciting year for the CHERIoT project!

This is a release that, both through the open-source CHERIoT Platform project and through partner companies that aim to ship CHERIoT products, we will be supporting for years to come.

Alignment with the RISC-V Y base

RISC-V International is currently standardising a CHERI base for RISC-V (tentatively named Y, alongside the existing I and E bases). We aim for this to be a common subset across all CHERI implementations, whether they are microcontroller, application cores, or domain-specific accelerators. The exact composition of this base is still under discussion, but we aim for CHERIoT 2.0 to be source-compatible with (and with equivalent functionality to) CHERIoT 1.0, but built atop this common base.

On the way to 2.0, we have been separating out the parts that are in the common core from those that are CHERIoT-specific. We also have the ct. prefix reserved for the CHERIoT vendor extensions in RISC-V and so will move any instructions that are not direct equivalents of standardised RISC-V instructions into that namespace. For compatibility, our toolchains will support both names, but we expect to transition to the RISC-V official names as the RISC-V standardisation process makes progress.

Expect to see some 1.x releases that make changes in things like assembler mnemonics (but not the underlying ISA).

Future plans beyond 2.0

No ISA specification is ever complete. Our 1.0 release is a solid foundation, not an endpoint.

We already have plans to integrate with the Zfinx and Zdinx extensions, which allow 32-bit and 64-bit floating point values to be held in integer registers on RISC-V. On CHERIoT, a 64-bit floating-point value could live in a single capability register (as, indeed, it is in the CHERIoT soft-float ABI). This avoids increasing compartment and context-switch times (which comes from a larger register file, and are why we used RV32E, not RV32I as our base), while still allowing hardware floating point acceleration.

We have two candidates for alternative bounds encodings, which each have different microarchitectural tradeoffs. They are 100% backwards compatible with the software and so we aim to add both to a future version of the specification. We have designed our ABI carefully to avoid leaking capability encoding details into the software model, which means that we can support implementers picking any of the three encodings, on the products where each makes sense, along with any vendor-specific variants that they may wish to support. Supporting different encodings does not require changes even to core RTOS code and the two proposed modifications do not require toolchain changes either, though vendor extensions may.

We’ve been exploring for a while the idea of having two integer subregisters for each capability. This would give us 30 integer registers or 15 capability registers, which would slightly reduce stack spills. This involves some complexity in the microarchitecture and in the compiler, so it isn’t an automatic win and there’s a lot more work to decide whether it’s desirable.

The current drafts for the RISC-V official specification provide a thread ID register. We’ve been considering how we could replace our use of the global-pointer register with this, to make it a compartment identifier. The specification also reserves loads and stores with the zero base register (which will always trap on a CHERI system) and so we could use these for GP-relative loads and stores.

Similarly, the current draft reserves space for a branch-if-tag-[not-]-set instruction, which would be quite commonly used in our software model. The RISC-V code-size extensions have push and pop / popret instructions, which are likely to want some adaptation to be most useful with CHERIoT.

And there are also a lot of interesting ideas in research. For example, BLACKOUT provided a way of building a data-centric constant-time programming model on top of CHERI. There’s a bit of work to see how to integrate that with CHERIoT but being able to have a good programmer model for constant-time programming would be worth it for a later release.

There are a lot of possible future directions, and we expect to be adding to the spec for a while. Having a solid core means that we can do this without breaking backwards compatibility. And, as with everything in CHERIoT today, future additions to the ISA will always be motivated by software requirements and careful consideration of the programmer model and the microarchitecture.

Sorry for the wait!

This is a slightly embarrassing post to write, because I thought we’d done a 1.0 release several months ago: The ISA has been sufficiently stable for a 1.0 stamp for a while. Unfortunately, when we initially thought about it, there was a bug in the Sail to LaTeX converter and so we postponed the release until we could generate a nice specification PDF. That bug was fixed quite quickly, but by then I’d forgotten that it had been the release blocker and thought we’d done this release back in the Spring.

Approaching a year without any ISA changes makes me even more confident than I was back in the Spring that this is a release that we can support for the next decade.

This week, some of the CHERIoT team were at The 31st Symposium on Operating Systems (SOSP 2025) presenting the first paper about the CHERIoT RTOS: [1]. This paper describes the CHERIoT RTOS and how it builds on the ISA features to deliver fine-grained compartmentalisation, easy programming, and a tiny trusted computing base (TCB). I also gave a keynote on how CHERI impacts operating system design for the KISV workshop associated with SOSP. There were a lot of good discussions, and I hope to see more folks looking at CHERIoT RTOS.

It was interesting to compare our approach with Tock OS, which remains the gold standard for security on non-CHERI embedded devices. One of the papers in the same session as ours discussed the problems Tock has with untrusted code in userspace violating the Rust invariants. A lot of these are intrinsic to the problem of interfacing a language that provides (and can therefore depend on) a very rich set of compile-time properties with one that does not guarantee any of these. It was particularly nice to see that the CHERIoT ISA allows CHERIoT RTOS to enforce some of these properties (such as non-aliasing arising from a no-capture guarantee) even across trust boundaries. That makes me optimistic that CHERIoT RTOS will be one of the best embedded targets for Rust code (more on this coming soon!).

Full citation

1. Saar Amar, Tony Chen, David Chisnall, Nathaniel Wesley Filardo, Ben Laurie, Hugo Lefeuvre, Kunyan Liu, Simon W. Moore, Robert Norton-Wright, Margo Seltzer, Yucong Tao, Robert N. M. Watson and Hongyan Xia. CHERIoT RTOS: An OS for Fine-Grained Memory-Safe Compartments on Low-Cost Embedded Devices. Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, Association for Computing Machinery (2025), 67–84. [pdf] [doi]
   BibTeX

   @inproceedings{10.1145/3731569.3764844,
     author = {Amar, Saar and Chen, Tony and Chisnall, David and Filardo, Nathaniel Wesley and Laurie, Ben and Lefeuvre, Hugo and Liu, Kunyan and Moore, Simon W. and Norton-Wright, Robert and Seltzer, Margo and Tao, Yucong and Watson, Robert N. M. and Xia, Hongyan},
     title = {CHERIoT RTOS: An OS for Fine-Grained Memory-Safe Compartments on Low-Cost Embedded Devices},
     year = {2025},
     isbn = {9798400718700},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3731569.3764844},
     doi = {10.1145/3731569.3764844},
     booktitle = {Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles},
     pages = {67–84},
     numpages = {18},
     location = {Lotte Hotel World, Seoul, Republic of Korea},
     series = {SOSP '25},
     pdf = {https://dl.acm.org/doi/pdf/10.1145/3731569.3764844}
   }
   

   Abstract

   Embedded systems do not benefit from strong memory protection, because they are designed to minimize cost. At the same time, there is increasing pressure to connect embedded devices to the internet, where their vulnerable nature makes them routinely subject to compromise. This fundamental tension leads to the current status-quo where exploitable devices put individuals and critical infrastructure at risk.We present the design of a dependable embedded OS where compartmentalization and memory safety are first-class citizens. We co-design the OS with an embedded hardware platform that implements CHERI capabilities at a similar cost profile to existing chips with minimal security. We demonstrate key design benefits: fine-grained fault-tolerant compartments, OS-level support for compartment-interface hardening, and auditing facilities to thwart supply-chain attacks, among others, and show that they come at a memory usage and performance cost that allows their widespread deployment in cheap, resource-constrained devices.

Message queues

On an operating system with a monolithic kernel, message queues are typically provided by the kernel. You do a system call and that copies some data from userspace into a kernel buffer, or blocks if the buffer is full. Receiving looks similar, copying out of the kernel buffer or blocking if the buffer is empty.

This is the approach taken on Linux, FreeBSD, and even FreeRTOS and Zephyr. There are good reasons for this approach. When you transition into the kernel, you are in a system that has access to the current userspace process’s memory and so this copy is easy.

There’s an obvious down side to this from the perspective of secure system design: The kernel has read and write access to all memory owned by user processes.

Adapting this to a microkernel design is harder. Microkernels try to provide a small kernel (which still typically has access to all memory), but they move most services out of the microkernel into separate, isolated tasks.

Typically, microkernels expose a message-passing model for IPC. seL4, for example, provides a synchronous message send, where the sender provides some data, the receiver provides a receive buffer, and the kernel can copy between them. If you want to send a large amount of data you will send a (software) capability to a page that you want to grant access to. The kernel will add this capability to the receiver’s capability tables, allowing the receiver to map the memory.

If you want to build a message queue with buffering on seL4, you expose a new task that owns the buffers and allows the kernel to copy in and out of them in response to send and receive events. Or, for large messages, you use seL4’s page-granularity capability model to temporarily allow the message-queue task to have access to some of the sender and receivers’ pages to do single-copy transfers. If you want to isolate different buffers (to protect yourself from bugs in the message queue logic: remember, this code is an addition that is not part of the formally verified core of seL4) then you need a new instance of this task for each message queue.

Embedded systems could do a bit better in some ways because MPUs tend to allow finer granularity than MMUs. An embedded microkernel could grant read access to the source range to the message-queue service as part of an IPC message. The message-queue service could then copy into a buffer that it owned and the kernel could then remove that permission at the end of the call.

There are some problems to do this securely:

• MPUs typically have limited numbers of active regions (16 is common). Adding more is painful for power and area because they’re an associative lookup that’s consulted on every memory access (including instruction fetch).
• The message-queue service needs to be careful that it’s really copying from the newly mapped region. If the region is adjacent to some other region that the service has access to, an overflow could cause problems.
• If you wanted to further privilege-separate the system, delegating access to the caller’s buffer and the message queue (or one element in the queue) adds more complexity.
• The message queue still needs to interact with the scheduler for blocking, which interacts with timeouts in complex ways.

All of this comes back to my refrain for the CHERI project:

Isolation is easy, (safe) sharing is hard.

An MPU or MMU allows you to isolate processes or tasks very easily, but this kind of privilege-separated design relies on being able to easily share memory.

On CHERIoT, this is easy. When you call into the message-queue compartment to send or receive a message queue, it’s a normal cross-compartment call. This takes a capability with read (for send) or write (for receive) permission and does not require global. The sharing is implicit: you pass a pointer as a function argument, the receiver can use the pointer. The hardware and switcher provide the following guarantees:

• If you removed store permission when sending, the message-queue compartment cannot modify your copy of the data. Even if that compartment is malicious, your copy is safe.
• If you removed load permission when sending, the message-queue compartment cannot read stale data from the receive buffer. Even if that compartment is malicious, it cannot violate confidentiality (except for message queue contents, though you can avoid even this by passing sealed capabilities in the queue).
• If you removed global permission (on-stack objects start life without this) then the message queue cannot have captured a copy of the pointer, so once the call returns you know that there’s no additional aliasing.

These are foundational guarantees in CHERIoT but they make implementing something like a message queue that needs to share buffers with callers and callees very easy.

There’s one more aspect to implementing something like a message queue: How do you identify one? On POSIX systems, this is via a file descriptor, an int that refers to an entry in a per-process table. Passing one of these between processes requires passing them over UNIX domain sockets, where they will receive a different (process-local) number at the far end.

On CHERIoT RTOS, if you want to have a message queue that you can safely use between compartments, the type is struct MessageQueue *__sealed_capability. This looks just like a pointer. You can pass it between compartments: You can pass the write end of a message queue to a compartment that you want to be able to send messages, you don’t need complex out-of-band messaging.

The CHERIoT message queue compartment has no mutable globals. This gives flow isolation for free. Each call into the this compartment is isolated from any other thread that calls into the same compartment, if that thread passes a (sealed) pointer to a different message queue. Even if an attacker somehow manages to gain arbitrary-code execution in that compartment, the worst that they can do is corrupt their own message queue. We use the same property in the TLS compartment to ensure that one TLS connection can’t leak another’s state.

Sealing is an extensible abstraction. If you want to add new kinds of file descriptor to a conventional kernel, you need to add a kernel module that provides an implementation of an abstract data type inside the kernel. To do the same with CHERIoT, you just seal with a different sealing key. Sealing gives you type safety for opaque types passed between mutually distrusting compartments. This lets you use a model that was created for clean software engineering (opaque types) as a foundation for building secure systems.

Waiting for interrupts

Embedded systems often run code from interrupt service routines (ISRs). This is a problem for security because ISRs run with elevated privilege (they block interrupt delivery and they may see register contents for interrupted code). They’re also not a great programmer model because code in ISRs can’t yield and so can’t acquire most kinds of lock (if the thread owning the lock was running, it would deadlock).

ThreadX encouraged programmers to think about lightweight threads instead of writing code directly in ISRs. This is the model followed by most big operating systems as well: code in ISRs should wake a thread which actually handles the event.

In CHERIoT RTOS, we’ve adopted this model but taken it further by unifying the mechanism that we use to block waiting for events. CHERIoT RTOS’ scheduler exposes a single primitive for waiting for events: A futex. This is a compare-and-wait operation, where you provide a 32-bit value and a pointer to such a value, and if the location in memory matches the expected value then you block until another thread does a futex_wake on that address. Linux uses futexes extensively because they’re a great building block for other things. CHERIoT RTOS builds ticket locks, flag locks, priority-inheriting flag locks, counting semaphores, message queues, and so on with this primitive. These all have a fast path that just requires an atomic operation and slow paths that call into the scheduler if they need to block or wake waiting threads.

The scheduler’s blocking APIs (wait for timeout, wait for a futex, wait for more than one futex) all take a pointer to a Timeout structure. This records the amount of time that a thread may spend blocking (along with the amount of time it has spent blocking). These are intended to be passed through multiple compartments, so a complex operation that does multiple calls that might block can reuse the timeout pointer from the caller and stop if the timeout is exceeded. This makes it easy to build rich APIs that have bounded blocking times. You wouldn’t build a system like this unless sharing a two-word structure across security boundaries is easy.

CHERIoT RTOS’ scheduler also exposes each interrupt as a futex word, which is incremented each time the interrupt fires. Users can see if an interrupt has fired by simply checking if this word has increased since they last processed an event (for a lightweight polling mode) and can block using the same futex_wait call as for software events (to return to interrupt-driven mode). You can use the RTOS’ multiwaiter APIs to wait for any of a set of futexes to be woken, which lets you wait for hardware and software events easily in the same call.

This is easy on CHERIoT because handing out a read-only view of a 32-bit word is just a matter of handing a compartment a pointer to that word. It would be possible with MPU-based systems but would require one MPU region per interrupt that a task wants to be able to wait for (remember, most MPUs are limited to 16 or fewer regions). You could make all interrupt counters a single readable region but then you’d have to add another mechanism if you wanted to limit which threads can wait for which interrupts. The same is true for MMUs, except per-interrupt permissions would require a single page (4-64KiB) per interrupt.

Allocating memory

CHERIoT provides a memory-safe shared heap. Dynamic memory allocation became popular in most systems four or five decades ago as a way of reducing overall resource requirements. If you have two workloads that need 64 KiB to 2 MiB of RAM then static provisioning would require you to have 4 MiB of RAM. If they can dynamically allocate memory and they don’t both hit their peaks at the same time then you may need as little as 2,122 KiB. Practically, you may need 3 MiB, but certainly less than 4 MiB. The more tasks you have, the less likely it is that they will hit peak memory requirements at the same time and so the savings are often larger.

The biggest risk with dynamic allocation is that it makes use-after-free bugs possible. If you never free memory, you can’t use it after you free it, so these problems are impossible on systems with no dynamic allocation. Operating systems for big systems mitigate this risk by having (at least) two layers of allocators:

• The operating system allocates pages to processes.
• An in-process allocator allocates objects to the rest of the program.

This two-layer approach has the benefit that use-after-free bugs in one process can’t impact another process because every process’s pointers refer to a private address space. On MPU-based embedded systems, you can approximate this by providing a heap memory range that can grow or shrink, but it’s very hard to design an efficient shared heap where each consumer’s allocations are contiguous in memory.

Most importantly, sharing heap objects between tasks, processes, or some equivalent is very hard with an MMU or MPU. With an MMU, you can’t share anything smaller than a page (and a lot of objects are much smaller than a page). With an MPU, you need to update MPU mappings for each task that has access to an object, which can quickly exhaust the number of regions that you have. This can work for sharing a single object, but what if you want to share an entire object graph? You’d need to walk the graph and find every object reachable from the root, then configure shared regions for these, which is even more likely to exhaust the set of available regions.

In a CHERIoT system, sharing memory is natural. You share an object by passing a pointer to that object to another compartment. The caller can restrict the pointer to provide deep or shallow read-only and no-capture guarantees. The callee can claim the object to prevent the caller from deallocating it in the middle of an operation (the caller can deallocate it, and the object will be removed from their quota immediately, it just won’t go away immediately). When an object is freed, the hardware ensures that all pointers to it are deleted before the memory is reused (so a use-after-free bug will trap).

This means that dynamically allocating and then sharing memory is both safe and a natural programmer abstraction. This, in turn, changes how a lot of operating system services are designed. In particular, all memory is allocated against some quota, encapsulated in an object passed by a sealed capability. This means that it’s easy to limit the amount of memory that a compartment can allocate, but also means that you can delegate the ability to allocate memory on someone else’s behalf.

Going back to the message queue example: the message queue compartment does not have its own quota. When you create a message queue, you pass in an allocation capability that encapsulates the quota against which the memory will be accounted. When you free the message queue, you pass in the same capability.

When you can allocate and share memory at an object granularity, including handing out type-safe opaque pointers to specific kinds of object. All of this means that your APIs for OS services look a lot more like conventional library APIs that don’t have a security boundary, only now they’re secure.

And that’s really the key reason for building a new RTOS. It’s not just about making a secure system. It’s easy to build a secure OS if you’re willing to sacrifice usability. Our goal was to build a platform that made it easy for other people to build secure systems on top. And that meant focusing on secure, reusable, programmer-friendly abstractions from the start. That’s not something that you can retrofit.