All security boundaries are software-engineering boundaries but not all software-engineering boundaries are security boundaries.

What are software engineering boundaries?

From the advent of structured programming onwards, a big part of software engineering has been about modularity and abstraction. Whether these components are functions, objects, modules, or libraries, the core idea is to expose them as abstractions. Users can understand what a component does without having to understand precisely how it does the thing.

This gives two significant benefits. It reduces the cognitive load for users of the component. They do not need to keep track mentally of all of the internal behaviour, whether it’s state or algorithms, they simply need to understand its public interfaces. It also increases freedom for the implementers of the component. They can replace private data structures and algorithms with different ones without needing the users of the component to change anything. They can completely refactor their code and that doesn’t matter to the caller. Similarly, the implementer of a module doesn’t need to understand the code that uses the module.

This happens even in trivial places and with simple abstractions. For example, modern C implementations often have aggressively optimised implementations of strlen or memcpy, but users of these functions don’t need to be aware of how these optimisations work.

What are security boundaries?

Security boundaries much stronger because they presume the presence of an adversary. Where a software engineering boundary says that you don’t need to see across the line, a security boundary may say that you cannot see across the line.

This can be a boundary in one or both directions. A sandboxed function can leak information to the caller but can’t leak the caller’s state to the callee. A safebox function must be able to protect its own data but doesn’t need to be protected from leaking caller state.

Importantly, all of these come with a threat model. They need to understand what they’re trying to protect, from whom, and what the attacker might have access to.

Not all software-engineering boundaries are security boundaries

The first place I encountered this distinction was looking at string classes. It’s a useful software-engineering abstraction to provide an abstract string type. Users don’t need to know how the character data is stored and they get a useful set of tools for things like creating substrings, and so on.

Anyone with a reference to a mutable string can mutate the data inside, so what would the threat model be? They may even be able to alias the storage of the string. Making a string class, or each instance of a string object, a security context would not add any meaningful benefit because the interfaces explicitly allow users to do anything. They would add overhead for no benefit.

None of that eliminates the software engineering benefits of a string type. The same is true for a lot of software-engineering abstractions. Some are intentionally porous and allow you to directly manipulate internal data structures if you know what you’re doing and are willing to take responsibility for bugs that you introduce when you do.

Even C APIs have problems like this. Should memcpy live in a libc compartment, or in its own compartment? The memcpy function needs at least read-only access to the source and read-write access to the destination. The memcpy function doesn’t own any mutable state, so it doesn’t need to protect itself from a caller. If it is a compartment of any kind, it’s a sandbox.

But, on a CHERI system where pointers have bounds, what does the caller expect might go wrong with memcpy? It can’t accidentally go wrong in any meaningful way. If you pass the wrong bounds, it will trap, but that’s a trap caused by the caller.

An actively malicious implementation of memcpy might walk up the stack and arbitrarily corrupt other data. Putting it in an isolated compartment would prevent this. But if memcpy is this broken then there’s very little chance that your program will work at all.

Security boundaries should be software-engineering boundaries

In the other direction, security boundaries do have a lot of the same properties as software-engineering boundaries. They are explicit abstraction layers, where you have information hiding across a boundary. A security boundary needs the same things that a software-engineering boundary needs, such as a clear boundary line, control over visibility across the line, and so on.

It also needs more. Arguments or function returns might not be trusted and need extra validation. The amount of implicit sharing is less across a security boundary (ideally none), whereas software-engineering boundaries can be slightly amorphous.

Software-engineering boundaries often use some form of opaque types. These hide the implementation details and give callers a handle (often a raw pointer). When you harden the interface into a security boundary, you discover that this is is a very powerful abstraction if you can protect those references. This has been known for a very long time. File descriptors in UNIX and handles on Windows have been there from the origins of their respective platforms and provide an opaque type exposed by the kernel. The software-engineering abstraction is used in existing secure interfaces, just with additional hardening (a Windows HANDLE looks like a pointer, but you can’t actually dereference it, for example).

A lot of effort in various platforms has gone into making their security boundaries look like software-engineering boundaries. System calls look like functions. Various remote procedure call (RPC) frameworks for communicating between privilege-separated components do the same in a more general way: make crossing a security boundary (an RPC to another process) look like a software engineering boundary (a function call).

CHERIoT’s answer

In CHERIoT, compartments define security boundaries. They are also software-engineering boundaries. They expose functions, can share data by passing pointers, and can use sealing to implement safe opaque types.

A compartment’s interface is a software-engineering boundary that is also a security boundary. The abstractions used to build security boundaries are all designed to reflect existing tools that programmers are already using to create clean software-engineering boundaries. The tools that we create for programmers to use for hardening those boundaries (checks on capabilities, sealing, claims, and so on) are strictly additive.

At the same time, we realise that not all code reuse should be across a security boundary. You can copy code from a source that you trust into a compartment. This code may have existing abstractions that you don’t explicitly pierce, but you don’t need to make them into security boundaries.

If this were the only way of reusing code, we’d end up with a lot of duplication. This was the original motivation for shared libraries in UNIX and other mainstream systems. It’s also the motivation for CHERIoT’s shared libraries.

Our shared libraries define a software-engineering boundary, while enabling run-time code sharing. CHERIoT shared libraries are stateless (they have no mutable globals), which means that the same code can be trivially shared on a CHERI system. Code in them is invoked directly, you don’t go via the switcher, which would clear registers and truncate the stack for a cross-compartment call. They’re fully trusted by the caller.

Our answer to ‘which compartment should memcpy live in?’ is that it lives in the freestanding library, a shared library that provides the functions needed for a freestanding C environment. The same is true of a lot of other useful functionality. And this means that it simultaneously lives in every compartment that calls memcpy.

By using the same abstractions in both layers, we can often trivially wrap one in the other. The CHERIoT message-queue components are a good example of this. If you trust the implementation to not be actively malicious and you’re communicating between components (such as threads in the same compartment) that you trust to not corrupt the queue state, you can use the message-queue library. If you want to communicate between threads in different compartments, or you simply don’t trust the library not to attack your code, the message-queue compartment wraps it in an isolated security context.

As a side effect, writing code in this style often gives flow isolation for free. The message-queue library has to be stateless because all CHERIoT libraries are. All of the state is held in the message queue, which is passed in by pointer as a function argument. When this is wrapped in a compartment, this becomes a (sealed) type-safe opaque type. The compartment still has no mutable globals, so when it’s operating on one message queue it has no access to any other queue’s state, or any other mutable state. Even if you can push a message into a queue that somehow gets you arbitrary-code execution in that compartment, you can’t violate the confidentiality or integrity of any other queue.

Building security boundaries on top of the same abstractions that you use for building software-engineering boundaries is very powerful.

You’re an absolute timeout

CHERIoT RTOS originally adopted the FreeRTOS notion of timeouts for blocking events. Specifically, the scheduler counted time in ticks (scheduling quanta) and you could limit the number of ticks that a callee would sleep for. We built an API around this model that was easy to compose. Blocking operations took timeouts that were pointers to a structure containing two fields. The first holds the number of ticks that had elapsed so far in blocking operations using this timeout. The second holds the number of ticks still allowed for blocking operations. This let you see how long a callee had slept for and you could simply forward this pointer to other blocking operations to accumulate time.

This works well for a lot of code and is a useful abstraction when you think about sleeping as yielding: allowing other threads to run. A tick is a configuration-specific measure that is intended to represent the time in which a thread can do a useful amount of work.

Sometimes you want to wait a more precise amount of time. In particular, you also want to be able to wake at a specific time. For example, imagine you’re driving an LED matrix display that needs refreshing at 100 Hz. You don’t want to draw one frame and then sleep for approximately 10ms, you want to wake up at 10ms intervals after the first time you ran.

The ticks abstraction is somewhat misleading. CHERIoT RTOS has had a tickless scheduler for a while now: there’s no periodic timer interrupt, the scheduler sets up a timer interrupt for the next time that a runnable thread will wake. Internally, the scheduler translates ticks into a wake time against the monotonic clock and suspends the thread with that position on the wake queue. Adding absolute timeouts was a small change in the scheduler: just bypass that computation. The hard part was maintaining backwards compatibility.

We wanted to allow blocking operations to continue to just forward the parameters that they were passed. Anything that already took a Timeout * as an argument and just forwarded it should be able to keep using the parameter and passing it down to the underlying APIs. At the same time, we wanted to avoid doubling a load of APIs.

We introduced a new type: TimeoutArgument to make this work. This is a union of a Timeout * and an AbsoluteMonotonicTimeout. Absolute timeouts are defined in cycles against the target’s monotonic clock source and are represented as a 64-bit integer (note: a 64-bit integer gives a range of thousands of years at typical monotonic-counter tick rates, so we may repurpose some of the top bits in the future as a discriminator). Because absolute timeouts in ticks are integers and the existing timeouts are pointers, we can use the CHERI tag bit to differentiate them.

Neither standard C nor C++ allow you to pass a Timeout * as an argument to a function that takes a union. Fortunately, this is an old problem. POSIX changed the interface of wait relative to what 4.1BSD shipped and so GCC provided (and clang adopted) a transparent union attribute, which tells the compiler to permit implicit conversions from any of the union members. C++ did not need this attribute because you can explicitly define a constructor for a union from the members.

The monotonic API also lets us more efficiently represent the two most common values of a timeout: 0 and infinity. A lot of callers of potentially blocking APIs want to either permit an unlimited amount of blocking or want to completely prohibit blocking. The new APIs just require passing -1 or 0, both of which can be materialised efficiently and without needing a stack allocation.

If you’re writing code that takes a timeout, please update your argument type from Timeout * to TimeoutArgument. Existing callers will keep working, new callers will be able to provide either kind of timeout.

At the same time, we’ve added some new helper APIs in timeout.h:

• timeout_is_valid checks if a TimeoutArgument is either an absolute timeout or a valid pointer to a Timeout structure.
• timeout_has_expired checks if a TimeoutArgument has already expired.
• timeout_may_block is a convenient helper that negates timeout_has_expired.
• timeout_elapse and timeout_elapse_from are intended for the use case where you want to block for less than the allowed time of a timeout (e.g. one tick). They handle updating the timeout to indicate either a specified number of ticks (or a number of ticks from another Timeout *) have elapsed.

There are also some useful symbolic constants: TimeoutWaitForever and TimeoutNoWait to indicate infinite blocking or no blocking are permitted.

It’s three o’clock and all’s well!

The timeouts are all relative to the monotonic clock. This is the clock that the scheduler uses. That used to be a fairly informal concept in CHERIoT RTOS because all of our simulation platforms and FPGAs had a cycle timer that elapsed at a fixed rate. Real chips may be able to run at different speeds to trade power consumption against performance.

We’ve made this abstraction explicit. Platforms are now required to implement the platform_monotonic_time_read function in platform-time.h. This is a C API that defines how any compartment may read the same clock that the scheduler uses. The scheduler interface remains in the TimerCore type in platform-timer.hh.

The read interface may be a simple CSR read or it may be exposed via a read-only view of a subset of the interface that the scheduler sees. The latter is nice and easy in CHERIoT: you can expose a read-write view of an MMIO range to the scheduler and a read-only view of a 64-bit field in the middle to everyone else.

For a lot of embedded systems, that’s the only time that matters: time intervals matter more than any absolute notion of time. For connected systems, or anything presenting a user interface, may also need to talk about some notion of time connected to a calendar. This has various names. POSIX refers to it as ‘real time’, which is confusing in the context of real-time systems and also implies that the other times are somehow not real. It’s also commonly called the ‘wall clock time’, which is the name that we’ve used, since it’s the time you might see on a clock on a wall. Note for younger readers: before everyone had a brain implant with a chronometer, clocks were often mounted on walls so that multiple people could see the same clock and have a shared understanding of the current time.

We already had some infrastructure for getting the wall-clock time: The SNTP compartment in the network stack exposed a shared object that allowed a gettimeofday implementation to read the time. Some concept of wall-clock time was important in the network stack because TLS requires a wall-clock time to determine whether a certificate is valid.

That worked nicely as long as NTP was the only source of wall-clock time that you cared about, but this isn’t the case for more complex systems. For example, you might have a battery-backed clock that can be used on first boot, which you synchronise with NTP once the network is available.

The approach for addressing this is covered in this design document. We now have a compartment in the RTOS that can be extended with other components that register read-only or read-write interfaces to clock sources. When invoked, it will query available clock sources and write the best one back to any that support storing the time. It will also expose the current delta between the wall clock and monotonic clock via a pre-shared object so that library functions can get the current time without a cross-compartment call.

Along with this, we’ve also added a useful subset of the POSIX time.h functions. These include gmtime for converting from a POSIX timestamp to a human-readable Gregorian calendar version and the BSD extension timegm for converting back. These do not support time zones, but limited time-zone support is easy to add on top.

Booking it in

All of these new features need new documentation!

Work on the second edition of the CHERIoT Programmers’ Guide is underway, including a new chapter on time.

The current working drafts contain the changes from the first edition, so you can see what else has been updated. This is all work in progress and has not yet been edited, but will eventually end up in the second edition after copyediting and review.

Motivation

C and C++ are low-level languages, nominally especially well suited for working with hardware and in performance-sensitive data processing applications, and yet, they lack convenient, portable mechanism for working with bit-packed structures, as often found in hardware memory-mapped I/O (MMIO) registers, network packets, and data serialization formats. By “bit-packed”, we mean that the logical data elements are not necessarily aligned to “natural” boundaries, as defined by the machine processing those data (for example, byte / octet or machine word boundaries), and/or may not fully span between two such “natural” boundaries (for example, a 5-bit field on an 8-bit byte machine). The primary, syntactic mechanism that these languages do offer for sub-word field management is called “bit-fields”, a provision to specify, in C++ jargon, the bit width of of a data member with integral or enumeration type. Concretely, one writes something like

struct MyFields
{
	bool enable : 1;
	enum MyEnum reference : 2;
	unsigned int bias : 8;
};

to specify a structure holding an 1-bit, bool-typed field named enable, a 2-bit enum MyEnum-typed reference, and an 8-bit, unsigned int-typed bias.

Unfortunately, the layout (that is, the in-machine-memory representation) of such definitions is almost completely unspecified by the language standards. C++23 (well, N4928) says:

Allocation of bit-fields within a class object is implementation defined. Alignment of bit-fields is implementation defined. Bit-fields are packed into some addressible allocation unit.

[Note 1: Bit-fields straddle allocation units on some machines and not on others. Bit-fields are assigned right-to-left on some machines, left-to-right on others. – end note]

While there is some provision for forcing alignments of, and controlling padding around, bit-fields, the implementation-defined nature makes even this challenging to actually understand and use.

Moreover, bit fields are very much second class citizens:

• they may not have their address taken (so there are no pointers to bit-fields; what would be the type of such a thing?);

• references to bit-fields must be const (allowing implementations to quietly convert the reference to one of the bit-field’s underlying type);

• because bit-fields exist within aggregate structures that are not themselves numeric types, converting between a bit-field aggregate and a numeric representation can be particularly troublesome, especially in constexpr contexts where reinterpret_cast-s are forbidden.

As a result, implementations of drivers and codecs and such in practice tend to eschew language bit-fields in favor of explicit bit-wise operations on numerically-typed words. They may even go so far as to eschew C/C++ aggregates (struct-s and such) completely and view entire MMIO regions as simply arrays of uint32_t-s, with information about the word and sub-word fields’ layout conveyed “out of band” via, for example, preprocessor #define-s.

Given the near ubiquity of dealing with bit-packed structures in embedded contexts, we wanted something better. Thus, on an experimental basis, we have introduced “bitpacks” to the CHERIoT-RTOS programming framework. Bitpacks are, at their core, simply machinery and syntax wrapping bitwise operators (shift, complement, AND, OR) on an underlying value of integral type (say, uint32_t), making it much more pleasant to view (and write code that views) that underlying value as containing sub-fields.

The remainder of this post gives examples of that machinery (and of different ways of thinking about, naming, and typing sub-fields) built around a simple, fictitious, but representative, MMIO device.

A Simple, Fictitious MMIO Device

Imagine a simple peripheral that generates data and has some configuration knobs about how it does so, something like a very rough sketch of an analog-to-digital converter. Let’s say that our device has

• a 32-bit control register with…

  • a control bit that enables or disables the device’s operation, which must be disabled to configure the peripheral. The mnemonics for this field’s values are NoGo and Go.

  • a status bit that indicates whether the device is enabled and active. The mnemonics for this field’s values are Off and Running.

  • a choice of “reference”, with one internal and two external sources, creatively named “A” and “B”, giving us an enumeration with mnemonics Internal, ExternalA, and ExternalB.

  • an 8-bit “bias” value. This field is more numeric than an enumeration, so has no associated mnemonics.

• a 32-bit data register that can be read once the device is active.

Graphically, we might render that MMIO layout something like this:

Code Without Bitpacks

An Array of uint32_t Is All You Need?

A minimal, easily generated software description of such a MMIO device might look like this:

#define MYDEVICE_OFFSET32_CONTROL 0

#define MYDEVICE_CONTROL_ENABLE_SHIFT 0
#define MYDEVICE_CONTROL_ENABLE_MASK 0x1
#define MYDEVICE_CONTROL_ENABLE_V_NOGO 0x0
#define MYDEVICE_CONTROL_ENABLE_V_GO 0x1

#define MYDEVICE_CONTROL_ACTIVE_SHIFT 1
#define MYDEVICE_CONTROL_ACTIVE_MASK 0x2
#define MYDEVICE_CONTROL_ACTIVE_V_OFF 0x0
#define MYDEVICE_CONTROL_ACTIVE_V_RUNNING 0x1

#define MYDEVICE_CONTROL_REFERENCE_SHIFT 8
#define MYDEVICE_CONTROL_REFERENCE_MASK 0x300
#define MYDEVICE_CONTROL_REFERENCE_V_INTERNAL 0x0
#define MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A 0x1
#define MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_B 0x3

#define MYDEVICE_CONTROL_BIAS_SHIFT 16
#define MYDEVICE_CONTROL_BIAS_MASK 0xFF0000

#define MYDEVICE_OFFSET32_DATA 1

(For a real-world example, the OpenTitan Register Generator tool produces C headers that look much like this.)

These #define-s give

• each register’s offset, in terms of 32 bit words,

• the shift to reach the least significant bit of each field,

• the (shifted, in-situ) mask of each field, and

• an enumeration of values for enumeration-typed fields.

An example function using such a definition to write code to configure the device, wait for it to become active, and then returning the first word of data could look like this, taking a pointer to the device’s MMIO region’s base:

uint32_t test(volatile uint32_t *device)
{
	// set device.control's enable field to NOGO
	device[MYDEVICE_OFFSET32_CONTROL] =
	  (device[MYDEVICE_OFFSET32_CONTROL] & ~MYDEVICE_CONTROL_ENABLE_MASK) |
	  (MYDEVICE_CONTROL_ENABLE_V_NOGO << MYDEVICE_CONTROL_ENABLE_SHIFT);

	/*
	 * Set device.control's reference field to EXTERNAL_A value and set bias to
	 * 0x7f at the same time.  Leave the enable field unchanged.
	 */
	device[MYDEVICE_OFFSET32_CONTROL] =
	  (device[MYDEVICE_OFFSET32_CONTROL] &
	   ~(MYDEVICE_CONTROL_REFERENCE_MASK | MYDEVICE_CONTROL_BIAS_MASK)) |
	  (MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A
	   << MYDEVICE_CONTROL_REFERENCE_SHIFT) |
	  (0x7f << MYDEVICE_CONTROL_BIAS_SHIFT);

	// set device.control's enable field to GO
	device[MYDEVICE_OFFSET32_CONTROL] =
	  (device[MYDEVICE_OFFSET32_CONTROL] & ~MYDEVICE_CONTROL_ENABLE_MASK) |
	  (MYDEVICE_CONTROL_ENABLE_V_GO << MYDEVICE_CONTROL_ENABLE_SHIFT);

	// wait for device.control's active field to be RUNNING
	while ((device[MYDEVICE_OFFSET32_CONTROL] & MYDEVICE_CONTROL_ACTIVE_MASK) !=
	       (MYDEVICE_CONTROL_ACTIVE_V_RUNNING << MYDEVICE_CONTROL_ACTIVE_SHIFT))
	{
		yield();
	}

	return device[MYDEVICE_OFFSET32_DATA];
}

How About a struct of uint32_t-s?

We can use C++ aggregates to at least make the “outer” layer of the MMIO structure, the layout and naming of the registers (but not their fields). Keeping the per-field parts of the above definition, but replacing the register layout information with slightly more idiomatic C++ makes our example look something like this:

struct RawDevice
{
	uint32_t control;

	uint32_t data;
};

// MYDEVICE_CONTROL_... as above

uint32_t test(volatile RawDevice &device)
{
	// set device.control's enable field to NOGO
	device.control =
	  (device.control & ~MYDEVICE_CONTROL_ENABLE_MASK) |
	  (MYDEVICE_CONTROL_ENABLE_V_NOGO << MYDEVICE_CONTROL_ENABLE_SHIFT);

	/*
	 * Set device.control's enable field to GO at the same time set the
	 * reference field to EXTERNAL_A value.
	 */
	device.control = (device.control & ~(MYDEVICE_CONTROL_REFERENCE_MASK |
	                                     MYDEVICE_CONTROL_BIAS_MASK)) |
	                 (MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A
	                  << MYDEVICE_CONTROL_REFERENCE_SHIFT) |
	                 (0x7f << MYDEVICE_CONTROL_BIAS_SHIFT);

	// set device.control's enable field to GO
	device.control =
	  (device.control & ~MYDEVICE_CONTROL_ENABLE_MASK) |
	  (MYDEVICE_CONTROL_ENABLE_V_GO << MYDEVICE_CONTROL_ENABLE_SHIFT);

	// wait for device.control's active field to be RUNNING
	while ((device.control & MYDEVICE_CONTROL_ACTIVE_MASK) !=
	       (MYDEVICE_CONTROL_ACTIVE_V_RUNNING << MYDEVICE_CONTROL_ACTIVE_SHIFT))
	{
		yield();
	}

	return device.data;
}

This compiles to the same code as the test(volatile uint32_t *) implementation above.

(ARM’s CMSIS contains many examples of using struct-s for the layout of register words and #define-s for fields within registers).

C++ Bitfields?

We can try to write our example with C++ bitfields, despite their implementation-defined nature (perhaps we believe that there is and will only ever be exactly one C++ ABI for our machine). That would make the MMIO region definition look something like this:

struct BFDevice
{
	struct Control
	{
		bool enabled : 1;

		bool active : 1;

		uint32_t : 6; // padding

		uint8_t reference : 2;

		uint32_t : 6; // padding

		uint32_t bias : 8;

		uint32_t : 8; // padding
	} control;

	uint32_t data;
};

static_assert(sizeof(BFDevice::Control) == 4);

And in use, we’d run into the fact that C++ does not implicitly define copy operators suitable for use with volatile bitfield aggregates, so we’d resort to doing more writes than the above examples, with

uint32_t test(volatile BFDevice &device)
{
	device.control.enabled = false;

	// challenging to make these into one MMIO write, like we could above
	device.control.reference = MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A;
	device.control.bias      = 0x7f;

	device.control.enabled = true;

	while (!device.control.active)
	{
		yield();
	}

	return device.data;
}

All the challenges aside, that sure looks like nicer code! So, how do bitpacks help?

Introducing Bitpacks

Bitpacks offer a way to define, view, and manipulate typed fields within a numeric word. Internally, fields are simply contiguous spans of bits within that word, and operations are phrased in terms of bitwise operations (shift, complement, AND, OR). Fields are accessed through “proxies”, each of which can be thought of as a reference to a field within a particular word.

One defines a bitpack by defining a class (or struct) that inherits from the Bitpack<typename Storage> class template. The Storage template parameter defines the size (and type) of the underlying numeric word.

Defining a field – specifying its position within the word – amounts to filling out a FieldInfo structure:

struct FieldInfo
{
	/// Minimum 0-indexed bit position occupied by this field
	size_t minIndex;

	/// Maximum 0-indexed bit position occupied by this field
	size_t maxIndex;

	/// Should proxies of this field not provide affordances for mutation?
	bool isConst = false;
};

A FieldInfo {3, 7}, for example, specifies a field spanning bits 3 through 7 of a word. From this, bitpacks can, internally, compute things like the _SHIFT and _MASK values used above. We will almost never encounter FieldInfo-s by name, but we will specify initializers for them when defining fields.

The simplest way to build a proxy of a field, so that we can access it, is to define a method within our bitpack that invokes the protected Bitpack::member<FieldType, FieldInfo> method with appropriate arguments. This is called a “named member” and the C++ syntax involved is encapsulated within the BITPACK_MEMBER_ADD macro, to which we now direct our attention.

Named Members

We can define a bitpack for our device’s control register, and specify a named member for each field within, thus:

struct NuDevice
{
	struct Control : Bitpack<uint32_t>
	{
		BITPACK_USUAL_PREFIX;

		BITPACK_MEMBER_ADD(enable, bool, 0, 0); // NoGo is false / Go is true

		BITPACK_MEMBER_ADD(active, bool, 1, 1, true); // Off / Running

		BITPACK_MEMBER_ADD(reference,
		                   uint8_t,
		                   8,
		                   9); // values from MYDEVICE_CONTROL_REFERENCE_V_...

		BITPACK_MEMBER_ADD(bias, uint8_t, 16, 23);
	} control;

	uint32_t data;
};

As discussed, struct Control now inherits from Bitpack<uint32_t>. The BITPACK_USUAL_PREFIX macro hides some syntactic chatter that’s often useful (including, notably, constructors and assignment operators).

Our fields are now specified by invocations of the BITPACK_MEMBER_ADD(n, T, ...) macro. These invocations define methods named by n, each returning a T-based field proxy for the field whose FieldInfo follows.

Field proxies offer a number of methods on the field they reference:

• Getting the current value of the field. Proxies implicitly convert to their field’s type, so this is often syntactically “free”, but it is also explicitly available as the proxy’s .raw() method. In implementation, this is the expected shift and a mask to extract the field’s value.

• Setting the field value. Proxies overload the assignment operator (=) to update the field’s bits appropriately. All other bits in the underlying word are not altered. In implementation, this is the expected sequence of shift, complement, AND, and OR operations (as we’ve seen above). This is not provided by proxies of fields whose FieldInfo::isConst flag is true.

• Read-modify-write functionality. Proxies provide a .alter(f) method which invokes the callback f with the field’s current value and updates the field’s value with the callback’s return value.

• Cloning with override. Proxies provide a .with() method that behaves much like .alter() except that it returns a modified copy of the word. .with() can also be given a new value directly, rather than a callback.

• Comparison. Proxies overload the <=> operator, and can be compared to other proxies (of compatible types) or values of the field’s type.

Continuing our example, the code that uses this bitpack-flavored definition looks like…

uint32_t test(volatile NuDevice &device)
{
	device.control.enable() = false;

	device.control.alter([](auto c) {
		c.reference() = MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A;
		c.bias()      = 0x7f;
		return c;
	});

	device.control.enable() = true;

	while (!device.control.read().active())
	{
		yield();
	}

	return device.data;
}

This compiles to the same code as the test(volatile RawDevice &) implementation above (and avoids the extra write that we had to introduce for test(volatile BFDevice &)).

Proxy assignments and .alter() methods are defined for volatile bitpack references (such as device.control in this example, because the containing NuDevice is volatile), and each such performs a read-modify-write operation on the underlying word. Proxy are more picky about field getter operations, and will refuse to operate on volatile bitpack references; the bitpack itself offers a .read() method that returns a non-volatile snapshot of the (volatile) bitpack. This serves to keep accesses explicit in the source and allows for clearly differentiating multiple tests of a single read snapshot from many tests each performing its own read of field value(s).

Compiler Error Messages

The compiler error messages for attempts to get a value from a volatile bitpack are, admittedly, nothing short of unfortunate. The best (well, most informative) case happens if we replace while (!device.control.read().active()) with while(!device.control.active().raw()) and looks something like this:

sdk/include/bitpack.hh:657:12: error: no matching conversion for static_cast from 'const Bitpack<...>::Field<bool, FieldInfo{...}>::Proxy<...>' to 'Field::Type' (aka 'bool')
...
./tmp/bitpack-blog.cc:172:34: note: in instantiation of member function 'Bitpack<...>::Field<bool, FieldInfo{...}>::Proxy<...>::raw' requested here
  172 |         while (!device.control.active().raw())
      |                                         ^
./sdk/include/bitpack.hh:649:14: note: candidate template ignored: constraints not satisfied [with Self = ...]
...
./sdk/include/bitpack.hh:648:10: note: because '!std::is_volatile_v<...>' evaluated to false
...

only with a lot more C++ template and Bitpack internal goo in place of those ellipses. At least it points at is_volatile as the problem!

Sometimes, if C++ is attempting further automagic (say, searching for implicit conversions) which fails, because we instead wrote while(!device.control.active()) (note the missing .raw()), the compiler will not actually emit such diagnostics of the root cause and will instead offer more mysterious reports like

./tmp/bitpack-blog.cc:172:9: error: invalid argument type 'typename Field<bool, FieldInfo{...}>::Proxy<...>'
      (aka 'Proxy<...>') to unary expression
  172 |         while (!device.control.active())
      |                ^~~~~~~~~~~~~~~~~~~~~~~~

This latter report is particularly confusing because it fails to point at volatile as a possible cause. Ideas for improving error messages are eagerly accepted.

Named Members with Custom Global Types

Much of the bitpack machinery is an attempt to encourage the use of fields with custom types. For example, we might give a name to the “reference” enumeration for our device, thus:

enum class MyDeviceControlReference : uint8_t
{
	Internal  = 0,
	ExternalA = 1,
	ExternalB = 3,
};

We can take advantage of this type by changing the reference member in our bitpack:

struct NeDevice
{
	struct Control : Bitpack<uint32_t>
	{
		BITPACK_USUAL_PREFIX;

		BITPACK_MEMBER_ADD(enable, bool, 0, 0); // NoGo is false / Go is true

		BITPACK_MEMBER_ADD(active, bool, 1, 1, true); // Off / Running

		BITPACK_MEMBER_ADD(reference, MyDeviceControlReference, 8, 9);

		BITPACK_MEMBER_ADD(bias, uint8_t, 16, 23);
	} control;

	uint32_t data;
};

The corresponding use code changes very little, but introduces our first use-side bitpack macro, BITPACK_OPERATE_QUALIFY:

uint32_t test(volatile NeDevice &device)
{
	device.control.enable() = false;

	device.control.alter([](auto c) {
		BITPACK_OPERATE_QUALIFY(c.reference(), =, ExternalA);
		c.bias() = 0x7f;
		return c;
	});

	device.control.enable() = true;

	// .read() makes read of volatile word clear in source
	while (!device.control.read().active())
	{
		yield();
	}

	return device.data;
}

Again, this compiles to the same code as the test(volatile NuDevice &) implementation above.

That BITPACK_OPERATE_QUALIFY macro invocation lets us avoid the need to manually qualify the ExternalA name; C++ name resolution rules would require that we write

c.reference() = MyDeviceControlReference::ExternalA;

The macro invocation takes advantage of the ability to look at the field’s type at compile time. It expands, essentially, to

c.reference() = (decltype(c.reference())::Field::Type)::ExternalA;

which is to say

c.reference() = MyDeviceControlReference::ExternalA;

Named Members with Custom Local Types

The same test code (other than the type of the parameter) also works (and compiles to the same code) when we move the reference field’s enumeration type into the Control structure itself:

struct NaDevice
{
	struct Control : Bitpack<uint32_t>
	{
		BITPACK_USUAL_PREFIX;

		BITPACK_MEMBER_ADD(enable, bool, 0, 0); // NoGo is false / Go is true

		BITPACK_MEMBER_ADD(active, bool, 1, 1, true); // Off / Running

		enum class Reference : uint8_t
		{
			Internal  = 0,
			ExternalA = 1,
			ExternalB = 3,
		};
		BITPACK_MEMBER_ADD(reference, Reference, 8, 9);

		BITPACK_MEMBER_ADD(bias, uint8_t, 16, 23);
	} control;

	uint32_t data;
};

With that definition, the BITPACK_OPERATE_QUALIFY(c.reference(), =, ExternalA) macro invocation in our test method still expands to

c.reference() = (decltype(c.reference())::Field::Type)::ExternalA;

but that now means

c.reference() = MyDevice::Control::Reference::ExternalA;

We strongly encourage the use of such local types, as it uses the device and register structures as namespaces, minimizing the amount of stuff at global scope and making the purpose of a type clearer.

Local Types as Member Names

In fact, fields can often be thought of as being named by their type: this device’s enable flag is of a different type from that device’s, even though they’re both active-high single-bit fields, and similarly for the other fields. It would be strange, after all, to read configuration values from one device and write them directly to another; we can ask the C++ type system to help ensure that we don’t accidentally do so.

Towards this end, bitpacks include a fair bit of machinery for introducing a custom type for a particular field and then using that type as the name of that field. To make use of this machinery, we first replace (or augment) the BITPACK_MEMBER_ADD macros above with BITPACK_MEMBER_ADD_* macros, which simultaneously introduce a new type and associate field information with it.

struct TyDevice
{
	struct Control : Bitpack<uint32_t>
	{
		BITPACK_USUAL_PREFIX;

		BITPACK_MEMBER_ADD_ENUM_BOOL(Enable, NoGo, Go, 0);

		BITPACK_MEMBER_ADD_ENUM_BOOL(Active, Off, Running, 1, true);

		BITPACK_MEMBER_ADD_ENUM(Reference, uint8_t, 8, 9) {
			Internal  = 0,
			ExternalA = 1,
			ExternalB = 3,
		};

		BITPACK_MEMBER_ADD_NUMERIC(Bias, uint8_t, 16, 23);
	} control;

	uint32_t data;
};

BITPACK_MEMBER_ADD_ENUM(T, B, ...) defines an enum class T with base type B; the definition of the enumeration follows the macro. BITPACK_MEMBER_ADD_ENUM_BOOL(T, VF, VT, bit, ...) is a wrapper thereof, defining an enum class T : bool with the enumerator VF having value false and VT true; because booleans occupy exactly one bit, it is not necessary to specify both the minimum and maximum bit positions. BITPACK_MEMBER_ADD_NUMERIC(T, B, ...) defines a class T which inherits from a Numeric<B> base, an internal helper class that provides, for example, implicit conversions to and from B. (Not all operations on B are supported by Numeric<T> and derived classes; if this proves problematic, do let us know and we’ll see what can be done!)

Rather than having named methods for each field of our bitpack, we now make use of a singular method template, .member<T>(), which returns a proxy of the field whose type is T. Because the types are defined within a bitpack type itself, they often need qualification:

• The BITPACK_MEMBER_DECLTYPE(b, T) macro gets the proxy of the field whose type is T qualified by the type of the bitpack value b. (That is, it expands to (b).template member<decltype(b)::T>().)

• If the type of the bitpack value b is “dependent” (in the C++ sense), use the BITPACK_MEMBER_DEPENDENT(b, T) macro, which differs only in that it calls for dependent resolution of the type named T. (That is, it expands to (b).template member<typename decltype(b)::T>().)

We can make use of macros that combine these and the OPERATE_QUALIFY form we’ve seen above when writing code that uses these type-centric bitpack definitions:

uint32_t test(volatile TyDevice &device)
{
	BITPACK_OPERATE_QUALIFY_DECLTYPE(device.control, Enable, =, NoGo);

	device.control.alter([](auto c) {
		BITPACK_OPERATE_QUALIFY_DEPENDENT(c, Reference, =, ExternalA);
		BITPACK_MEMBER_DEPENDENT(c, Bias) = 0x7f;
		return c;
	});

	BITPACK_OPERATE_QUALIFY_DECLTYPE(device.control, Enable, =, Go);

	while (BITPACK_OPERATE_QUALIFY_DECLTYPE(
	  device.control.read(), Active, !=, Running))
	{
		yield();
	}

	return device.data;
}

This too compiles to the same code as the test(volatile NuDevice &) implementation above.

BITPACK_OPERATE_QUALIFY_DECLTYPE(b, T, op, v) is just BITPACK_OPERATE_QUALIFY(BITPACK_MEMBER_DECLTYPE(b, T), op, v). Here, rather than using a method like .reference() to obtain a field proxy by name we use BITPACK_MEMBER_DECLTYPE to use the field’s type as its name and construct the appropriate proxy. BITPACK_OPERATE_QUALIFY_DEPENDENT, analogously, uses BITPACK_MEMBER_DEPENDENT internally. In much more detail, BITPACK_OPERATE_QUALIFY_DECLTYPE(c, Reference, =, ExternalA) can be thought of as expanding to

BITPACK_OPERATE_DECLTYPE(c.template member<TyDevice::Reference>(), =, ExternalA)

which then expands to

c.template member<TyDevice::Reference>() =
  decltype(c.template member<TyDevice::Reference>)::Field::Type::ExternalA

which is to say

c.template member<TyDevice::Reference>() = TyDevice::Reference::ExternalA

(Internally, the difference between explicitly named members and type-named members amounts to the former invoking .member<FieldType, FieldInfo>() while the latter invokes .member<FieldType>(). This second form uses some type-level computation machinery buried inside the BITPACK_USUAL_PREFIX and BITPACK_MEMBER_ADD_* macros to map from a FieldType to its FieldInfo.)

We could also have written, instead of BITPACK_MEMBER_DEPENDENT(c, Bias) = 0x7f, BITPACK_OPERATE_WRAP_DEPENDENT(c, Bias, =, 0x7f). The former makes use of Numeric<T>’s implicit conversion from T, while the latter is more general. The BITPACK_OPERATE_WRAP_... macros are similar to the BITPACK_OPERATE_QUALIFY_... except that they expand to invoke the indicated type’s constructor, rather than simply qualify the value. That is, BITPACK_OPERATE_WRAP(proxy, op, value) expands to (proxy) op (F{value}), invoking the F constructor, where F is the type of the field being proxied by proxy (specifically, decltype(proxy)::Field::Type).

Looking to the Future

Bitpacks are a public experiment and, as such, a request for comments. We are especially open to feedback about use cases that remain un-ergonomic or otherwise challenging.

At present, most of our use of bitpacks has been with manually written bitpack definitions. In the interest of reducing the number of “sources of truth” that must be kept synchronized, we have been pondering tools that could generate bitpack definitions from other register description languages (especially https://opentitan.org/book/util/reggen/index.html), or assertions of equivalence between a manually-written bitpack definition and one in another language. Should there be external interest, we would be keen to collaborate on this front.

Readers might rightly object that bitpacks, and their professed typed perspective of sub-word fields, lean rather heavily on C preprocessor macros (which act on the level of source code tokens, well below any notion of types). We hope that C++’s reflection machinery (to first appear in C++26) should allow many of these macros to go away once it’s a bit more complete.

Conclusion

This has been a whirlwind tour of the “bitpack” machinery available in the CHERIoT-RTOS programming environment. Bitpacks are meant to increase legibility, portability, and type-safety of low-level code that manipulates sub-fields of numeric words. We hope that you find they have achieved their purpose!

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away

– Antoine de Saint-Exupéry

In the last week or so, we’ve started landing patches in our compiler to remove the auicgp instruction. This is something that we’d hoped to do for a while because we want to remove the instruction entirely in CHERIoT v2 (to be based on the upcoming RV32YE standard). This post will explain why we had it, why we wanted to get rid of it, and how we’ve removed the need for it.

What is auicgp?

In CHERIoT, every global is either accessed relative to the program counter (PC) for code or immutable globals, or the global pointer (GP) for read-write globals. On RISC-V, PC-relative addressing is done via the auipc instruction, which takes a 20-bit immediate, left shifts it by 12, and adds it to the PC. This is used paired with a load, store, or add (if you just want to materialise the address as a pointer) to fill in the remaining bits.

For symmetry with base RISC-V (more detail on this later), we introduced a similar instruction that added a shifted immediate to the global pointer: add upper immediate to (capability) global pointer (auicgp). Originally, because we’re based on RV32E, we used one of the bits in the destination-register field to differentiate this from auipcc instruction, so it fitted into some spare encoding space.

In the initial public release, we changed this because RISC-V promises never to use that bit of encoding space for standard extensions and we hoped to standardise it. Since then, the auicgp instruction has been the least popular bit of the CHERIoT ISA, for a variety of reasons.

First, it’s an enormous instruction. With a register target and a 20-bit immediate, it uses a full major opcode. That’s 1/128 of the total encoding space. In practice, it’s worse than that, because the RISC-V encoding requires 32-bit instructions to start 11 (01, 10, and 00 are reserved for 16-bit encodings), it’s actually 1/32 of the space available for 32-bit instructions. That’s a lot to pay, though it might be worth it if the instruction were used a lot.

Unfortunately, auicgp is very rarely used. Our ABI points the global pointer to the middle of the globals region, which means that we can use the entire 4 KiB range of the 12-bit immediate in loads, stores, and adds. On RISC-V, the normal model is for the compiler to emit a long instruction sequence for operations like global accesses and then the linker to relax this by deleting instructions. We followed this pattern for auicgp, with the linker deleting it if the immediate is 0. As a result, almost all of the auicgp instructions were deleted. Our original linker version didn’t implement support for relaxations (upstream LLD didn’t support them for RISC-V when we started working on CHERIoT!) and so we weren’t initially sure how feasible deleting them would be, but we’ve had linker relaxations working for a few years now and they work very well.

The RTOS test suite contains a few auicgps, but only because we’ve intentionally written tests to make sure that large globals work. In most firmware images, there are no instances of the instruction that survive relaxation. So we’re using 1/32 of the encoding space for 32-bit instructions for an instruction that, to a first approximation, never gets used.

If that isn’t bad enough, it’s also annoying for simple pipelines to implement. RISC-V is designed so that the source and destination registers are always in the same place in instructions. Once you move to long pipelines, have register rename, or extensions with multiple register files (such as floating-point or vector extensions) then this is of little or no benefit. But CHERIoT targets the kind of microcontroller design that RISC-V overfitted for. Simple in-order pipelines find it useful to do register fetch early. The auicgp instruction is the only one that has fetches from the general-purpose register file but does not have the registers in the correct location. This adds extra multiplexing that, on a short in-order pipeline such as CHERIoT Ibex, impacts the critical-path length.

So, unfortunately, there is no situation in which this instruction is a good idea, except that it made it easy to create dense code via largely orthogonal code paths. To understand why we incorporated it, you need to understand how RISC-V normally generates pointers to globals.

Why did we have auicgp?

Let’s start with the simple case in RISC-V, of a static binary (how you’d normally compile embedded firmware) accessing a global. A simple sequence loading an int from a global called x would look like this:

	lui     a0, %hi(x)
	lw      a0, %lo(x)(a0)

This has two relocations. The load upper immediate instruction will materialise the top 20 bits of the address via the hi relocation, then the load will materialise the low 12 bits in its immediate from the lo relocation. This is very efficient, but doesn’t work on a CHERI system because you can’t just make up an integer address and use it for loads and stores.

That’s not a big problem, because it’s the same issue that position-independent code needs to handle, and RISC-V does this with the same number of instructions. When you’re compiling position-independent code, RISC-V will instead generate the following sequence:

.Lpcrel_hi0:
	auipc   a0, %pcrel_hi(x)
	lw      a0, %pcrel_lo(.Lpcrel_hi0)(a0)

Again, this materialises the address in two instructions, including folding half of the address-materialisation into the load. The difference is that the high bits are not created as an absolute address, they are created by adding a value to the current PC value. The %pcrel_lo relocation looks a bit odd because it refers to the address of the auipc instruction, not the address of x. This is because of limitations in ELF relocations. The linker needs to know both the address of the auipc instruction (which doesn’t have to be next to the lw, and might not be if the lw is in a loop) and of the target, but ELF relocations can’t express both of these. Fortunately, the linker can work around this by looking up the relocations that apply to the address of the auipc instruction to find the target.

On CHERIoT, accessing read-only globals works in exactly the same way. They are stored within the bounds of the program counter for the current compartment (or library) and so we can compute a displacement and load. Read-write globals are instead stored within a region reachable via the global pointer. These capabilities don’t overlap, and they have distinct permissions.

This is where we encounter the first problem. Most of the time, the compiler knows whether a global is read-only or not and so knows whether it needs to emit a sequence relative to the program counter or the global pointer. This is not universally true. In particular, C inherited a notion of ‘common linkage’ from Fortran and this makes it possible for different compilation units to disagree on whether something is read-only and leaves it for the linker to figure it out.

For read-only globals, the sequence looks like this (note: the ct. prefix is the official vendor prefix for CHERIoT instructions):

.LBB0_1:
	ct.auipcc    a0, %cheriot_compartment_hi(x)
	ct.clw       a0, %cheriot_compartment_lo_i(.LBB0_1)(a0)

But for read-write globals, it looks like this:

.LBB0_1:
	ct.auicgp   a0, %cheriot_compartment_hi(x)
	ct.clw      a0, %cheriot_compartment_lo_i(.LBB0_1)(a0)

This hopefully makes it clear how the auicgp instruction is useful. These two sequences look identical and so the linker can rewrite one to the other by simply rewriting the opcode of the first instruction. If the compiler tries to do PC-relative addressing for a read-write global, the compiler can turn the auipcc into an auicgp, or vice versa.

For read-write globals, the linker can also do relaxation. In the (common, almost universal) case that the address of x is within the displacement from the global pointer of a single instruction, the second sequence becomes this:

	ct.clw	a0, {displacement of x from the global pointer}(c)

Note that this is now fewer instructions than the baseline RISC-V case. We hit this in a lot of cases, because a lot of global accesses are simple reads and writes of scalar values. That’s not universally true. If the address of the global escapes the analyses that the compiler uses to prove that a dereference is in-bounds (which are currently fairly naïve, but will improve over time), then we must materialise a full capability.

This is why it’s important to design a CHERI ABI along with a threat model. Our threat model requires that any compartment be able to enforce a list of memory-safety properties against untrusted code, but treats things like memory safety for stack or global variables within a compartment as defence-in-depth properties. Code within a compartment is allowed to access all globals for that compartment and so the threat model says it’s fine to make the compiler responsible for enforcing those bounds. But when a pointer to a global is passed to another compartment (possibly indirectly via another compilation unit, maybe in a different language, in the same compartment) then the compiler taking the address of that global must be able to apply bounds and permissions.

If we are taking the address of x, the compiler will generate a sequence like this:

.LBB0_1:
	ct.auicgp       a0, %cheriot_compartment_hi(x)
	ct.cincoffset   a0, a0, %cheriot_compartment_lo_i(.LBB0_1)
	ct.csetbounds   a0, a0, %cheriot_compartment_size(x)

Note that now the second part of the address isn’t folded into the load instruction. This doesn’t hurt code size as much as you might think, because now we’re likely to do offset-zero loads and stores on the result and RISC-V lets us express these as 16-bit instructions.

The new instruction is applying the bounds. As before, the linker will normally relax away the first instruction, so we end up with this sequence for taking the address of a global:

	ct.cincoffset   a0, a0, {offset of x}(gp)
	ct.csetbounds   a0, a0, {size of x}(x)

Again, this is the same number of instructions as baseline RISC-V, but now with bounds applied. For comparison, here is the RISC-V version:

	auipc   a0, %pcrel_hi(x)
	addi    a0, a0, %pcrel_lo(.Lpcrel_hi0)

There is one annoying case though: what happens if the bounds are too large for an immediate? Here, we need an extra instruction (and register) to hold the immediate. This means that our worst case (large bounds, address taken or not provably in bounds) is five instructions (if it takes two instructions to materialise the immediate for the bounds), much worse than our best case of one.

The approach taken on the ABI for big CHERI is to use an indirection layer, still referred to as a global offset table (GOT), even though it doesn’t contain offsets. This mirrors the baseline RISC-V GOT model, where the same sequence would be:

        auipc   a0, %got_pcrel_hi(x)
        lw      a0, %pcrel_lo(.Lpcrel_hi0)(a0)

At two instructions, this looks cheap, but note that the second is a load, which is loading the address from the GOT. This means that, although it’s only eight bytes of instructions, it’s also one pointer’s worth of data (eight more bytes for CHERIoT, sixteen for big CHERI). The second instruction is accessing memory (even on CHERIoT systems with local SRAM, memory-access instructions are typically slower than ALU instructions).

This is worse than our common case, but is often better than our worst case.

The new model

In the new approach, any global that isn’t reachable via a short displacement from the global pointer (i.e. things that would need auicgp) is turned into a GOT-relative access.

This also serves as a building block for other optimisations. The three-instruction sequence is smaller than the GOT-relative access if a global is accessed only once, but is larger if the same global has its address taken three or more times. The linker has full visibility into the number of accesses to a global and so can make this decision.

Similarly, globals that are too big to have their bounds applied in a single instruction will be able to fall back to the GOT mechanism. These are rare but they do occur for things like frame buffers. These are close to break even for a single use. For example, imagine if x is an array of 1028 ints. Taking the address is currently this (long!) sequence:

	lui             a0, 1
	addi            a0, a0, 16
.LBB0_1:
	ct.auicgp       a1, %cheriot_compartment_hi(x)
	ct.cincoffset   a1, a1, %cheriot_compartment_lo_i(.LBB0_1)
	ct.csetbounds   a0, a1, a0

Assuming that the auicgp is relaxed away, this becomes:

	lui             a0, 1
	addi            a0, a0, 16
	ct.cincoffset   a1, a1, {offset of x}(gp)
	ct.csetbounds   a0, a1, a0

This is four instructions, but both the lui and addi will, in this case, be the 16-bit variants, so this is 12 instructions: smaller than the GOT-relative sequence plus the GOT entry. This sequence also requires an additional register, and removing that requirement can improve code generation in the rest of the function.

To support this, we’ve introduced a new ct.auipcc.data pseudo-instruction that emits padding nop that is normally relaxed away, but provides space for the linker to emit the longest sequence. The sequence the compiler will emit looks like this:

.LBB4_1
	ct.auipcc.data  s0, %cheriot_compartment_data_hi(x)
	ct.cincoffset   s0, s0, %cheriot_compartment_lo_i(.LBB4_1)
	ct.csetbounds   s0, s0, %cheriot_compartment_size(x)
	ct.cincoffset   a0, s0, a0

The linker can then relax this to any of the sequences described above, almost always to something much shorter.

The extra padding is needed only in one specific case, but emitting it in all cases that may use the global pointer simplifies the linker logic. If the compiler statically knows that the access is in bounds, it will emit the short two-instruction sequence of auipcc followed by a load or store. This is two 32-bit instructions (8 bytes total). If the global is large, this must be transformed into a GOT load followed by a load or store, which is three instructions, two of which might be 16-bit, but in the worst case all three will need the full range of the 32-bit variants. The linker always removes the nop but may use the space that it reserved for the longer sequence, in this one corner case.

A final note

CHERIoT benefits enormously from having co-designed the ISA, ABI, and programmer model. The cheriot-audit tool makes it easy to reason about the security of a firmware image. This tool is possible because we defined our ABI so that everything that a compartment accesses outside of its code and globals region is explicit (in the programmer model).

Several of these optimisations are possible only because we took a holistic view to system design. If, for example, we allowed globals from other compartments to be transparently accessed then this would have required a GOT approach from the start and would also have made reasoning about communication between compartments much harder. Instead, we have an explicit notion of a pre-shared object, which must come via a compartment’s import table and so is exposed to auditing. By making the import explicit, we can also go beyond core C features and make the permissions explicit, so we can see by reading the source code, and confirm with cheriot-audit that exactly what permissions a compartment has on a shared object.

Decades of prior work have shown us that security policies divorced from the source code are hard to reason about and often get out of sync with the implementation. When they get out of sync, people notice when components have too few permissions (they stop working) but not when they have too many (until an attacker notices).

For CHERIoT, we had an explicit design goal that you should be able to reason about the security of your code by reading the source and do coarser-grained auditing of other people’s code with additional tooling. Whether a function is in your compartment or elsewhere is explicit in the source. Whether a global is uniquely owned by you or a pre-shared object is explicit in the source. The ABI we co-designed with the programmer model to enable this and it lets us emit very short instruction sequences in the common case, while falling back to larger ones to handle corner cases.

Most CHERIoT work to date has been done on software or FPGA simulations. We have several such implementations: The executable model built from our formal ISA specification, the MPact simulator from Google, Microsoft’s CHERIoT SAFE FPGA target for the Arty A7, and of course lowRISC’s beautiful Sonata FPGA board, which is designed to simulate CHERIoT systems. These were always intended to be developing and prototyping systems, so I’m delighted to announce that SCI Semiconductor has the first silicon CHERIoT implementation.

[ Conflict disclaimer: I am a co-founder of SCI Semiconductor. ]

The dev board pictured above contains one of the first batch of ICENI chips to come back from the fab. This is a complete CHERIoT system, with all of the core CHERI properties (spatial memory safety, no pointer injection, and so on) along with all of the CHERIoT extensions that provide deterministic use-after-free protection, auditable control over interrupt state, and everything that we need for an aggressively compartmentalised RTOS.

This chip uses the CHERIoT Ibex core, running at up to 250 MHz, and includes a few feature that accelerate temporal safety, improve interrupt determinism, and so on. These build on top of all of the benefits of any CHERIoT implementation: deterministic mitigation of memory safety bugs from simple buffer overflows up to use-after-free, fine-grained compartmentalisation, and a programming model co-designed with both the ISA and the software stack to provide a tiny TCB. Anything that works on CHERIoT SAFE or Sonata should be very easy to port to ICENI for production use. Anything that runs on the software simulators should just work.

We’ll be showing the chips at Embedded World (Stand 4A - 131) next week and at CHERI Blossoms a couple of weeks later. From tomorrow, one will also be on display in the CHERI 15th anniversary exhibit in the Cambridge Computer Laboratory.

Aside: The Iceni tribe were one of the pre-Roman tribes in Britain and are famous for their chariots (though more due to this statue than historical fact). I am only partially to blame for the bad puns in the naming.

The top three (CWEs 79, 89, and 352) are not typically applicable on embedded platforms. Two are cross-site things that apply to web applications, one is SQL injection.

At position 4, we have CWE-862, missing authorisation. This is not something that’s automatically mitigated by CHERIoT, but the design of CHERIoT RTOS and the programming model that we expose makes it easy to write code that avoids this kind of issue. The CHERIoT pattern for any operation that you do on behalf of another compartment is to require an authorising capability. For example, if you allocate memory, you must present an allocation capability that encapsulates your right to allocate memory (and your quota). If you want to create a socket, you must present a capability that authorises you to bind to a specific port (for server sockets) or connect to a specific remote host. The same applies for dynamically created things, such as sockets themselves, message-queue endpoints, and so on. If you forget to authorise something, it will not have the capability to perform the action and the action will fail. This is a general property of capability systems and not something specific to CHERIoT.

Position 5 is CWE-787, out-of-bounds write, also known as a buffer overflow. This one is deterministically mitigated by any CHERI platform.

Technically, CHERIoT is not vulnerable to the path-traversal bugs in position 6 (CWE-22), but only because we don’t yet ship a filesystem. But, again, this kind of issue is a solved problem in capability systems. Capsicum, for example, eliminates this kind of vulnerability and I’d expect our filesystem APIs to follow a similar shape. There’s no excuse for writing APIs that are vulnerable to path traversal in the 2020s.

The next two are good old-fashioned memory-safety vulnerabilities. Position 7 is use after free (CWE-416), and 8 is out-of-bounds read (CWE-125). The latter is mitigated by any CHERI platform. The former is usually made unexploitable by CHERI systems, and is deterministically mitigated by CHERIoT.

The next two are unlikely to apply to embedded platforms. CWE-78 at position 9 is largely to do with failing to validate dynamically created command lines that you pass to a shell. Then CWE-94 (Improper Control of Generation of Code) at position 10 is typically introduced with scripting languages producing output that can be influenced by an attacker and is then executed by another interpreter, a rare situation on embedded devices.

Position 11 (CWE-120) is a ‘Classic Buffer Overflow’, i.e. something that CHERI deterministically mitigates. Not to be confused with the buffer overflow we had at position 5.

The 12th entry is another that’s rare on embedded devices. CWE-434 relates to unrestricted uploads of dangerous file types, something that matters a lot to web apps and far less to other classes of program.

Next, position 13, is a null-pointer dereference where a valid pointer was expected (CWE-476). CHERI guarantees that this will trap (even if an attacker can provide arbitrary offsets to the null pointer) and CHERIoT makes this a recoverable error either via our scoped error handlers or by simply unwinding the compartment to the caller.

Buffer overflows seem to be popular and position 14 is the third instance of this kind to make the list, this time on the stack (CWE-121). Again, this will deterministically trap on any CHERI platform.

The next entry is more interesting. Unsafe deserialisation of untrusted data (CWE-502) is something a lot of people get wrong. Phil Day wrote about how to do this safely a couple of years ago. Lightweight compartmentalisation makes it easy to limit the scope of damage that this kind of bug can do, to almost nothing.

Did I mention that buffer overflows are a recurring theme on this list? Position 16 (CWE-122) is yet another buffer overflow, this time on the heap. One more that any CHERI platform deterministically mitigates.

Positions 19–21 all relate to incorrect access control at the application layer and, sadly, are not mitigated by CHERI. Position 24 is similar.

In between these, we have another web app problem (CWE-918, server-side request forgery) and another command injection (CWE-77). These are unlikely to be present on embedded devices.

Finally, at position 25, we have a fairly broad category of availability issues that arise from not constraining resource allocation (CWE-770). These are normally mitigated by the software capability layer on CHERIoT. For example, a compartment can’t allocate memory unless it has a capability that authorises it to do so. That capability encapsulates a quota and so provides a limit to the total amount of allocation. Other resources that can be dynamically allocated are normally managed in the same way.

So, what’s the final score card?

Not applicable in embedded contexts: 1, 2, 3, 9, 10, 12, 22, and 23

Deterministically mitigated with just a recompile: 5, 7, 8, 11, 14, and 16.

Mitigated by CHERIoT design patterns and software model: 4, 6, 13, 15, 25.

That still leaves six that we don’t mitigate (17, 18, 19, 20, 21, and 24), but now hopefully the cognitive load is much lower from not having to think about the eleven that we do prevent and you can avoid some of these as well!

What are ML-KEM and ML-DSA for?

I am not a mathematician and so I’m not going to try to explain how these algorithms work, but I am going to explain what they’re for.

Module-Lattice-Based Key-Encapsulation Mechanism (ML-KEM) is, as the name suggests, an algorithm for key encapsulation. One side holds a public key and uses it (plus some entropy source) to generate a secret in both plain and encapsulated forms. The encapsulated secret can be sent to a remote party who holds the corresponding private key. The receiver can then recover unencrypted version of the secret (and detect tampering). Now, both parties have the same secret and can use it with some key-derivation function to produce something like an AES key for future communication.

Note that this is somewhat more restrictive than traditional key-exchange protocols. You don’t get to exchange an arbitrary value, the generation step is part of encapsulation. This also means that it’s a fixed size, defined by the algorithm, which is why you typically feed it into a key-derivation function rather than using it directly.

Module-Lattice Digital Signature Algorithm (ML-DSA) has a similarly informative name. It is intended for providing and validating digital signatures. It takes a private key, an arbitrary-sized document and context, and produces a signature. A holder of the associated public key can then validate that the document matches the version signed with the private key and context.

These are both quite low-level building blocks for higher-level protocols. For example, TLS can use ML-KEM for key exchange and ML-DSA for certificate validation, but also incorporates traditional algorithms in case the PQC algorithms have unexpected weaknesses against classical computers.

Initial porting

As is usually the case for CHERIoT, porting the C implementations of ML-KEM and ML-DSA required no code changes. I worked with upstream to slightly simplify the platform-integration layer, so we just provide a single header describing the port. For example, the port header for ML-DSA configures the build to produce ML-DSA44 support, defines a custom function for zeroing memory and getting entropy, and adds the __cheriot_libcall attribute to the all exported APIs (so we can build them as shared libraries, rather than embedded in a single compartment). The file for ML-KEM is almost identical.

With these defined, it is possible to build both libraries as CHERIoT shared libraries. This motivated a bit of cleanup. We have a device interface for entropy sources, but it wasn’t implemented on the Sail model (which doesn’t have an entropy source). It has a way of exposing the fact that entropy is insecure, so that wasn’t a problem, it just needed doing, so I refactored all of the insecure entropy-source drivers to use a common base. Most encryption algorithms want an API that fills a buffer with entropy. It’s nice if these don’t all need to touch the driver directly, so I created a compartment that provides this API and exposes it. Now, both libraries are simply consumers of this API. This also makes it easier to add stateful whitening for entropy drivers for hardware entropy sources that don’t do the whitening in hardware.

Most CHERIoT stacks are on the order of 1-2 KiBs. The PQC algorithms use much more space. More, in fact, than we permitted.

The previous limitation was based on the precision of bounds rounding. A CHERI capability compresses the bounds representation by taking advantage of the fact that, for a pointer to an allocation, there is a lot of redundancy between the address of the pointer, the address of the end of the allocation (the top), and the address of the start of the allocation (the base). The distance from the address to base and top are stored as floating-point values with a shared exponent. In practical terms, this means that the larger an allocation is, the more strongly aligned its start and end addresses must be. The same restrictions apply for any capability that grants access to less than an entire object.

When you call a function in another compartment, the switcher will truncate the stack capability so that the callee sees only the bit of the stack that you weren’t using. The top and base of the stack must be 16-byte aligned (as an ABI requirement), but a very large stack may have hardware requirements for greater alignment and so may require a gap between the bottom of the caller’s stack and the top of the callee’s.

Fortunately, we’d added an instruction precisely for this kind of use case: CSetBoundsRoundDown. This takes a capability and a length and truncates it to at most that length. It was a fairly small tweak to the switcher to make it do this, and a much larger amount of time with SMT solvers to convince ourselves that this was a safe thing to do.

This also showed up a bug in our linker’s handling of the CAPALIGN directive, which rounds a section’s base and size up to the required alignment to be representable. This was not working for sections that followed an explicit alignment directive. Our stacks must be both at least 16-byte aligned and representable as capabilities. This is now fixed.

So now we support stacks up to almost 64 KiB, a limitation imposed by the current loader metadata format rather than anything intrinsic to how the system operates after booting. We could easily increase this limit but 64 KiB ought to be enough for anyone.

Performance on CHERIoT Ibex

The repository contains a simple benchmark example that tries each of the operations and reports both the cycle time and stack usage. The output on the CHERIoT Ibex verilator simulation is:

PQC benchmark: Starting: stack used: 224 bytes, cycles elapsed: 41
PQC benchmark: Generated ML-KEM key pair: stack used: 14304 bytes, cycles elapsed: 5143987
PQC benchmark: Encrypted secret pair with ML-KEM: stack used: 17440 bytes, cycles elapsed: 1773235
PQC benchmark: Decrypted secret pair with ML-KEM: stack used: 18464 bytes, cycles elapsed: 2176226
PQC benchmark: Compared results successfully for ML-KEM: stack used: 224 bytes, cycles elapsed: 414
PQC benchmark: Generated ML-DSA key pair: stack used: 46912 bytes, cycles elapsed: 3622132
PQC benchmark: Signed message with ML-DSA: stack used: 60544 bytes, cycles elapsed: 5391177
PQC benchmark: Verified message signature with ML-DSA: stack used: 44672 bytes, cycles elapsed: 3674071
PQC benchmark: Correctly failed to verify message signature with ML-DSA after tampering: stack used: 44672 bytes, cycles elapsed: 3673706

The ML-KEM encrypt (generate shared secret and encrypted version) and decrypt (recover shared secret from encrypted version) each use around 18 KiB of stack and run in around two million cycles. CHERIoT Ibex should scale up to 200-300 MHz (though may be clocked lower for power reasons in some deployments), but even at 100 MHz that’s 50 encryption or decryption operations per second. Remember that this is an operation that typically happens when you establish a connection, then you use a stream cypher such as AES with the exchanged key.

The ML-DSA operations are slower and use a lot more stack space (almost 60 KiB for signing!). But, even there, the performance is reasonable, under 4 M cycles. This means that you can do 20 signature-verification operations per second at 100 MHz.

Even using ML-KEM for key exchange and ML-DSA for certificate validation in a TLS flow is unlikely to add more than a few tens of milliseconds to the handshake time, which is perfectly acceptable for the common use case for embedded devices.

In terms of code size, both are small. The ML-KEM implementation is around 12 KiB, the ML-DSA implementation 18 KiB. These both include a SHA3 (FIPS 202) implementation, so there’s scope for code-size reduction on systems that need both, but 30 KiB of code isn’t too bad.

Future plans

The stack usage is very high. Upstream has some plans to allow pluggable allocators, which will allow us to move a lot of this to the heap. This is precisely the kind of use case that CHERIoT’s memory-safe heap is great for: something needs 60 KiB of RAM for 4,000,000 cycles, but then doesn’t need that RAM again for a long time. That memory can then be used for something else, even in a mutually distrusting compartment.

Currently, the library builds are very thin wrappers around the upstream projects. This is great as a building block, but we should make more use of CHERIoT features in the longer term.

Both ML-KEM and ML-DSA depend on SHA3 (FIPS 202). Ideally, we’d factor that out as some common code, rather than carrying a copy in each library. Similarly, the libraries provide an option to plug in your own SHA3 implementation. This is likely to be a common hardware operation even for chips that don’t have full PQC implementations, so we should expose this option in the build system.

Is it secure?

Security always depends on the threat model.

For signature validation, you don’t have any secret data, just a public key, a document, and a signature. The only concerns are whether there are weaknesses in the algorithm, or bugs, that would allow an attacker to substitute a different document for the same signature. CHERIoT prevents memory-safety bugs, so this is concerned solely with logic errors. The code upstream is checked against a set of test vectors that aim to trigger corner cases in the logic of the underlying implementation, so hopefully is secure in this way.

For signing or key exchange, you need to worry about the key leaking. On a CHERI system, it’s unlikely to leak explicitly, but may leak via side channels. The security section of the upstream projects discusses a number of techniques that they use to mitigate this kind of attack.

That’s typically sufficient. It’s been recommended practice for embedded devices to have per-device secrets for a long time. This means that leaking a key from one device doesn’t compromise the device class, only that specific device.

For some very high-assurance use cases, that secret may matter and need to be robust against an adversary with physical access to the device. Hardware encryption engines typically care about confidentiality breaches via power side channels and integrity breaches via glitch injection. Power side channels are difficult to mitigate in software: the power requirements of multiplying two numbers together may depend on the number of carry bits set, for example. They’re much easier to mitigate in hardware, by simply doing the same calculation twice in parallel, once with the original inputs and once with the inputs permuted to have the opposite power characteristics.

Glitch injection takes the chip out of its specified power or frequency (or thermal) envelope and attempts to introduce bit flips, which can corrupt state in such a way that tamper with signing or leak a key. These are also effectively impossible to mitigate in software because the software that’s attempting the mitigation is vulnerable to the same glitches. There are some compiler techniques that can make these harder, but they come with a high performance cost.

If power analysis and glitch injection are part of your threat model, the software implementations are not sufficient. In this case you may also need to worry about someone removing the top of the chip and using a scanning-tunnelling electron microscope to read bits from non-volatile memory. This used to require tens of thousands of dollars but is now much cheaper. Devices that need to worry about this often have tiny explosive charges in the package to destroy the chip in cases of tampering. If that’s your threat model, hardware PQC implementations may not be sufficient, at least alone.

But if you care about attackers on the network being unable to compromise the security of the class of devices, even if they have a magical and imaginary quantum computer, then these should be sufficient.

TockOS is implemented in Rust and has safety as a priority. I generally regard it as the gold standard for an RTOS that is forced to operate under the constraints of working with existing hardware. But note where the drivers are: In the kernel, right at the bottom.

Here’s a similar diagram from the Zephyr RTOS:

Here, there is some separation between drivers and the core of the kernel, but the default configuration runs with no privilege separation. Indeed, the Zephyr Security Overview says:

The security architecture is based on a monolithic design where the Zephyr kernel and all applications are compiled into a single static binary. System calls are implemented as function calls without requiring context switches.

In fact, the only exception I’ve seen to this recently is LionsOS, a multiserver system built on top of the formally verified seL4 microkernel:

This runs all of the device drivers in unprivileged contexts. Unfortunately, seL4 assumes an MMU and so is not feasible for small embedded devices (seL4 uses more memory to hold page tables than a lot of CHERIoT firmware images use in total).

Drivers are attack surface

Device drivers, at least for I/O devices, are code that interacts with the outside world. An attacker trying to compromise a device has a much easier job if they don’t need to take the chip apart. The easiest attacks to mount are ones that work from an across the network. The next easiest are ones that drive local I/O, which may be reachable remotely via other paths.

Drivers for I/O devices, by definition, make decisions based on the values that they read from the device. These operations include mapping error codes to some type-safe enumeration (which can fail if the hardware is buggy), or using the values to index into other structures. These are hard to get right, even in a memory-safe language, because they often sit below the language’s abstraction layer. Microsoft estimates that 70% of Windows crashes are caused by bugs in device drivers.

Any bug in a driver for an I/O device is a useful building block for an attacker. This problem is made much worse if the device driver is in a privileged component. For example, at the end of last month there were three bluetooth CVEs in Zephyr that all could lead to compromise and, if the Bluetooth stack is not privilege separated, can lead to arbitrary code execution by an attacker who gets within a few meters of the device (or compromises another Bluetooth-enabled device nearby).

Device drivers do abstraction and multiplexing

This design results from conflating the two functions of a device driver.

A device driver has to provide an abstraction over a particular device. Sometimes this happens in multiple layers. For example, an Ethernet device may have an abstraction for sending and receiving Ethernet frames, but this is then the foundation for a further abstraction layer for sending IP packets, which is then used to expose TCP streams and UDP datagrams.

A device driver often also has to provide some secure multiplexing. For example, two mutually distrusting components may be allowed to create sockets for different TCP connections that flow over the same Ethernet device. Or they may be allowed to talk to two different USB bus endpoints via the same USB controller.

The first of these requirements is a software engineering problem. The second is primarily a security problem, but typically the multiplexed abstractions need to be device independent and so it’s also a software-engineering problem.

In embedded development, there’s often a distinction between a hardware-abstraction layer (HAL) and a driver, with the former providing only the abstraction and the latter also providing multiplexing. This is a useful distinction because, in a lot of cases, embedded systems have a single consumer for a device. For example, you may have multiple SPI or I²C interfaces on a device, but each one is used for a single purpose. It is convenient to be able to write software to talk to a SPI device without having to know exactly which SPI controller this chip uses, but you don’t need to handle safely sharing those pins with other components.

CHERIoT RTOS distrusts drivers

In CHERIoT RTOS, the core platform provides device abstractions that meet the earlier definition of a HAL: they provide abstractions over classes of device, but do not attempt to provide security. The RTOS also provides a trivial way of auditing which compartments can access which devices, so that you can ensure that devices are not accessible to compartments that are not trusted to interface with them.

The platform’s device code runs within whatever compartment you instantiate it in. It has no elevated privileges except the MMIO region(s) that you explicitly pass it for talking to a particular device.

This makes it easy to support both bespoke and reusable security models. If you need to share a device between two compartments with some secure multiplexing based on a custom policy, you can do that by instantiating the driver in a compartment and exposing APIs to the two others.

Sometimes, the desired abstractions are reusable. For example, the CHERIoT network stack is assembled out of the following compartments:

graph TD
  Network
  subgraph Firewall["On-device firewall"]
    DeviceDriver["Device Driver"]
  end
  TCPIP["TCP/IP"]:::ThirdParty
  User["User Code "]
  NetAPI["Network API"]
  DNS["DNS Resolver"]
  SNTP:::ThirdParty
  TLS:::ThirdParty
  MQTT:::ThirdParty
  DeviceDriver <-- "Network traffic" --> Network
  TCPIP <-- "Send and receive Ethernet frames" --> Firewall
  DNS <-- "Send and receive Ethernet frames" --> Firewall
  NetAPI -- "Perform DNS lookups" --> DNS
  NetAPI -- "Add and remove rules" --> Firewall
  TLS -- "Request network connections" --> NetAPI
  TLS -- "Send and receive" --> TCPIP
  NetAPI -- "Create connections and perform DNS requests" --> TCPIP
  MQTT -- "Create TLS connections and exchange data" --> TLS
  User -- "Create connections to MQTT server and publish / subscribe" --> MQTT
  MQTT -- "Callbacks for acknowledgements and subscription notifications" --> User
  SNTP -- "Create UDP socket, authorise endpoints" --> NetAPI
  SNTP -- "Send and receive SNTP (UDP) packets" --> TCPIP
  TLS -- "Request wall-clock time for certificate checks" --> SNTP
  style User fill: #5b5
  classDef ThirdParty fill: #e44

Note that the driver for the Ethernet device is instantiated in the firewall compartment. What happens if an attacker gets arbitrary-code execution here? They could mount a denial of service attack (refuse to forward Ethernet frames in or out). They could tamper with Ethernet frames.

This sounds bad but the rest of a network stack already has to assume that things like this can happen. Packets coming over the network are intrinsically untrusted. The TCP/IP stack has to assume that they may be malicious. It isn’t always good at this. The FreeRTOS TCP/IP stack that we use has had 15 CVEs disclosed since it was released, but our compartmentalisation strategy mitigates all of them. By placing the parts of the system that are exposed to an attacker in the least, not most, trusted places, we make it easy to build secure systems.

The funding was awarded back in September, but the embargo was lifted last week and so we can talk about it publicly.

Rust and CHERIoT have complementary benefits

I’ve written previously about why safe languages and CHERI are complementary. Superficially, both Rust and CHERI provide similar benefits in terms of memory safety. That similarity goes away when you look at the details.

Rust provides a very rich set of type-system guarantees. If you are writing software, you can use Rust’s type system to enforce a range of properties that go beyond simple memory safety. The key part here is ‘if you are writing software’.

The industry learned some important lessons from Java and JavaScript attempts at language-level sandboxing: it’s very hard to write a compiler that assumes that the programmer is an adversary. Any soundness issue in the underlying type system or any bug in the compiler can be a security vulnerability if you assume that the person writing the software is malicious and actively trying to break the guarantees that the language aims to enforce.

The Rust compiler is currently tracking 107 bugs marked as soundness issues. A typical Rust programmer is unlikely to encounter these. Encountering these bugs typically require poking at corner cases of the language that you’re unlikely to hit by accident. In contrast, a malicious programmer wanting to insert a supply-chain vulnerability into something that you consume has a rich set of tools.

The CHERIoT compartmentalisation was designed with this kind of adversary in mind. It assumes that you may be incorporating arbitrarily buggy or malicious code into a device’s firmware and need to be able to protect against components that are compromised. The checks that CHERIoT does at run time are less rich than those that Rust enforces at compile time, but are not bypassable, even with unsafe code or inline assembly.

This means that you can use Rust’s type system to ensure that your code has strong confidentiality, integrity, and availability guarantees, while simultaneously using CHERIoT to ensure that supply-chain code cannot violate these guarantees (at least with respect to confidentiality and integrity, though availability is a bit harder). Rust’s guarantees make achieving high levels of confidence in your code much easier than if you used C/C++. Rust is one of the few languages that delivers this kind of guarantee and is also able to run on small embedded devices (such as CHERIoT implementations), making it a very exciting choice for future CHERIoT development.

CHERI and Rust give you the benefits of Rust faster

Rust gives rich guarantees to Rust code. Most software is not written in a single language, and especially not in a relatively young language. What if you have some legacy C/C++ component? You can use it from Rust, but in most systems this means bugs in the legacy code can completely undermine the security guarantees of the Rust code. A memory-safety bug in C code can corrupt any Rust object in the same process.

On a CHERI system, you get different levels of guarantee depending on how you mix languages. Calling C code from Rust within the same compartment requires the C code to follow the CHERI rules. If Rust code passes a pointer to C, the C code may tamper with objects reachable from there, but can’t arbitrarily affect the system unless it has a much narrower set of bugs. Stack uninitialised use bugs may cause C to access objects left on the stack, for example.

If you put the C code in another compartment, these guarantees become a lot stronger. C code can tamper with objects passed as arguments (or objects reachable from them) but has no access to the Rust compartment’s stack or globals. This makes it much easier to reason about the impact of bugs in C code when adopting Rust.

You can take this even further and restrict the permissions on pointers passed from Rust to C. Do you want C code to be able to read a Rust object or object graph? Or perhaps modify an object but not capture a pointer to it (similar to borrow semantics)? The hardware can enforce these properties.

This lets you get all of the great benefits of Rust from the very first function that you write in Rust, rather than having to wait until you’ve rewritten everything in Rust.

What are we trying to do?

Initially, we aim to make Rust work as a source language for targeting CHERIoT. CHERIoT is an embedded platform, so the no_std + alloc mode for Rust (you can dynamically allocate memory, but can’t use most of the standard library) makes sense as a target. This will make it easy to port existing embedded Rust code to CHERIoT and to write new Rust components. As the project runs, there will be a lot of quality-of-implementation work to ensure that it’s in a state to upstream and, until then, in a state where we can support it.

The next step in the project is to make sure that Rust is a first-class citizen of the CHERIoT platform. We have a set of C/C++ language extensions to provide rich compartmentalisation features. Rust will need equivalents of these. A direct port of the features is quite easy and works as a minimum viable product, but we’d also like to make sure that these work as idiomatic Rust: you shouldn’t have to write C-like Rust to use CHERIoT features.

Finally, there are several things in the Rust type system that can be dynamically enforced in CHERIoT. In current Rust, every call to non-Rust code must be unsafe. I hope that we’ll be able to relax this requirement and have the compiler enforce Rust properties by removing permissions from pointers before calling C.

There are a lot of subtleties here. The authors of Tock discovered that exposing Rust to untrusted code has some issues and provided solutions that worked in their specific context. I’m optimistic that the richer substrate of the CHERIoT ISA gives the compiler more tools to provide generic solutions to these issues.

The funded project is specifically scoped to CHERIoT, but we’re building on work from Morello and hope to make it easy to support other CHERI platforms.

Rust enables verification

The Verus project builds language-integrated formal verification tools in Rust. This is particularly interesting for core parts of CHERIoT RTOS. The lowest level parts of an operating system are tricky for safer systems languages because the things that they do are intrinsically unsafe. They must be able to use escape hatches that opt out of parts of the safety guarantees of a language like Rust because they are the things that implement those guarantees.

You can think of rich type systems as off-the-shelf verification tools (they define some generic properties and prove them for every program, and reject those for which they can’t prove them), whereas the lowest-level parts of systems code need bespoke verification to prove one-off sets of properties. There was some great work at SOSP this year showing how Verus can prove key aspects of a kernel and I’m excited to see how far we can get with this in CHERIoT.

Where will this project live?

The CHERIoT Rust project has its own web site which is where we’ll post more public information. The compiler repository is public, but is not yet ready for general use. We’ll post calls for testing when it reaches an early preview state. As with any other aspect of CHERIoT, contributions are welcome. Please join our public Signal chat if you want to discuss the project or help out!

CHERI is an abstract architecture

CHERI is a conceptual extension to conventional computing architectures define a capability model for accessing memory within an address space. There are a lot of concrete instantiations of this conceptual model. The original CHERI research was done on a 64-bit MIPS variant. Arm Morello extended ARMv8 with CHERI extensions. The University of Cambridge CHERI ISA v9 Technical Report contains a RISC-V CHERI adaptations and even a sketch of CHERI for x86. And there is an ongoing effort to standardise RISC-V CHERI base architectures.

These all provide the same core set of features. Most software isn’t written for a single target instruction-set architecture (ISA), it’s written assuming a set of language-level abstractions that can be represented in a particular target. Software written for CHERI in C (or any language that is higher-level than C) doesn’t care the bit patterns of capabilities, how bounds or permissions are encoded, and so on. It cares only that you have some capability registers, that you can store capabilities in memory, and that the C language notion of a pointer can be represented by a CHERI capability for the target ISA.

I like to use memory-management units (MMUs) as an analogy for CHERI. MIPS and SPARC had software-managed translation-lookaside buffers (TLBs). x86 and Arm had architectural page tables that hardware walked to fill the TLB automatically on miss. PowerPC had an MMU design that, in the words of Linus Torvalds, ‘can be used to frighten small children’. All of these implemented the same set of abstractions in terms of processes, shared memory, copy-on-write regions, and so on. All of these are MMU designs, just as there are many possible CHERI implementations.

There are lots of things like this in ISAs. Arm’s Neon, PowerPC’s AltiVec, and x86 SSE are all SIMD extensions, which provide similar programming models, but they all have quite different low-level details. It’s quite possible to write code that targets all of them.

CHERIoT is a concrete platform

CHERIoT is a complete ISA specification, co-designed with a software stack. In the same way that x86 page tables or MIPS’ software-managed TLB are both concrete instances of the abstract idea of a memory-management unit, CHERIoT is a concrete instantiation of CHERI.

This means that there is no ‘CHERI or CHERIoT’ choice. The real choices are between CHERI or not CHERI, or between CHERIoT and some other concrete instantiation of CHERI. Having worked on CHERI since 2012, I am obviously biased on the first of these: given a choice between CHERI and not-CHERI, you should definitely pick CHERI! The second choice (CHERIoT or some other CHERI variant) is more nuanced, as I’ll discuss more in the rest of this post. There are good reasons for CHERIoT but there are also domains where it is not the best choice.

CHERIoT is a rich CHERI platform

A minimal CHERI specification provides pointer integrity, bounds enforcement, and some permissions. Pointer integrity means (among other things) that you can precisely differentiate between pointers and other kinds of non-pointer data in memory and so can build temporal safety on top. Most of the CHERI temporal safety work has provided users with protection against use-after-reallocation issues. A pointer in a CHERI system may point to deallocated memory, but it will not point to reallocated memory. Something in the system will ensure that deallocated pointers are gone before the memory allocator reuses memory. This means that any use-after-free bug will either point to the old object, or will trap. It will never point to a new object and cause accidental aliasing.

CHERIoT builds on top of the core CHERI ideas with a couple of hardware features that guarantee deterministic trapping for use-after-free bugs. CHERIoT also provides a rich sealing design that works with our software model. It extends the sentry mechanism in CHERI systems to provide stronger control-flow integrity properties than most other CHERI platforms, and also uses them to enforce interrupt control.

The entire system is designed to allow fine-grained auditing of compartment rights, which lets us respect the principle of least privilege by having compartments as small as a single function and statically reasoning about what they can do in the presence of a compromise. This set of building blocks lets us do things like privilege separate the network stack and make callers respect the principle of intentional use, again with auditable guarantees that let you reason about which compartments can talk to which remote servers.

This adds up to an easy-to-use programmer model and richer security guarantees than, we believe, any other system (CHERI or otherwise).

CHERIoT is designed for resource-constrained systems

Some of the strengths of CHERIoT come from the fact that we were willing to tailor the system to small devices. Prior CHERI systems rely on the MMU to support concurrent revocation as their mechanism for temporal memory safety. CHERIoT does not include or need an MMU, it uses an alternative approach for temporal safety, inspired by our previous CHERI+MTE work

You don’t want an MMU on an embedded device. An MMU is typically larger than an entire microcontroller core! Page tables are large data structures and they enforce a minimum granule size for memory protection and accounting. A minimal process isolated with a typical MMU needs one page for the stack, one page of code, and one for read-write globals. On top of that, it needs a complete page table, which is at least two pages. This means you need a minimum of 20 KiB for an isolated component, of which 8 KiB is purely book-keeping overhead. On a CHERIoT system, a lot of our compartments are smaller than the page tables that would be necessary to isolate them. In addition, MMUs bring nondeterminism, which is not ideal in any system with even fairly soft realtime requirements.

The temporal memory system in CHERIoT is designed to scale across the range of microcontroller cores, including low-core-count multicore systems. We check whether capabilities are valid when you load them. Most pointers are used more than once, so this is more efficient than checking on every use (as an MTE system must do), but does not scale as well to large multicore systems or complex memory hierarchies. You would not want to build a temporal-safety system like ours for a large server system, for example.

CHERIoT also extends CHERI’s sealed function pointer (“sentry”) mechanism for interrupt control. This has some enormous benefits for the programmer model. It is quite easy to implement on microcontroller-class systems, even dual-issue designs such as CHERIoT-Kudu. It would be very hard to scale to big out-of-order cores.

And that’s all fine, for the same reason it’s fine to run different operating systems on different scales of core: different abstractions and different optimisations make sense at different scales. By the time you’ve built a system where you start to run into scalability challenges with CHERIoT, you have enough area that an MMU doesn’t add much overhead and enough RAM that you would benefit from running an OS designed for larger computers.

What about standard RISC-V CHERI?

The RISC-V standardisation process has an unenviable task of trying to create a standard base for RISC-V CHERI systems that will work for 2-stage microcontrollers, 25-stage superscalar out-of-order server chips, GPUs, AI accelerators, SmartNICs, and so on. This is likely to end up being a (hopefully small) family of bases and a set of extensions.

If all goes well, CHERIoT v2 will be one of those bases plus some extensions. RVB26 will be a RISC-V profile assembled from those bases and a set of extensions necessary to run a CHERI Linux or CheriBSD.

As with the rest of RISC-V, any useful chip implements a base architecture and a set of extensions. Profiles exist to corral a set of extensions that software can depend on in environments where binary compatibility is important, such as those running off-the-shelf operating systems and programs.

So when should I use CHERIoT vs some other CHERI?

If you are looking for a microcontroller-class system, CHERIoT is probably the right answer. It is co-designed with a compartmentalisation model and a rich set of software abstractions (implemented in CHERIoT RTOS), and provides temporal safety as a baseline feature. It is a mature and stable target, supported by multiple organisations.

If you are looking for an application core, CHERIoT is not for you. There are a few cases where people have moved to using Linux on an Arm A-profile system and could run the same workload on a cheaper CHERIoT system, but don’t expect to be able to run arbitrary Linux workloads on a CHERIoT system (ever). The upcoming CHERI RISC-V profile is a far better choice for these use cases. Capabilities Limited and lowRISC are implementing this in the open-source CVA6 core, so there will soon be a path to a useful open-source CHERI core in this part of the design space. Codasip’s X730 core is also available for commercial licenses and is targeting this spec (and pre-standard versions of it until it is ratified). Hopefully there will be more implementations available in coming years.

If you are looking for an ISA as a base for an accelerator or coprocessor, CHERIoT may be the right choice, depending on the rest of the system. Depending on your requirements, you may want to do something completely custom. As long as it provides the same underlying security guarantees, it can still be CHERI!