Motivation

C and C++ are low-level languages, nominally especially well suited for working with hardware and in performance-sensitive data processing applications, and yet, they lack convenient, portable mechanism for working with bit-packed structures, as often found in hardware memory-mapped I/O (MMIO) registers, network packets, and data serialization formats. By “bit-packed”, we mean that the logical data elements are not necessarily aligned to “natural” boundaries, as defined by the machine processing those data (for example, byte / octet or machine word boundaries), and/or may not fully span between two such “natural” boundaries (for example, a 5-bit field on an 8-bit byte machine). The primary, syntactic mechanism that these languages do offer for sub-word field management is called “bit-fields”, a provision to specify, in C++ jargon, the bit width of of a data member with integral or enumeration type. Concretely, one writes something like

struct MyFields
{
	bool enable : 1;
	enum MyEnum reference : 2;
	unsigned int bias : 8;
};

to specify a structure holding an 1-bit, bool-typed field named enable, a 2-bit enum MyEnum-typed reference, and an 8-bit, unsigned int-typed bias.

Unfortunately, the layout (that is, the in-machine-memory representation) of such definitions is almost completely unspecified by the language standards. C++23 (well, N4928) says:

Allocation of bit-fields within a class object is implementation defined. Alignment of bit-fields is implementation defined. Bit-fields are packed into some addressible allocation unit.

[Note 1: Bit-fields straddle allocation units on some machines and not on others. Bit-fields are assigned right-to-left on some machines, left-to-right on others. – end note]

While there is some provision for forcing alignments of, and controlling padding around, bit-fields, the implementation-defined nature makes even this challenging to actually understand and use.

Moreover, bit fields are very much second class citizens:

• they may not have their address taken (so there are no pointers to bit-fields; what would be the type of such a thing?);

• references to bit-fields must be const (allowing implementations to quietly convert the reference to one of the bit-field’s underlying type);

• because bit-fields exist within aggregate structures that are not themselves numeric types, converting between a bit-field aggregate and a numeric representation can be particularly troublesome, especially in constexpr contexts where reinterpret_cast-s are forbidden.

As a result, implementations of drivers and codecs and such in practice tend to eschew language bit-fields in favor of explicit bit-wise operations on numerically-typed words. They may even go so far as to eschew C/C++ aggregates (struct-s and such) completely and view entire MMIO regions as simply arrays of uint32_t-s, with information about the word and sub-word fields’ layout conveyed “out of band” via, for example, preprocessor #define-s.

Given the near ubiquity of dealing with bit-packed structures in embedded contexts, we wanted something better. Thus, on an experimental basis, we have introduced “bitpacks” to the CHERIoT-RTOS programming framework. Bitpacks are, at their core, simply machinery and syntax wrapping bitwise operators (shift, complement, AND, OR) on an underlying value of integral type (say, uint32_t), making it much more pleasant to view (and write code that views) that underlying value as containing sub-fields.

The remainder of this post gives examples of that machinery (and of different ways of thinking about, naming, and typing sub-fields) built around a simple, fictitious, but representative, MMIO device.

A Simple, Fictitious MMIO Device

Imagine a simple peripheral that generates data and has some configuration knobs about how it does so, something like a very rough sketch of an analog-to-digital converter. Let’s say that our device has

• a 32-bit control register with…

  • a control bit that enables or disables the device’s operation, which must be disabled to configure the peripheral. The mnemonics for this field’s values are NoGo and Go.

  • a status bit that indicates whether the device is enabled and active. The mnemonics for this field’s values are Off and Running.

  • a choice of “reference”, with one internal and two external sources, creatively named “A” and “B”, giving us an enumeration with mnemonics Internal, ExternalA, and ExternalB.

  • an 8-bit “bias” value. This field is more numeric than an enumeration, so has no associated mnemonics.

• a 32-bit data register that can be read once the device is active.

Graphically, we might render that MMIO layout something like this:

Code Without Bitpacks

An Array of uint32_t Is All You Need?

A minimal, easily generated software description of such a MMIO device might look like this:

#define MYDEVICE_OFFSET32_CONTROL 0

#define MYDEVICE_CONTROL_ENABLE_SHIFT 0
#define MYDEVICE_CONTROL_ENABLE_MASK 0x1
#define MYDEVICE_CONTROL_ENABLE_V_NOGO 0x0
#define MYDEVICE_CONTROL_ENABLE_V_GO 0x1

#define MYDEVICE_CONTROL_ACTIVE_SHIFT 1
#define MYDEVICE_CONTROL_ACTIVE_MASK 0x2
#define MYDEVICE_CONTROL_ACTIVE_V_OFF 0x0
#define MYDEVICE_CONTROL_ACTIVE_V_RUNNING 0x1

#define MYDEVICE_CONTROL_REFERENCE_SHIFT 8
#define MYDEVICE_CONTROL_REFERENCE_MASK 0x300
#define MYDEVICE_CONTROL_REFERENCE_V_INTERNAL 0x0
#define MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A 0x1
#define MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_B 0x3

#define MYDEVICE_CONTROL_BIAS_SHIFT 16
#define MYDEVICE_CONTROL_BIAS_MASK 0xFF0000

#define MYDEVICE_OFFSET32_DATA 1

(For a real-world example, the OpenTitan Register Generator tool produces C headers that look much like this.)

These #define-s give

• each register’s offset, in terms of 32 bit words,

• the shift to reach the least significant bit of each field,

• the (shifted, in-situ) mask of each field, and

• an enumeration of values for enumeration-typed fields.

An example function using such a definition to write code to configure the device, wait for it to become active, and then returning the first word of data could look like this, taking a pointer to the device’s MMIO region’s base:

uint32_t test(volatile uint32_t *device)
{
	// set device.control's enable field to NOGO
	device[MYDEVICE_OFFSET32_CONTROL] =
	  (device[MYDEVICE_OFFSET32_CONTROL] & ~MYDEVICE_CONTROL_ENABLE_MASK) |
	  (MYDEVICE_CONTROL_ENABLE_V_NOGO << MYDEVICE_CONTROL_ENABLE_SHIFT);

	/*
	 * Set device.control's reference field to EXTERNAL_A value and set bias to
	 * 0x7f at the same time.  Leave the enable field unchanged.
	 */
	device[MYDEVICE_OFFSET32_CONTROL] =
	  (device[MYDEVICE_OFFSET32_CONTROL] &
	   ~(MYDEVICE_CONTROL_REFERENCE_MASK | MYDEVICE_CONTROL_BIAS_MASK)) |
	  (MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A
	   << MYDEVICE_CONTROL_REFERENCE_SHIFT) |
	  (0x7f << MYDEVICE_CONTROL_BIAS_SHIFT);

	// set device.control's enable field to GO
	device[MYDEVICE_OFFSET32_CONTROL] =
	  (device[MYDEVICE_OFFSET32_CONTROL] & ~MYDEVICE_CONTROL_ENABLE_MASK) |
	  (MYDEVICE_CONTROL_ENABLE_V_GO << MYDEVICE_CONTROL_ENABLE_SHIFT);

	// wait for device.control's active field to be RUNNING
	while ((device[MYDEVICE_OFFSET32_CONTROL] & MYDEVICE_CONTROL_ACTIVE_MASK) !=
	       (MYDEVICE_CONTROL_ACTIVE_V_RUNNING << MYDEVICE_CONTROL_ACTIVE_SHIFT))
	{
		yield();
	}

	return device[MYDEVICE_OFFSET32_DATA];
}

How About a struct of uint32_t-s?

We can use C++ aggregates to at least make the “outer” layer of the MMIO structure, the layout and naming of the registers (but not their fields). Keeping the per-field parts of the above definition, but replacing the register layout information with slightly more idiomatic C++ makes our example look something like this:

struct RawDevice
{
	uint32_t control;

	uint32_t data;
};

// MYDEVICE_CONTROL_... as above

uint32_t test(volatile RawDevice &device)
{
	// set device.control's enable field to NOGO
	device.control =
	  (device.control & ~MYDEVICE_CONTROL_ENABLE_MASK) |
	  (MYDEVICE_CONTROL_ENABLE_V_NOGO << MYDEVICE_CONTROL_ENABLE_SHIFT);

	/*
	 * Set device.control's enable field to GO at the same time set the
	 * reference field to EXTERNAL_A value.
	 */
	device.control = (device.control & ~(MYDEVICE_CONTROL_REFERENCE_MASK |
	                                     MYDEVICE_CONTROL_BIAS_MASK)) |
	                 (MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A
	                  << MYDEVICE_CONTROL_REFERENCE_SHIFT) |
	                 (0x7f << MYDEVICE_CONTROL_BIAS_SHIFT);

	// set device.control's enable field to GO
	device.control =
	  (device.control & ~MYDEVICE_CONTROL_ENABLE_MASK) |
	  (MYDEVICE_CONTROL_ENABLE_V_GO << MYDEVICE_CONTROL_ENABLE_SHIFT);

	// wait for device.control's active field to be RUNNING
	while ((device.control & MYDEVICE_CONTROL_ACTIVE_MASK) !=
	       (MYDEVICE_CONTROL_ACTIVE_V_RUNNING << MYDEVICE_CONTROL_ACTIVE_SHIFT))
	{
		yield();
	}

	return device.data;
}

This compiles to the same code as the test(volatile uint32_t *) implementation above.

(ARM’s CMSIS contains many examples of using struct-s for the layout of register words and #define-s for fields within registers).

C++ Bitfields?

We can try to write our example with C++ bitfields, despite their implementation-defined nature (perhaps we believe that there is and will only ever be exactly one C++ ABI for our machine). That would make the MMIO region definition look something like this:

struct BFDevice
{
	struct Control
	{
		bool enabled : 1;

		bool active : 1;

		uint32_t : 6; // padding

		uint8_t reference : 2;

		uint32_t : 6; // padding

		uint32_t bias : 8;

		uint32_t : 8; // padding
	} control;

	uint32_t data;
};

static_assert(sizeof(BFDevice::Control) == 4);

And in use, we’d run into the fact that C++ does not implicitly define copy operators suitable for use with volatile bitfield aggregates, so we’d resort to doing more writes than the above examples, with

uint32_t test(volatile BFDevice &device)
{
	device.control.enabled = false;

	// challenging to make these into one MMIO write, like we could above
	device.control.reference = MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A;
	device.control.bias      = 0x7f;

	device.control.enabled = true;

	while (!device.control.active)
	{
		yield();
	}

	return device.data;
}

All the challenges aside, that sure looks like nicer code! So, how do bitpacks help?

Introducing Bitpacks

Bitpacks offer a way to define, view, and manipulate typed fields within a numeric word. Internally, fields are simply contiguous spans of bits within that word, and operations are phrased in terms of bitwise operations (shift, complement, AND, OR). Fields are accessed through “proxies”, each of which can be thought of as a reference to a field within a particular word.

One defines a bitpack by defining a class (or struct) that inherits from the Bitpack<typename Storage> class template. The Storage template parameter defines the size (and type) of the underlying numeric word.

Defining a field – specifying its position within the word – amounts to filling out a FieldInfo structure:

struct FieldInfo
{
	/// Minimum 0-indexed bit position occupied by this field
	size_t minIndex;

	/// Maximum 0-indexed bit position occupied by this field
	size_t maxIndex;

	/// Should proxies of this field not provide affordances for mutation?
	bool isConst = false;
};

A FieldInfo {3, 7}, for example, specifies a field spanning bits 3 through 7 of a word. From this, bitpacks can, internally, compute things like the _SHIFT and _MASK values used above. We will almost never encounter FieldInfo-s by name, but we will specify initializers for them when defining fields.

The simplest way to build a proxy of a field, so that we can access it, is to define a method within our bitpack that invokes the protected Bitpack::member<FieldType, FieldInfo> method with appropriate arguments. This is called a “named member” and the C++ syntax involved is encapsulated within the BITPACK_MEMBER_ADD macro, to which we now direct our attention.

Named Members

We can define a bitpack for our device’s control register, and specify a named member for each field within, thus:

struct NuDevice
{
	struct Control : Bitpack<uint32_t>
	{
		BITPACK_USUAL_PREFIX;

		BITPACK_MEMBER_ADD(enable, bool, 0, 0); // NoGo is false / Go is true

		BITPACK_MEMBER_ADD(active, bool, 1, 1, true); // Off / Running

		BITPACK_MEMBER_ADD(reference,
		                   uint8_t,
		                   8,
		                   9); // values from MYDEVICE_CONTROL_REFERENCE_V_...

		BITPACK_MEMBER_ADD(bias, uint8_t, 16, 23);
	} control;

	uint32_t data;
};

As discussed, struct Control now inherits from Bitpack<uint32_t>. The BITPACK_USUAL_PREFIX macro hides some syntactic chatter that’s often useful (including, notably, constructors and assignment operators).

Our fields are now specified by invocations of the BITPACK_MEMBER_ADD(n, T, ...) macro. These invocations define methods named by n, each returning a T-based field proxy for the field whose FieldInfo follows.

Field proxies offer a number of methods on the field they reference:

• Getting the current value of the field. Proxies implicitly convert to their field’s type, so this is often syntactically “free”, but it is also explicitly available as the proxy’s .raw() method. In implementation, this is the expected shift and a mask to extract the field’s value.

• Setting the field value. Proxies overload the assignment operator (=) to update the field’s bits appropriately. All other bits in the underlying word are not altered. In implementation, this is the expected sequence of shift, complement, AND, and OR operations (as we’ve seen above). This is not provided by proxies of fields whose FieldInfo::isConst flag is true.

• Read-modify-write functionality. Proxies provide a .alter(f) method which invokes the callback f with the field’s current value and updates the field’s value with the callback’s return value.

• Cloning with override. Proxies provide a .with() method that behaves much like .alter() except that it returns a modified copy of the word. .with() can also be given a new value directly, rather than a callback.

• Comparison. Proxies overload the <=> operator, and can be compared to other proxies (of compatible types) or values of the field’s type.

Continuing our example, the code that uses this bitpack-flavored definition looks like…

uint32_t test(volatile NuDevice &device)
{
	device.control.enable() = false;

	device.control.alter([](auto c) {
		c.reference() = MYDEVICE_CONTROL_REFERENCE_V_EXTERNAL_A;
		c.bias()      = 0x7f;
		return c;
	});

	device.control.enable() = true;

	while (!device.control.read().active())
	{
		yield();
	}

	return device.data;
}

This compiles to the same code as the test(volatile RawDevice &) implementation above (and avoids the extra write that we had to introduce for test(volatile BFDevice &)).

Proxy assignments and .alter() methods are defined for volatile bitpack references (such as device.control in this example, because the containing NuDevice is volatile), and each such performs a read-modify-write operation on the underlying word. Proxy are more picky about field getter operations, and will refuse to operate on volatile bitpack references; the bitpack itself offers a .read() method that returns a non-volatile snapshot of the (volatile) bitpack. This serves to keep accesses explicit in the source and allows for clearly differentiating multiple tests of a single read snapshot from many tests each performing its own read of field value(s).

Compiler Error Messages

The compiler error messages for attempts to get a value from a volatile bitpack are, admittedly, nothing short of unfortunate. The best (well, most informative) case happens if we replace while (!device.control.read().active()) with while(!device.control.active().raw()) and looks something like this:

sdk/include/bitpack.hh:657:12: error: no matching conversion for static_cast from 'const Bitpack<...>::Field<bool, FieldInfo{...}>::Proxy<...>' to 'Field::Type' (aka 'bool')
...
./tmp/bitpack-blog.cc:172:34: note: in instantiation of member function 'Bitpack<...>::Field<bool, FieldInfo{...}>::Proxy<...>::raw' requested here
  172 |         while (!device.control.active().raw())
      |                                         ^
./sdk/include/bitpack.hh:649:14: note: candidate template ignored: constraints not satisfied [with Self = ...]
...
./sdk/include/bitpack.hh:648:10: note: because '!std::is_volatile_v<...>' evaluated to false
...

only with a lot more C++ template and Bitpack internal goo in place of those ellipses. At least it points at is_volatile as the problem!

Sometimes, if C++ is attempting further automagic (say, searching for implicit conversions) which fails, because we instead wrote while(!device.control.active()) (note the missing .raw()), the compiler will not actually emit such diagnostics of the root cause and will instead offer more mysterious reports like

./tmp/bitpack-blog.cc:172:9: error: invalid argument type 'typename Field<bool, FieldInfo{...}>::Proxy<...>'
      (aka 'Proxy<...>') to unary expression
  172 |         while (!device.control.active())
      |                ^~~~~~~~~~~~~~~~~~~~~~~~

This latter report is particularly confusing because it fails to point at volatile as a possible cause. Ideas for improving error messages are eagerly accepted.

Named Members with Custom Global Types

Much of the bitpack machinery is an attempt to encourage the use of fields with custom types. For example, we might give a name to the “reference” enumeration for our device, thus:

enum class MyDeviceControlReference : uint8_t
{
	Internal  = 0,
	ExternalA = 1,
	ExternalB = 3,
};

We can take advantage of this type by changing the reference member in our bitpack:

struct NeDevice
{
	struct Control : Bitpack<uint32_t>
	{
		BITPACK_USUAL_PREFIX;

		BITPACK_MEMBER_ADD(enable, bool, 0, 0); // NoGo is false / Go is true

		BITPACK_MEMBER_ADD(active, bool, 1, 1, true); // Off / Running

		BITPACK_MEMBER_ADD(reference, MyDeviceControlReference, 8, 9);

		BITPACK_MEMBER_ADD(bias, uint8_t, 16, 23);
	} control;

	uint32_t data;
};

The corresponding use code changes very little, but introduces our first use-side bitpack macro, BITPACK_OPERATE_QUALIFY:

uint32_t test(volatile NeDevice &device)
{
	device.control.enable() = false;

	device.control.alter([](auto c) {
		BITPACK_OPERATE_QUALIFY(c.reference(), =, ExternalA);
		c.bias() = 0x7f;
		return c;
	});

	device.control.enable() = true;

	// .read() makes read of volatile word clear in source
	while (!device.control.read().active())
	{
		yield();
	}

	return device.data;
}

Again, this compiles to the same code as the test(volatile NuDevice &) implementation above.

That BITPACK_OPERATE_QUALIFY macro invocation lets us avoid the need to manually qualify the ExternalA name; C++ name resolution rules would require that we write

c.reference() = MyDeviceControlReference::ExternalA;

The macro invocation takes advantage of the ability to look at the field’s type at compile time. It expands, essentially, to

c.reference() = (decltype(c.reference())::Field::Type)::ExternalA;

which is to say

c.reference() = MyDeviceControlReference::ExternalA;

Named Members with Custom Local Types

The same test code (other than the type of the parameter) also works (and compiles to the same code) when we move the reference field’s enumeration type into the Control structure itself:

struct NaDevice
{
	struct Control : Bitpack<uint32_t>
	{
		BITPACK_USUAL_PREFIX;

		BITPACK_MEMBER_ADD(enable, bool, 0, 0); // NoGo is false / Go is true

		BITPACK_MEMBER_ADD(active, bool, 1, 1, true); // Off / Running

		enum class Reference : uint8_t
		{
			Internal  = 0,
			ExternalA = 1,
			ExternalB = 3,
		};
		BITPACK_MEMBER_ADD(reference, Reference, 8, 9);

		BITPACK_MEMBER_ADD(bias, uint8_t, 16, 23);
	} control;

	uint32_t data;
};

With that definition, the BITPACK_OPERATE_QUALIFY(c.reference(), =, ExternalA) macro invocation in our test method still expands to

c.reference() = (decltype(c.reference())::Field::Type)::ExternalA;

but that now means

c.reference() = MyDevice::Control::Reference::ExternalA;

We strongly encourage the use of such local types, as it uses the device and register structures as namespaces, minimizing the amount of stuff at global scope and making the purpose of a type clearer.

Local Types as Member Names

In fact, fields can often be thought of as being named by their type: this device’s enable flag is of a different type from that device’s, even though they’re both active-high single-bit fields, and similarly for the other fields. It would be strange, after all, to read configuration values from one device and write them directly to another; we can ask the C++ type system to help ensure that we don’t accidentally do so.

Towards this end, bitpacks include a fair bit of machinery for introducing a custom type for a particular field and then using that type as the name of that field. To make use of this machinery, we first replace (or augment) the BITPACK_MEMBER_ADD macros above with BITPACK_MEMBER_ADD_* macros, which simultaneously introduce a new type and associate field information with it.

struct TyDevice
{
	struct Control : Bitpack<uint32_t>
	{
		BITPACK_USUAL_PREFIX;

		BITPACK_MEMBER_ADD_ENUM_BOOL(Enable, NoGo, Go, 0);

		BITPACK_MEMBER_ADD_ENUM_BOOL(Active, Off, Running, 1, true);

		BITPACK_MEMBER_ADD_ENUM(Reference, uint8_t, 8, 9) {
			Internal  = 0,
			ExternalA = 1,
			ExternalB = 3,
		};

		BITPACK_MEMBER_ADD_NUMERIC(Bias, uint8_t, 16, 23);
	} control;

	uint32_t data;
};

BITPACK_MEMBER_ADD_ENUM(T, B, ...) defines an enum class T with base type B; the definition of the enumeration follows the macro. BITPACK_MEMBER_ADD_ENUM_BOOL(T, VF, VT, bit, ...) is a wrapper thereof, defining an enum class T : bool with the enumerator VF having value false and VT true; because booleans occupy exactly one bit, it is not necessary to specify both the minimum and maximum bit positions. BITPACK_MEMBER_ADD_NUMERIC(T, B, ...) defines a class T which inherits from a Numeric<B> base, an internal helper class that provides, for example, implicit conversions to and from B. (Not all operations on B are supported by Numeric<T> and derived classes; if this proves problematic, do let us know and we’ll see what can be done!)

Rather than having named methods for each field of our bitpack, we now make use of a singular method template, .member<T>(), which returns a proxy of the field whose type is T. Because the types are defined within a bitpack type itself, they often need qualification:

• The BITPACK_MEMBER_DECLTYPE(b, T) macro gets the proxy of the field whose type is T qualified by the type of the bitpack value b. (That is, it expands to (b).template member<decltype(b)::T>().)

• If the type of the bitpack value b is “dependent” (in the C++ sense), use the BITPACK_MEMBER_DEPENDENT(b, T) macro, which differs only in that it calls for dependent resolution of the type named T. (That is, it expands to (b).template member<typename decltype(b)::T>().)

We can make use of macros that combine these and the OPERATE_QUALIFY form we’ve seen above when writing code that uses these type-centric bitpack definitions:

uint32_t test(volatile TyDevice &device)
{
	BITPACK_OPERATE_QUALIFY_DECLTYPE(device.control, Enable, =, NoGo);

	device.control.alter([](auto c) {
		BITPACK_OPERATE_QUALIFY_DEPENDENT(c, Reference, =, ExternalA);
		BITPACK_MEMBER_DEPENDENT(c, Bias) = 0x7f;
		return c;
	});

	BITPACK_OPERATE_QUALIFY_DECLTYPE(device.control, Enable, =, Go);

	while (BITPACK_OPERATE_QUALIFY_DECLTYPE(
	  device.control.read(), Active, !=, Running))
	{
		yield();
	}

	return device.data;
}

This too compiles to the same code as the test(volatile NuDevice &) implementation above.

BITPACK_OPERATE_QUALIFY_DECLTYPE(b, T, op, v) is just BITPACK_OPERATE_QUALIFY(BITPACK_MEMBER_DECLTYPE(b, T), op, v). Here, rather than using a method like .reference() to obtain a field proxy by name we use BITPACK_MEMBER_DECLTYPE to use the field’s type as its name and construct the appropriate proxy. BITPACK_OPERATE_QUALIFY_DEPENDENT, analogously, uses BITPACK_MEMBER_DEPENDENT internally. In much more detail, BITPACK_OPERATE_QUALIFY_DECLTYPE(c, Reference, =, ExternalA) can be thought of as expanding to

BITPACK_OPERATE_DECLTYPE(c.template member<TyDevice::Reference>(), =, ExternalA)

which then expands to

c.template member<TyDevice::Reference>() =
  decltype(c.template member<TyDevice::Reference>)::Field::Type::ExternalA

which is to say

c.template member<TyDevice::Reference>() = TyDevice::Reference::ExternalA

(Internally, the difference between explicitly named members and type-named members amounts to the former invoking .member<FieldType, FieldInfo>() while the latter invokes .member<FieldType>(). This second form uses some type-level computation machinery buried inside the BITPACK_USUAL_PREFIX and BITPACK_MEMBER_ADD_* macros to map from a FieldType to its FieldInfo.)

We could also have written, instead of BITPACK_MEMBER_DEPENDENT(c, Bias) = 0x7f, BITPACK_OPERATE_WRAP_DEPENDENT(c, Bias, =, 0x7f). The former makes use of Numeric<T>’s implicit conversion from T, while the latter is more general. The BITPACK_OPERATE_WRAP_... macros are similar to the BITPACK_OPERATE_QUALIFY_... except that they expand to invoke the indicated type’s constructor, rather than simply qualify the value. That is, BITPACK_OPERATE_WRAP(proxy, op, value) expands to (proxy) op (F{value}), invoking the F constructor, where F is the type of the field being proxied by proxy (specifically, decltype(proxy)::Field::Type).

Looking to the Future

Bitpacks are a public experiment and, as such, a request for comments. We are especially open to feedback about use cases that remain un-ergonomic or otherwise challenging.

At present, most of our use of bitpacks has been with manually written bitpack definitions. In the interest of reducing the number of “sources of truth” that must be kept synchronized, we have been pondering tools that could generate bitpack definitions from other register description languages (especially https://opentitan.org/book/util/reggen/index.html), or assertions of equivalence between a manually-written bitpack definition and one in another language. Should there be external interest, we would be keen to collaborate on this front.

Readers might rightly object that bitpacks, and their professed typed perspective of sub-word fields, lean rather heavily on C preprocessor macros (which act on the level of source code tokens, well below any notion of types). We hope that C++’s reflection machinery (to first appear in C++26) should allow many of these macros to go away once it’s a bit more complete.

Conclusion

This has been a whirlwind tour of the “bitpack” machinery available in the CHERIoT-RTOS programming environment. Bitpacks are meant to increase legibility, portability, and type-safety of low-level code that manipulates sub-fields of numeric words. We hope that you find they have achieved their purpose!

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away

– Antoine de Saint-Exupéry

In the last week or so, we’ve started landing patches in our compiler to remove the auicgp instruction. This is something that we’d hoped to do for a while because we want to remove the instruction entirely in CHERIoT v2 (to be based on the upcoming RV32YE standard). This post will explain why we had it, why we wanted to get rid of it, and how we’ve removed the need for it.

What is auicgp?

In CHERIoT, every global is either accessed relative to the program counter (PC) for code or immutable globals, or the global pointer (GP) for read-write globals. On RISC-V, PC-relative addressing is done via the auipc instruction, which takes a 20-bit immediate, left shifts it by 12, and adds it to the PC. This is used paired with a load, store, or add (if you just want to materialise the address as a pointer) to fill in the remaining bits.

For symmetry with base RISC-V (more detail on this later), we introduced a similar instruction that added a shifted immediate to the global pointer: add upper immediate to (capability) global pointer (auicgp). Originally, because we’re based on RV32E, we used one of the bits in the destination-register field to differentiate this from auipcc instruction, so it fitted into some spare encoding space.

In the initial public release, we changed this because RISC-V promises never to use that bit of encoding space for standard extensions and we hoped to standardise it. Since then, the auicgp instruction has been the least popular bit of the CHERIoT ISA, for a variety of reasons.

First, it’s an enormous instruction. With a register target and a 20-bit immediate, it uses a full major opcode. That’s 1/128 of the total encoding space. In practice, it’s worse than that, because the RISC-V encoding requires 32-bit instructions to start 11 (01, 10, and 00 are reserved for 16-bit encodings), it’s actually 1/32 of the space available for 32-bit instructions. That’s a lot to pay, though it might be worth it if the instruction were used a lot.

Unfortunately, auicgp is very rarely used. Our ABI points the global pointer to the middle of the globals region, which means that we can use the entire 4 KiB range of the 12-bit immediate in loads, stores, and adds. On RISC-V, the normal model is for the compiler to emit a long instruction sequence for operations like global accesses and then the linker to relax this by deleting instructions. We followed this pattern for auicgp, with the linker deleting it if the immediate is 0. As a result, almost all of the auicgp instructions were deleted. Our original linker version didn’t implement support for relaxations (upstream LLD didn’t support them for RISC-V when we started working on CHERIoT!) and so we weren’t initially sure how feasible deleting them would be, but we’ve had linker relaxations working for a few years now and they work very well.

The RTOS test suite contains a few auicgps, but only because we’ve intentionally written tests to make sure that large globals work. In most firmware images, there are no instances of the instruction that survive relaxation. So we’re using 1/32 of the encoding space for 32-bit instructions for an instruction that, to a first approximation, never gets used.

If that isn’t bad enough, it’s also annoying for simple pipelines to implement. RISC-V is designed so that the source and destination registers are always in the same place in instructions. Once you move to long pipelines, have register rename, or extensions with multiple register files (such as floating-point or vector extensions) then this is of little or no benefit. But CHERIoT targets the kind of microcontroller design that RISC-V overfitted for. Simple in-order pipelines find it useful to do register fetch early. The auicgp instruction is the only one that has fetches from the general-purpose register file but does not have the registers in the correct location. This adds extra multiplexing that, on a short in-order pipeline such as CHERIoT Ibex, impacts the critical-path length.

So, unfortunately, there is no situation in which this instruction is a good idea, except that it made it easy to create dense code via largely orthogonal code paths. To understand why we incorporated it, you need to understand how RISC-V normally generates pointers to globals.

Why did we have auicgp?

Let’s start with the simple case in RISC-V, of a static binary (how you’d normally compile embedded firmware) accessing a global. A simple sequence loading an int from a global called x would look like this:

	lui     a0, %hi(x)
	lw      a0, %lo(x)(a0)

This has two relocations. The load upper immediate instruction will materialise the top 20 bits of the address via the hi relocation, then the load will materialise the low 12 bits in its immediate from the lo relocation. This is very efficient, but doesn’t work on a CHERI system because you can’t just make up an integer address and use it for loads and stores.

That’s not a big problem, because it’s the same issue that position-independent code needs to handle, and RISC-V does this with the same number of instructions. When you’re compiling position-independent code, RISC-V will instead generate the following sequence:

.Lpcrel_hi0:
	auipc   a0, %pcrel_hi(x)
	lw      a0, %pcrel_lo(.Lpcrel_hi0)(a0)

Again, this materialises the address in two instructions, including folding half of the address-materialisation into the load. The difference is that the high bits are not created as an absolute address, they are created by adding a value to the current PC value. The %pcrel_lo relocation looks a bit odd because it refers to the address of the auipc instruction, not the address of x. This is because of limitations in ELF relocations. The linker needs to know both the address of the auipc instruction (which doesn’t have to be next to the lw, and might not be if the lw is in a loop) and of the target, but ELF relocations can’t express both of these. Fortunately, the linker can work around this by looking up the relocations that apply to the address of the auipc instruction to find the target.

On CHERIoT, accessing read-only globals works in exactly the same way. They are stored within the bounds of the program counter for the current compartment (or library) and so we can compute a displacement and load. Read-write globals are instead stored within a region reachable via the global pointer. These capabilities don’t overlap, and they have distinct permissions.

This is where we encounter the first problem. Most of the time, the compiler knows whether a global is read-only or not and so knows whether it needs to emit a sequence relative to the program counter or the global pointer. This is not universally true. In particular, C inherited a notion of ‘common linkage’ from Fortran and this makes it possible for different compilation units to disagree on whether something is read-only and leaves it for the linker to figure it out.

For read-only globals, the sequence looks like this (note: the ct. prefix is the official vendor prefix for CHERIoT instructions):

.LBB0_1:
	ct.auipcc    a0, %cheriot_compartment_hi(x)
	ct.clw       a0, %cheriot_compartment_lo_i(.LBB0_1)(a0)

But for read-write globals, it looks like this:

.LBB0_1:
	ct.auicgp   a0, %cheriot_compartment_hi(x)
	ct.clw      a0, %cheriot_compartment_lo_i(.LBB0_1)(a0)

This hopefully makes it clear how the auicgp instruction is useful. These two sequences look identical and so the linker can rewrite one to the other by simply rewriting the opcode of the first instruction. If the compiler tries to do PC-relative addressing for a read-write global, the compiler can turn the auipcc into an auicgp, or vice versa.

For read-write globals, the linker can also do relaxation. In the (common, almost universal) case that the address of x is within the displacement from the global pointer of a single instruction, the second sequence becomes this:

	ct.clw	a0, {displacement of x from the global pointer}(c)

Note that this is now fewer instructions than the baseline RISC-V case. We hit this in a lot of cases, because a lot of global accesses are simple reads and writes of scalar values. That’s not universally true. If the address of the global escapes the analyses that the compiler uses to prove that a dereference is in-bounds (which are currently fairly naïve, but will improve over time), then we must materialise a full capability.

This is why it’s important to design a CHERI ABI along with a threat model. Our threat model requires that any compartment be able to enforce a list of memory-safety properties against untrusted code, but treats things like memory safety for stack or global variables within a compartment as defence-in-depth properties. Code within a compartment is allowed to access all globals for that compartment and so the threat model says it’s fine to make the compiler responsible for enforcing those bounds. But when a pointer to a global is passed to another compartment (possibly indirectly via another compilation unit, maybe in a different language, in the same compartment) then the compiler taking the address of that global must be able to apply bounds and permissions.

If we are taking the address of x, the compiler will generate a sequence like this:

.LBB0_1:
	ct.auicgp       a0, %cheriot_compartment_hi(x)
	ct.cincoffset   a0, a0, %cheriot_compartment_lo_i(.LBB0_1)
	ct.csetbounds   a0, a0, %cheriot_compartment_size(x)

Note that now the second part of the address isn’t folded into the load instruction. This doesn’t hurt code size as much as you might think, because now we’re likely to do offset-zero loads and stores on the result and RISC-V lets us express these as 16-bit instructions.

The new instruction is applying the bounds. As before, the linker will normally relax away the first instruction, so we end up with this sequence for taking the address of a global:

	ct.cincoffset   a0, a0, {offset of x}(gp)
	ct.csetbounds   a0, a0, {size of x}(x)

Again, this is the same number of instructions as baseline RISC-V, but now with bounds applied. For comparison, here is the RISC-V version:

	auipc   a0, %pcrel_hi(x)
	addi    a0, a0, %pcrel_lo(.Lpcrel_hi0)

There is one annoying case though: what happens if the bounds are too large for an immediate? Here, we need an extra instruction (and register) to hold the immediate. This means that our worst case (large bounds, address taken or not provably in bounds) is five instructions (if it takes two instructions to materialise the immediate for the bounds), much worse than our best case of one.

The approach taken on the ABI for big CHERI is to use an indirection layer, still referred to as a global offset table (GOT), even though it doesn’t contain offsets. This mirrors the baseline RISC-V GOT model, where the same sequence would be:

        auipc   a0, %got_pcrel_hi(x)
        lw      a0, %pcrel_lo(.Lpcrel_hi0)(a0)

At two instructions, this looks cheap, but note that the second is a load, which is loading the address from the GOT. This means that, although it’s only eight bytes of instructions, it’s also one pointer’s worth of data (eight more bytes for CHERIoT, sixteen for big CHERI). The second instruction is accessing memory (even on CHERIoT systems with local SRAM, memory-access instructions are typically slower than ALU instructions).

This is worse than our common case, but is often better than our worst case.

The new model

In the new approach, any global that isn’t reachable via a short displacement from the global pointer (i.e. things that would need auicgp) is turned into a GOT-relative access.

This also serves as a building block for other optimisations. The three-instruction sequence is smaller than the GOT-relative access if a global is accessed only once, but is larger if the same global has its address taken three or more times. The linker has full visibility into the number of accesses to a global and so can make this decision.

Similarly, globals that are too big to have their bounds applied in a single instruction will be able to fall back to the GOT mechanism. These are rare but they do occur for things like frame buffers. These are close to break even for a single use. For example, imagine if x is an array of 1028 ints. Taking the address is currently this (long!) sequence:

	lui             a0, 1
	addi            a0, a0, 16
.LBB0_1:
	ct.auicgp       a1, %cheriot_compartment_hi(x)
	ct.cincoffset   a1, a1, %cheriot_compartment_lo_i(.LBB0_1)
	ct.csetbounds   a0, a1, a0

Assuming that the auicgp is relaxed away, this becomes:

	lui             a0, 1
	addi            a0, a0, 16
	ct.cincoffset   a1, a1, {offset of x}(gp)
	ct.csetbounds   a0, a1, a0

This is four instructions, but both the lui and addi will, in this case, be the 16-bit variants, so this is 12 instructions: smaller than the GOT-relative sequence plus the GOT entry. This sequence also requires an additional register, and removing that requirement can improve code generation in the rest of the function.

To support this, we’ve introduced a new ct.auipcc.data pseudo-instruction that emits padding nop that is normally relaxed away, but provides space for the linker to emit the longest sequence. The sequence the compiler will emit looks like this:

.LBB4_1
	ct.auipcc.data  s0, %cheriot_compartment_data_hi(x)
	ct.cincoffset   s0, s0, %cheriot_compartment_lo_i(.LBB4_1)
	ct.csetbounds   s0, s0, %cheriot_compartment_size(x)
	ct.cincoffset   a0, s0, a0

The linker can then relax this to any of the sequences described above, almost always to something much shorter.

The extra padding is needed only in one specific case, but emitting it in all cases that may use the global pointer simplifies the linker logic. If the compiler statically knows that the access is in bounds, it will emit the short two-instruction sequence of auipcc followed by a load or store. This is two 32-bit instructions (8 bytes total). If the global is large, this must be transformed into a GOT load followed by a load or store, which is three instructions, two of which might be 16-bit, but in the worst case all three will need the full range of the 32-bit variants. The linker always removes the nop but may use the space that it reserved for the longer sequence, in this one corner case.

A final note

CHERIoT benefits enormously from having co-designed the ISA, ABI, and programmer model. The cheriot-audit tool makes it easy to reason about the security of a firmware image. This tool is possible because we defined our ABI so that everything that a compartment accesses outside of its code and globals region is explicit (in the programmer model).

Several of these optimisations are possible only because we took a holistic view to system design. If, for example, we allowed globals from other compartments to be transparently accessed then this would have required a GOT approach from the start and would also have made reasoning about communication between compartments much harder. Instead, we have an explicit notion of a pre-shared object, which must come via a compartment’s import table and so is exposed to auditing. By making the import explicit, we can also go beyond core C features and make the permissions explicit, so we can see by reading the source code, and confirm with cheriot-audit that exactly what permissions a compartment has on a shared object.

Decades of prior work have shown us that security policies divorced from the source code are hard to reason about and often get out of sync with the implementation. When they get out of sync, people notice when components have too few permissions (they stop working) but not when they have too many (until an attacker notices).

For CHERIoT, we had an explicit design goal that you should be able to reason about the security of your code by reading the source and do coarser-grained auditing of other people’s code with additional tooling. Whether a function is in your compartment or elsewhere is explicit in the source. Whether a global is uniquely owned by you or a pre-shared object is explicit in the source. The ABI we co-designed with the programmer model to enable this and it lets us emit very short instruction sequences in the common case, while falling back to larger ones to handle corner cases.

Most CHERIoT work to date has been done on software or FPGA simulations. We have several such implementations: The executable model built from our formal ISA specification, the MPact simulator from Google, Microsoft’s CHERIoT SAFE FPGA target for the Arty A7, and of course lowRISC’s beautiful Sonata FPGA board, which is designed to simulate CHERIoT systems. These were always intended to be developing and prototyping systems, so I’m delighted to announce that SCI Semiconductor has the first silicon CHERIoT implementation.

[ Conflict disclaimer: I am a co-founder of SCI Semiconductor. ]

The dev board pictured above contains one of the first batch of ICENI chips to come back from the fab. This is a complete CHERIoT system, with all of the core CHERI properties (spatial memory safety, no pointer injection, and so on) along with all of the CHERIoT extensions that provide deterministic use-after-free protection, auditable control over interrupt state, and everything that we need for an aggressively compartmentalised RTOS.

This chip uses the CHERIoT Ibex core, running at up to 250 MHz, and includes a few feature that accelerate temporal safety, improve interrupt determinism, and so on. These build on top of all of the benefits of any CHERIoT implementation: deterministic mitigation of memory safety bugs from simple buffer overflows up to use-after-free, fine-grained compartmentalisation, and a programming model co-designed with both the ISA and the software stack to provide a tiny TCB. Anything that works on CHERIoT SAFE or Sonata should be very easy to port to ICENI for production use. Anything that runs on the software simulators should just work.

We’ll be showing the chips at Embedded World (Stand 4A - 131) next week and at CHERI Blossoms a couple of weeks later. From tomorrow, one will also be on display in the CHERI 15th anniversary exhibit in the Cambridge Computer Laboratory.

Aside: The Iceni tribe were one of the pre-Roman tribes in Britain and are famous for their chariots (though more due to this statue than historical fact). I am only partially to blame for the bad puns in the naming.

The top three (CWEs 79, 89, and 352) are not typically applicable on embedded platforms. Two are cross-site things that apply to web applications, one is SQL injection.

At position 4, we have CWE-862, missing authorisation. This is not something that’s automatically mitigated by CHERIoT, but the design of CHERIoT RTOS and the programming model that we expose makes it easy to write code that avoids this kind of issue. The CHERIoT pattern for any operation that you do on behalf of another compartment is to require an authorising capability. For example, if you allocate memory, you must present an allocation capability that encapsulates your right to allocate memory (and your quota). If you want to create a socket, you must present a capability that authorises you to bind to a specific port (for server sockets) or connect to a specific remote host. The same applies for dynamically created things, such as sockets themselves, message-queue endpoints, and so on. If you forget to authorise something, it will not have the capability to perform the action and the action will fail. This is a general property of capability systems and not something specific to CHERIoT.

Position 5 is CWE-787, out-of-bounds write, also known as a buffer overflow. This one is deterministically mitigated by any CHERI platform.

Technically, CHERIoT is not vulnerable to the path-traversal bugs in position 6 (CWE-22), but only because we don’t yet ship a filesystem. But, again, this kind of issue is a solved problem in capability systems. Capsicum, for example, eliminates this kind of vulnerability and I’d expect our filesystem APIs to follow a similar shape. There’s no excuse for writing APIs that are vulnerable to path traversal in the 2020s.

The next two are good old-fashioned memory-safety vulnerabilities. Position 7 is use after free (CWE-416), and 8 is out-of-bounds read (CWE-125). The latter is mitigated by any CHERI platform. The former is usually made unexploitable by CHERI systems, and is deterministically mitigated by CHERIoT.

The next two are unlikely to apply to embedded platforms. CWE-78 at position 9 is largely to do with failing to validate dynamically created command lines that you pass to a shell. Then CWE-94 (Improper Control of Generation of Code) at position 10 is typically introduced with scripting languages producing output that can be influenced by an attacker and is then executed by another interpreter, a rare situation on embedded devices.

Position 11 (CWE-120) is a ‘Classic Buffer Overflow’, i.e. something that CHERI deterministically mitigates. Not to be confused with the buffer overflow we had at position 5.

The 12th entry is another that’s rare on embedded devices. CWE-434 relates to unrestricted uploads of dangerous file types, something that matters a lot to web apps and far less to other classes of program.

Next, position 13, is a null-pointer dereference where a valid pointer was expected (CWE-476). CHERI guarantees that this will trap (even if an attacker can provide arbitrary offsets to the null pointer) and CHERIoT makes this a recoverable error either via our scoped error handlers or by simply unwinding the compartment to the caller.

Buffer overflows seem to be popular and position 14 is the third instance of this kind to make the list, this time on the stack (CWE-121). Again, this will deterministically trap on any CHERI platform.

The next entry is more interesting. Unsafe deserialisation of untrusted data (CWE-502) is something a lot of people get wrong. Phil Day wrote about how to do this safely a couple of years ago. Lightweight compartmentalisation makes it easy to limit the scope of damage that this kind of bug can do, to almost nothing.

Did I mention that buffer overflows are a recurring theme on this list? Position 16 (CWE-122) is yet another buffer overflow, this time on the heap. One more that any CHERI platform deterministically mitigates.

Positions 19–21 all relate to incorrect access control at the application layer and, sadly, are not mitigated by CHERI. Position 24 is similar.

In between these, we have another web app problem (CWE-918, server-side request forgery) and another command injection (CWE-77). These are unlikely to be present on embedded devices.

Finally, at position 25, we have a fairly broad category of availability issues that arise from not constraining resource allocation (CWE-770). These are normally mitigated by the software capability layer on CHERIoT. For example, a compartment can’t allocate memory unless it has a capability that authorises it to do so. That capability encapsulates a quota and so provides a limit to the total amount of allocation. Other resources that can be dynamically allocated are normally managed in the same way.

So, what’s the final score card?

Not applicable in embedded contexts: 1, 2, 3, 9, 10, 12, 22, and 23

Deterministically mitigated with just a recompile: 5, 7, 8, 11, 14, and 16.

Mitigated by CHERIoT design patterns and software model: 4, 6, 13, 15, 25.

That still leaves six that we don’t mitigate (17, 18, 19, 20, 21, and 24), but now hopefully the cognitive load is much lower from not having to think about the eleven that we do prevent and you can avoid some of these as well!

What are ML-KEM and ML-DSA for?

I am not a mathematician and so I’m not going to try to explain how these algorithms work, but I am going to explain what they’re for.

Module-Lattice-Based Key-Encapsulation Mechanism (ML-KEM) is, as the name suggests, an algorithm for key encapsulation. One side holds a public key and uses it (plus some entropy source) to generate a secret in both plain and encapsulated forms. The encapsulated secret can be sent to a remote party who holds the corresponding private key. The receiver can then recover unencrypted version of the secret (and detect tampering). Now, both parties have the same secret and can use it with some key-derivation function to produce something like an AES key for future communication.

Note that this is somewhat more restrictive than traditional key-exchange protocols. You don’t get to exchange an arbitrary value, the generation step is part of encapsulation. This also means that it’s a fixed size, defined by the algorithm, which is why you typically feed it into a key-derivation function rather than using it directly.

Module-Lattice Digital Signature Algorithm (ML-DSA) has a similarly informative name. It is intended for providing and validating digital signatures. It takes a private key, an arbitrary-sized document and context, and produces a signature. A holder of the associated public key can then validate that the document matches the version signed with the private key and context.

These are both quite low-level building blocks for higher-level protocols. For example, TLS can use ML-KEM for key exchange and ML-DSA for certificate validation, but also incorporates traditional algorithms in case the PQC algorithms have unexpected weaknesses against classical computers.

Initial porting

As is usually the case for CHERIoT, porting the C implementations of ML-KEM and ML-DSA required no code changes. I worked with upstream to slightly simplify the platform-integration layer, so we just provide a single header describing the port. For example, the port header for ML-DSA configures the build to produce ML-DSA44 support, defines a custom function for zeroing memory and getting entropy, and adds the __cheriot_libcall attribute to the all exported APIs (so we can build them as shared libraries, rather than embedded in a single compartment). The file for ML-KEM is almost identical.

With these defined, it is possible to build both libraries as CHERIoT shared libraries. This motivated a bit of cleanup. We have a device interface for entropy sources, but it wasn’t implemented on the Sail model (which doesn’t have an entropy source). It has a way of exposing the fact that entropy is insecure, so that wasn’t a problem, it just needed doing, so I refactored all of the insecure entropy-source drivers to use a common base. Most encryption algorithms want an API that fills a buffer with entropy. It’s nice if these don’t all need to touch the driver directly, so I created a compartment that provides this API and exposes it. Now, both libraries are simply consumers of this API. This also makes it easier to add stateful whitening for entropy drivers for hardware entropy sources that don’t do the whitening in hardware.

Most CHERIoT stacks are on the order of 1-2 KiBs. The PQC algorithms use much more space. More, in fact, than we permitted.

The previous limitation was based on the precision of bounds rounding. A CHERI capability compresses the bounds representation by taking advantage of the fact that, for a pointer to an allocation, there is a lot of redundancy between the address of the pointer, the address of the end of the allocation (the top), and the address of the start of the allocation (the base). The distance from the address to base and top are stored as floating-point values with a shared exponent. In practical terms, this means that the larger an allocation is, the more strongly aligned its start and end addresses must be. The same restrictions apply for any capability that grants access to less than an entire object.

When you call a function in another compartment, the switcher will truncate the stack capability so that the callee sees only the bit of the stack that you weren’t using. The top and base of the stack must be 16-byte aligned (as an ABI requirement), but a very large stack may have hardware requirements for greater alignment and so may require a gap between the bottom of the caller’s stack and the top of the callee’s.

Fortunately, we’d added an instruction precisely for this kind of use case: CSetBoundsRoundDown. This takes a capability and a length and truncates it to at most that length. It was a fairly small tweak to the switcher to make it do this, and a much larger amount of time with SMT solvers to convince ourselves that this was a safe thing to do.

This also showed up a bug in our linker’s handling of the CAPALIGN directive, which rounds a section’s base and size up to the required alignment to be representable. This was not working for sections that followed an explicit alignment directive. Our stacks must be both at least 16-byte aligned and representable as capabilities. This is now fixed.

So now we support stacks up to almost 64 KiB, a limitation imposed by the current loader metadata format rather than anything intrinsic to how the system operates after booting. We could easily increase this limit but 64 KiB ought to be enough for anyone.

Performance on CHERIoT Ibex

The repository contains a simple benchmark example that tries each of the operations and reports both the cycle time and stack usage. The output on the CHERIoT Ibex verilator simulation is:

PQC benchmark: Starting: stack used: 224 bytes, cycles elapsed: 41
PQC benchmark: Generated ML-KEM key pair: stack used: 14304 bytes, cycles elapsed: 5143987
PQC benchmark: Encrypted secret pair with ML-KEM: stack used: 17440 bytes, cycles elapsed: 1773235
PQC benchmark: Decrypted secret pair with ML-KEM: stack used: 18464 bytes, cycles elapsed: 2176226
PQC benchmark: Compared results successfully for ML-KEM: stack used: 224 bytes, cycles elapsed: 414
PQC benchmark: Generated ML-DSA key pair: stack used: 46912 bytes, cycles elapsed: 3622132
PQC benchmark: Signed message with ML-DSA: stack used: 60544 bytes, cycles elapsed: 5391177
PQC benchmark: Verified message signature with ML-DSA: stack used: 44672 bytes, cycles elapsed: 3674071
PQC benchmark: Correctly failed to verify message signature with ML-DSA after tampering: stack used: 44672 bytes, cycles elapsed: 3673706

The ML-KEM encrypt (generate shared secret and encrypted version) and decrypt (recover shared secret from encrypted version) each use around 18 KiB of stack and run in around two million cycles. CHERIoT Ibex should scale up to 200-300 MHz (though may be clocked lower for power reasons in some deployments), but even at 100 MHz that’s 50 encryption or decryption operations per second. Remember that this is an operation that typically happens when you establish a connection, then you use a stream cypher such as AES with the exchanged key.

The ML-DSA operations are slower and use a lot more stack space (almost 60 KiB for signing!). But, even there, the performance is reasonable, under 4 M cycles. This means that you can do 20 signature-verification operations per second at 100 MHz.

Even using ML-KEM for key exchange and ML-DSA for certificate validation in a TLS flow is unlikely to add more than a few tens of milliseconds to the handshake time, which is perfectly acceptable for the common use case for embedded devices.

In terms of code size, both are small. The ML-KEM implementation is around 12 KiB, the ML-DSA implementation 18 KiB. These both include a SHA3 (FIPS 202) implementation, so there’s scope for code-size reduction on systems that need both, but 30 KiB of code isn’t too bad.

Future plans

The stack usage is very high. Upstream has some plans to allow pluggable allocators, which will allow us to move a lot of this to the heap. This is precisely the kind of use case that CHERIoT’s memory-safe heap is great for: something needs 60 KiB of RAM for 4,000,000 cycles, but then doesn’t need that RAM again for a long time. That memory can then be used for something else, even in a mutually distrusting compartment.

Currently, the library builds are very thin wrappers around the upstream projects. This is great as a building block, but we should make more use of CHERIoT features in the longer term.

Both ML-KEM and ML-DSA depend on SHA3 (FIPS 202). Ideally, we’d factor that out as some common code, rather than carrying a copy in each library. Similarly, the libraries provide an option to plug in your own SHA3 implementation. This is likely to be a common hardware operation even for chips that don’t have full PQC implementations, so we should expose this option in the build system.

Is it secure?

Security always depends on the threat model.

For signature validation, you don’t have any secret data, just a public key, a document, and a signature. The only concerns are whether there are weaknesses in the algorithm, or bugs, that would allow an attacker to substitute a different document for the same signature. CHERIoT prevents memory-safety bugs, so this is concerned solely with logic errors. The code upstream is checked against a set of test vectors that aim to trigger corner cases in the logic of the underlying implementation, so hopefully is secure in this way.

For signing or key exchange, you need to worry about the key leaking. On a CHERI system, it’s unlikely to leak explicitly, but may leak via side channels. The security section of the upstream projects discusses a number of techniques that they use to mitigate this kind of attack.

That’s typically sufficient. It’s been recommended practice for embedded devices to have per-device secrets for a long time. This means that leaking a key from one device doesn’t compromise the device class, only that specific device.

For some very high-assurance use cases, that secret may matter and need to be robust against an adversary with physical access to the device. Hardware encryption engines typically care about confidentiality breaches via power side channels and integrity breaches via glitch injection. Power side channels are difficult to mitigate in software: the power requirements of multiplying two numbers together may depend on the number of carry bits set, for example. They’re much easier to mitigate in hardware, by simply doing the same calculation twice in parallel, once with the original inputs and once with the inputs permuted to have the opposite power characteristics.

Glitch injection takes the chip out of its specified power or frequency (or thermal) envelope and attempts to introduce bit flips, which can corrupt state in such a way that tamper with signing or leak a key. These are also effectively impossible to mitigate in software because the software that’s attempting the mitigation is vulnerable to the same glitches. There are some compiler techniques that can make these harder, but they come with a high performance cost.

If power analysis and glitch injection are part of your threat model, the software implementations are not sufficient. In this case you may also need to worry about someone removing the top of the chip and using a scanning-tunnelling electron microscope to read bits from non-volatile memory. This used to require tens of thousands of dollars but is now much cheaper. Devices that need to worry about this often have tiny explosive charges in the package to destroy the chip in cases of tampering. If that’s your threat model, hardware PQC implementations may not be sufficient, at least alone.

But if you care about attackers on the network being unable to compromise the security of the class of devices, even if they have a magical and imaginary quantum computer, then these should be sufficient.

TockOS is implemented in Rust and has safety as a priority. I generally regard it as the gold standard for an RTOS that is forced to operate under the constraints of working with existing hardware. But note where the drivers are: In the kernel, right at the bottom.

Here’s a similar diagram from the Zephyr RTOS:

Here, there is some separation between drivers and the core of the kernel, but the default configuration runs with no privilege separation. Indeed, the Zephyr Security Overview says:

The security architecture is based on a monolithic design where the Zephyr kernel and all applications are compiled into a single static binary. System calls are implemented as function calls without requiring context switches.

In fact, the only exception I’ve seen to this recently is LionsOS, a multiserver system built on top of the formally verified seL4 microkernel:

This runs all of the device drivers in unprivileged contexts. Unfortunately, seL4 assumes an MMU and so is not feasible for small embedded devices (seL4 uses more memory to hold page tables than a lot of CHERIoT firmware images use in total).

Drivers are attack surface

Device drivers, at least for I/O devices, are code that interacts with the outside world. An attacker trying to compromise a device has a much easier job if they don’t need to take the chip apart. The easiest attacks to mount are ones that work from an across the network. The next easiest are ones that drive local I/O, which may be reachable remotely via other paths.

Drivers for I/O devices, by definition, make decisions based on the values that they read from the device. These operations include mapping error codes to some type-safe enumeration (which can fail if the hardware is buggy), or using the values to index into other structures. These are hard to get right, even in a memory-safe language, because they often sit below the language’s abstraction layer. Microsoft estimates that 70% of Windows crashes are caused by bugs in device drivers.

Any bug in a driver for an I/O device is a useful building block for an attacker. This problem is made much worse if the device driver is in a privileged component. For example, at the end of last month there were three bluetooth CVEs in Zephyr that all could lead to compromise and, if the Bluetooth stack is not privilege separated, can lead to arbitrary code execution by an attacker who gets within a few meters of the device (or compromises another Bluetooth-enabled device nearby).

Device drivers do abstraction and multiplexing

This design results from conflating the two functions of a device driver.

A device driver has to provide an abstraction over a particular device. Sometimes this happens in multiple layers. For example, an Ethernet device may have an abstraction for sending and receiving Ethernet frames, but this is then the foundation for a further abstraction layer for sending IP packets, which is then used to expose TCP streams and UDP datagrams.

A device driver often also has to provide some secure multiplexing. For example, two mutually distrusting components may be allowed to create sockets for different TCP connections that flow over the same Ethernet device. Or they may be allowed to talk to two different USB bus endpoints via the same USB controller.

The first of these requirements is a software engineering problem. The second is primarily a security problem, but typically the multiplexed abstractions need to be device independent and so it’s also a software-engineering problem.

In embedded development, there’s often a distinction between a hardware-abstraction layer (HAL) and a driver, with the former providing only the abstraction and the latter also providing multiplexing. This is a useful distinction because, in a lot of cases, embedded systems have a single consumer for a device. For example, you may have multiple SPI or I²C interfaces on a device, but each one is used for a single purpose. It is convenient to be able to write software to talk to a SPI device without having to know exactly which SPI controller this chip uses, but you don’t need to handle safely sharing those pins with other components.

CHERIoT RTOS distrusts drivers

In CHERIoT RTOS, the core platform provides device abstractions that meet the earlier definition of a HAL: they provide abstractions over classes of device, but do not attempt to provide security. The RTOS also provides a trivial way of auditing which compartments can access which devices, so that you can ensure that devices are not accessible to compartments that are not trusted to interface with them.

The platform’s device code runs within whatever compartment you instantiate it in. It has no elevated privileges except the MMIO region(s) that you explicitly pass it for talking to a particular device.

This makes it easy to support both bespoke and reusable security models. If you need to share a device between two compartments with some secure multiplexing based on a custom policy, you can do that by instantiating the driver in a compartment and exposing APIs to the two others.

Sometimes, the desired abstractions are reusable. For example, the CHERIoT network stack is assembled out of the following compartments:

graph TD
  Network
  subgraph Firewall["On-device firewall"]
    DeviceDriver["Device Driver"]
  end
  TCPIP["TCP/IP"]:::ThirdParty
  User["User Code "]
  NetAPI["Network API"]
  DNS["DNS Resolver"]
  SNTP:::ThirdParty
  TLS:::ThirdParty
  MQTT:::ThirdParty
  DeviceDriver <-- "Network traffic" --> Network
  TCPIP <-- "Send and receive Ethernet frames" --> Firewall
  DNS <-- "Send and receive Ethernet frames" --> Firewall
  NetAPI -- "Perform DNS lookups" --> DNS
  NetAPI -- "Add and remove rules" --> Firewall
  TLS -- "Request network connections" --> NetAPI
  TLS -- "Send and receive" --> TCPIP
  NetAPI -- "Create connections and perform DNS requests" --> TCPIP
  MQTT -- "Create TLS connections and exchange data" --> TLS
  User -- "Create connections to MQTT server and publish / subscribe" --> MQTT
  MQTT -- "Callbacks for acknowledgements and subscription notifications" --> User
  SNTP -- "Create UDP socket, authorise endpoints" --> NetAPI
  SNTP -- "Send and receive SNTP (UDP) packets" --> TCPIP
  TLS -- "Request wall-clock time for certificate checks" --> SNTP
  style User fill: #5b5
  classDef ThirdParty fill: #e44

Note that the driver for the Ethernet device is instantiated in the firewall compartment. What happens if an attacker gets arbitrary-code execution here? They could mount a denial of service attack (refuse to forward Ethernet frames in or out). They could tamper with Ethernet frames.

This sounds bad but the rest of a network stack already has to assume that things like this can happen. Packets coming over the network are intrinsically untrusted. The TCP/IP stack has to assume that they may be malicious. It isn’t always good at this. The FreeRTOS TCP/IP stack that we use has had 15 CVEs disclosed since it was released, but our compartmentalisation strategy mitigates all of them. By placing the parts of the system that are exposed to an attacker in the least, not most, trusted places, we make it easy to build secure systems.

The funding was awarded back in September, but the embargo was lifted last week and so we can talk about it publicly.

Rust and CHERIoT have complementary benefits

I’ve written previously about why safe languages and CHERI are complementary. Superficially, both Rust and CHERI provide similar benefits in terms of memory safety. That similarity goes away when you look at the details.

Rust provides a very rich set of type-system guarantees. If you are writing software, you can use Rust’s type system to enforce a range of properties that go beyond simple memory safety. The key part here is ‘if you are writing software’.

The industry learned some important lessons from Java and JavaScript attempts at language-level sandboxing: it’s very hard to write a compiler that assumes that the programmer is an adversary. Any soundness issue in the underlying type system or any bug in the compiler can be a security vulnerability if you assume that the person writing the software is malicious and actively trying to break the guarantees that the language aims to enforce.

The Rust compiler is currently tracking 107 bugs marked as soundness issues. A typical Rust programmer is unlikely to encounter these. Encountering these bugs typically require poking at corner cases of the language that you’re unlikely to hit by accident. In contrast, a malicious programmer wanting to insert a supply-chain vulnerability into something that you consume has a rich set of tools.

The CHERIoT compartmentalisation was designed with this kind of adversary in mind. It assumes that you may be incorporating arbitrarily buggy or malicious code into a device’s firmware and need to be able to protect against components that are compromised. The checks that CHERIoT does at run time are less rich than those that Rust enforces at compile time, but are not bypassable, even with unsafe code or inline assembly.

This means that you can use Rust’s type system to ensure that your code has strong confidentiality, integrity, and availability guarantees, while simultaneously using CHERIoT to ensure that supply-chain code cannot violate these guarantees (at least with respect to confidentiality and integrity, though availability is a bit harder). Rust’s guarantees make achieving high levels of confidence in your code much easier than if you used C/C++. Rust is one of the few languages that delivers this kind of guarantee and is also able to run on small embedded devices (such as CHERIoT implementations), making it a very exciting choice for future CHERIoT development.

CHERI and Rust give you the benefits of Rust faster

Rust gives rich guarantees to Rust code. Most software is not written in a single language, and especially not in a relatively young language. What if you have some legacy C/C++ component? You can use it from Rust, but in most systems this means bugs in the legacy code can completely undermine the security guarantees of the Rust code. A memory-safety bug in C code can corrupt any Rust object in the same process.

On a CHERI system, you get different levels of guarantee depending on how you mix languages. Calling C code from Rust within the same compartment requires the C code to follow the CHERI rules. If Rust code passes a pointer to C, the C code may tamper with objects reachable from there, but can’t arbitrarily affect the system unless it has a much narrower set of bugs. Stack uninitialised use bugs may cause C to access objects left on the stack, for example.

If you put the C code in another compartment, these guarantees become a lot stronger. C code can tamper with objects passed as arguments (or objects reachable from them) but has no access to the Rust compartment’s stack or globals. This makes it much easier to reason about the impact of bugs in C code when adopting Rust.

You can take this even further and restrict the permissions on pointers passed from Rust to C. Do you want C code to be able to read a Rust object or object graph? Or perhaps modify an object but not capture a pointer to it (similar to borrow semantics)? The hardware can enforce these properties.

This lets you get all of the great benefits of Rust from the very first function that you write in Rust, rather than having to wait until you’ve rewritten everything in Rust.

What are we trying to do?

Initially, we aim to make Rust work as a source language for targeting CHERIoT. CHERIoT is an embedded platform, so the no_std + alloc mode for Rust (you can dynamically allocate memory, but can’t use most of the standard library) makes sense as a target. This will make it easy to port existing embedded Rust code to CHERIoT and to write new Rust components. As the project runs, there will be a lot of quality-of-implementation work to ensure that it’s in a state to upstream and, until then, in a state where we can support it.

The next step in the project is to make sure that Rust is a first-class citizen of the CHERIoT platform. We have a set of C/C++ language extensions to provide rich compartmentalisation features. Rust will need equivalents of these. A direct port of the features is quite easy and works as a minimum viable product, but we’d also like to make sure that these work as idiomatic Rust: you shouldn’t have to write C-like Rust to use CHERIoT features.

Finally, there are several things in the Rust type system that can be dynamically enforced in CHERIoT. In current Rust, every call to non-Rust code must be unsafe. I hope that we’ll be able to relax this requirement and have the compiler enforce Rust properties by removing permissions from pointers before calling C.

There are a lot of subtleties here. The authors of Tock discovered that exposing Rust to untrusted code has some issues and provided solutions that worked in their specific context. I’m optimistic that the richer substrate of the CHERIoT ISA gives the compiler more tools to provide generic solutions to these issues.

The funded project is specifically scoped to CHERIoT, but we’re building on work from Morello and hope to make it easy to support other CHERI platforms.

Rust enables verification

The Verus project builds language-integrated formal verification tools in Rust. This is particularly interesting for core parts of CHERIoT RTOS. The lowest level parts of an operating system are tricky for safer systems languages because the things that they do are intrinsically unsafe. They must be able to use escape hatches that opt out of parts of the safety guarantees of a language like Rust because they are the things that implement those guarantees.

You can think of rich type systems as off-the-shelf verification tools (they define some generic properties and prove them for every program, and reject those for which they can’t prove them), whereas the lowest-level parts of systems code need bespoke verification to prove one-off sets of properties. There was some great work at SOSP this year showing how Verus can prove key aspects of a kernel and I’m excited to see how far we can get with this in CHERIoT.

Where will this project live?

The CHERIoT Rust project has its own web site which is where we’ll post more public information. The compiler repository is public, but is not yet ready for general use. We’ll post calls for testing when it reaches an early preview state. As with any other aspect of CHERIoT, contributions are welcome. Please join our public Signal chat if you want to discuss the project or help out!

CHERI is an abstract architecture

CHERI is a conceptual extension to conventional computing architectures define a capability model for accessing memory within an address space. There are a lot of concrete instantiations of this conceptual model. The original CHERI research was done on a 64-bit MIPS variant. Arm Morello extended ARMv8 with CHERI extensions. The University of Cambridge CHERI ISA v9 Technical Report contains a RISC-V CHERI adaptations and even a sketch of CHERI for x86. And there is an ongoing effort to standardise RISC-V CHERI base architectures.

These all provide the same core set of features. Most software isn’t written for a single target instruction-set architecture (ISA), it’s written assuming a set of language-level abstractions that can be represented in a particular target. Software written for CHERI in C (or any language that is higher-level than C) doesn’t care the bit patterns of capabilities, how bounds or permissions are encoded, and so on. It cares only that you have some capability registers, that you can store capabilities in memory, and that the C language notion of a pointer can be represented by a CHERI capability for the target ISA.

I like to use memory-management units (MMUs) as an analogy for CHERI. MIPS and SPARC had software-managed translation-lookaside buffers (TLBs). x86 and Arm had architectural page tables that hardware walked to fill the TLB automatically on miss. PowerPC had an MMU design that, in the words of Linus Torvalds, ‘can be used to frighten small children’. All of these implemented the same set of abstractions in terms of processes, shared memory, copy-on-write regions, and so on. All of these are MMU designs, just as there are many possible CHERI implementations.

There are lots of things like this in ISAs. Arm’s Neon, PowerPC’s AltiVec, and x86 SSE are all SIMD extensions, which provide similar programming models, but they all have quite different low-level details. It’s quite possible to write code that targets all of them.

CHERIoT is a concrete platform

CHERIoT is a complete ISA specification, co-designed with a software stack. In the same way that x86 page tables or MIPS’ software-managed TLB are both concrete instances of the abstract idea of a memory-management unit, CHERIoT is a concrete instantiation of CHERI.

This means that there is no ‘CHERI or CHERIoT’ choice. The real choices are between CHERI or not CHERI, or between CHERIoT and some other concrete instantiation of CHERI. Having worked on CHERI since 2012, I am obviously biased on the first of these: given a choice between CHERI and not-CHERI, you should definitely pick CHERI! The second choice (CHERIoT or some other CHERI variant) is more nuanced, as I’ll discuss more in the rest of this post. There are good reasons for CHERIoT but there are also domains where it is not the best choice.

CHERIoT is a rich CHERI platform

A minimal CHERI specification provides pointer integrity, bounds enforcement, and some permissions. Pointer integrity means (among other things) that you can precisely differentiate between pointers and other kinds of non-pointer data in memory and so can build temporal safety on top. Most of the CHERI temporal safety work has provided users with protection against use-after-reallocation issues. A pointer in a CHERI system may point to deallocated memory, but it will not point to reallocated memory. Something in the system will ensure that deallocated pointers are gone before the memory allocator reuses memory. This means that any use-after-free bug will either point to the old object, or will trap. It will never point to a new object and cause accidental aliasing.

CHERIoT builds on top of the core CHERI ideas with a couple of hardware features that guarantee deterministic trapping for use-after-free bugs. CHERIoT also provides a rich sealing design that works with our software model. It extends the sentry mechanism in CHERI systems to provide stronger control-flow integrity properties than most other CHERI platforms, and also uses them to enforce interrupt control.

The entire system is designed to allow fine-grained auditing of compartment rights, which lets us respect the principle of least privilege by having compartments as small as a single function and statically reasoning about what they can do in the presence of a compromise. This set of building blocks lets us do things like privilege separate the network stack and make callers respect the principle of intentional use, again with auditable guarantees that let you reason about which compartments can talk to which remote servers.

This adds up to an easy-to-use programmer model and richer security guarantees than, we believe, any other system (CHERI or otherwise).

CHERIoT is designed for resource-constrained systems

Some of the strengths of CHERIoT come from the fact that we were willing to tailor the system to small devices. Prior CHERI systems rely on the MMU to support concurrent revocation as their mechanism for temporal memory safety. CHERIoT does not include or need an MMU, it uses an alternative approach for temporal safety, inspired by our previous CHERI+MTE work

You don’t want an MMU on an embedded device. An MMU is typically larger than an entire microcontroller core! Page tables are large data structures and they enforce a minimum granule size for memory protection and accounting. A minimal process isolated with a typical MMU needs one page for the stack, one page of code, and one for read-write globals. On top of that, it needs a complete page table, which is at least two pages. This means you need a minimum of 20 KiB for an isolated component, of which 8 KiB is purely book-keeping overhead. On a CHERIoT system, a lot of our compartments are smaller than the page tables that would be necessary to isolate them. In addition, MMUs bring nondeterminism, which is not ideal in any system with even fairly soft realtime requirements.

The temporal memory system in CHERIoT is designed to scale across the range of microcontroller cores, including low-core-count multicore systems. We check whether capabilities are valid when you load them. Most pointers are used more than once, so this is more efficient than checking on every use (as an MTE system must do), but does not scale as well to large multicore systems or complex memory hierarchies. You would not want to build a temporal-safety system like ours for a large server system, for example.

CHERIoT also extends CHERI’s sealed function pointer (“sentry”) mechanism for interrupt control. This has some enormous benefits for the programmer model. It is quite easy to implement on microcontroller-class systems, even dual-issue designs such as CHERIoT-Kudu. It would be very hard to scale to big out-of-order cores.

And that’s all fine, for the same reason it’s fine to run different operating systems on different scales of core: different abstractions and different optimisations make sense at different scales. By the time you’ve built a system where you start to run into scalability challenges with CHERIoT, you have enough area that an MMU doesn’t add much overhead and enough RAM that you would benefit from running an OS designed for larger computers.

What about standard RISC-V CHERI?

The RISC-V standardisation process has an unenviable task of trying to create a standard base for RISC-V CHERI systems that will work for 2-stage microcontrollers, 25-stage superscalar out-of-order server chips, GPUs, AI accelerators, SmartNICs, and so on. This is likely to end up being a (hopefully small) family of bases and a set of extensions.

If all goes well, CHERIoT v2 will be one of those bases plus some extensions. RVB26 will be a RISC-V profile assembled from those bases and a set of extensions necessary to run a CHERI Linux or CheriBSD.

As with the rest of RISC-V, any useful chip implements a base architecture and a set of extensions. Profiles exist to corral a set of extensions that software can depend on in environments where binary compatibility is important, such as those running off-the-shelf operating systems and programs.

So when should I use CHERIoT vs some other CHERI?

If you are looking for a microcontroller-class system, CHERIoT is probably the right answer. It is co-designed with a compartmentalisation model and a rich set of software abstractions (implemented in CHERIoT RTOS), and provides temporal safety as a baseline feature. It is a mature and stable target, supported by multiple organisations.

If you are looking for an application core, CHERIoT is not for you. There are a few cases where people have moved to using Linux on an Arm A-profile system and could run the same workload on a cheaper CHERIoT system, but don’t expect to be able to run arbitrary Linux workloads on a CHERIoT system (ever). The upcoming CHERI RISC-V profile is a far better choice for these use cases. Capabilities Limited and lowRISC are implementing this in the open-source CVA6 core, so there will soon be a path to a useful open-source CHERI core in this part of the design space. Codasip’s X730 core is also available for commercial licenses and is targeting this spec (and pre-standard versions of it until it is ratified). Hopefully there will be more implementations available in coming years.

If you are looking for an ISA as a base for an accelerator or coprocessor, CHERIoT may be the right choice, depending on the rest of the system. Depending on your requirements, you may want to do something completely custom. As long as it provides the same underlying security guarantees, it can still be CHERI!

A quick intro to sealing

CHERI capabilities are most commonly used as pointers (they can also be used for very coarse-grained sandboxing, with a model similar to WebAssembly). Each capability is an address plus some metadata, protected by guarded manipulation. The CPU enforces properties on how capabilities can be manipulated (for example, their bounds can be shrunk, but not expanded). Importantly, it also checks the metadata before the capability can be used for any operation. For example, if you use a capability as the base for a load instruction then the core will check that the entire width of the load is within the bounds and that the capability has load permission.

The metadata for a CHERI capability has an object type (otype) field. If you’re just using capabilities to represent pointers in unmodified C/C++ code, then you will only ever see capabilities with zero in their otype field. This represents an unsealed capability.

The CSeal instruction (which has different spellings in some CHERI variants) combines two capabilities. One is a ‘normal’ pointer-like capability. The other, the sealing key, has the permit-seal permission and its address (sometimes referred to as ‘value’ or ‘cursor’) does not represent a memory location, but instead represents a value in the space of types. The result of sealing is a copy of the pointer-like input with its otype field set to the address of the sealing key, making it a sealed capability.

You can pass this around just like any other pointer, but you can’t use or modify it. If you try to use it as the base for loads or stores, it will trap. If you try to modify it (the address or the metadata), you will get an untagged (invalid) capability.

If you use that capability as one operand to CUnseal and provide an unsealing key (a capability with permit-unseal permission) that has the address from the sealed capability’s otype, you get back the original pointer-like capability. This lets you pass untrusted code a pointer to something that they can pass back but can’t tamper with.

Sealing gives type safety

The object type in a CHERI capability doesn’t necessarily have to have a 1:1 mapping to a language-level type. It’s quite common for a set of types to have some kind of internal type discriminator. For example, in our ASPLOS 2017 CHERI JNI paper, we used three otypes for everything passed from the JVM to native code. One each for field-ID and method-ID structures, and one for all Java objects. Each Java object starts with a pointer to its class, so we didn’t need one otype for each Java class type, just one for all Java objects. The virtualised sealing mechanism in CHERIoT uses this same approach to multiplex a huge number of possible object types onto two hardware otypes (one for statically allocated objects and one for dynamic, so the memory allocator is not in the TCB for statically allocated sealed objects). More on this later.

If you couple a sealing key type with a language-level type (or family of self-disambiguating types), then you can build type safety that works in the presence of an attacker. It’s common in C/C++ to expose an opaque type at API boundaries. This is usually implemented as a pointer to forward-declared structure type. C and C++ will not let you dereference this pointer unless you cast it to another type. CHERI sealing lets you enforce the no-dereference rule, even if malicious code does cast it to some other type. This means that you can hand sealed capabilities to other compartments and they must treat them as opaque types. When you get them back, you have a lightweight check that they really are the type that you expect.

After the 2015 paper, we separated the permit-seal and permit-unseal permissions. In most CHERIoT use cases, the entity that can seal and unseal pointers with a particular otype is the same. This isn’t universal. C++ provided one of our original use cases for separating them. If you seal C++ vtables with a well-known otype, and make the permit-unseal capability for it available anywhere, then you can have a C++ ABI where only the loader can forge vtables. This makes code reuse attacks harder.

There are other situations where an object type is used for integrity but not confidentiality. It is an attestation that some software trusted to seal with a particular otype has done so, even if anyone is then able to unseal the result. The sentry mechanism (discussed later) is a variant of this idea.

Virtualised sealing shares hardware object types

Morello had a 15-bit object-type field, which is sufficient for a lot of things. When we scaled CHERI down to 32-bit systems for CHERIoT, we ended up with only three bits of space. Of these, the zero value means not-sealed, so there were only seven bit patterns available. Even simple embedded software typically has more than seven types. CHERIoT systems usually have more than seven compartments, so seven types isn’t even enough for one type per compartment (and some compartments wish to offer more than one sealed type). Because they are (mostly) used for different purposes, we differentiated executable and data capabilities, so 3 bits actually gives 14 sealing types, seven for data and seven for sentries (see later).

The virtualised sealing mechanism was designed to work around this shortage of sealing types. We reserve two of the object types for objects that are instances of a structure with the following layout:

struct SealedObject
{
    /// The sealing type for this object.
    uint32_t type;
    /// Padding for alignment
    uint32_t padding;
    /// The real data for this.
    char data[];
};

The first 32-bit word is the type, followed by 32 bits of padding, and then the real object.

Our virtualised mechanism takes advantage of the fact that we can represent a lot more values in the address field (32 bits) of a capability than in the otype field (three bits). Any address in a permit-seal or permit-unseal capability that cannot fit into the otype field is available to authorise sealing or unsealing with the virtualised mechanism. This gives us almost four billion types that we can represent with the virtualised mechanism, at the expense of being able to seal only complete objects and of each object having a single type. This is fine for most use cases, though language VMs on bigger systems would benefit from at least a few hardware otypes. We also reserve two data otypes for future use, so have some flexibility going forward.

The CHERIoT token_unseal function unseals the object using the hardware sealing key and then compares the virtual otype in the header to the type of the permit-unseal capability passed as the key. If they match, it returns an unsealed capability to the object without the header, so the caller doesn’t ever have access to the header.

Sealed pointers using the software sealing mechanism always point to the end of the header. The header must be strongly aligned, which means that the low three bits are unused. We use these to implement three software permissions. As with the CHERI permissions, these can be cleared but not set.

Type safety gives easy-to-use handles

A lot of CHERIoT compartments use type-safe pointers as handles. For example, if you open a socket, you get back a sealed capability to that socket’s state. The same thing happens for message queues, TLS sessions, and so on.

In a conventional monolithic kernel, these would all be handles or file descriptors looked up in some table in the kernel. In a microkernel, they’d either be indexed per caller, or managed via some handle-manager service. In CHERIoT, these are simply opaque pointers. If these are allocated from the heap, they also benefit from the temporal safety mechanisms: Freeing the object that a handle refers to invalidates all handles, without requiring any further synchronisation.

Exactly the same abstractions that you use for data hiding in good API design work between trust domains, with a little bit of hardware help. This is great for programmers because you can easily retrofit a security boundary. If you have built an API around exposing opaque types, turning it into a robust security boundary simply means doing a few checks on the public APIs.

Type safety gives static software-defined capabilities

The same sealing mechanism that provides type safety for dynamic allocations also works for static objects. You can create a CHERIoT compartment that has access to a sealed object where another compartment owns the sealing key. The contents of these show up in the audit log.

We use these throughout the system to implement software capabilities. CHERI (hardware) capabilities are unforgeable (delegable) tokens of authority to perform some architectural action; they authorize operations carried out on your behalf by the hardware. CHERIoT software capabilities are unforgeable (delegable) tokens of authority to perform some software action; they authorise operations carried out on your behalf by some other compartment.

The first use of these most programmers will make is to allocate memory. If you call malloc, this is a thin compatibility wrapper around heap_allocate, which requires an allocator (software) capability as an argument. The allocator capability is a sealed object that contains a quota. It authorises you to allocate memory until your quota is exhausted. It also authorises you to free objects that were allocated from your quota and reclaim the quota. Similarly, when you create a connected TCP socket, you pass a capability that authorises you to connect to a specific host and port.

These static capabilities can be inspected with CHERIoT Audit so you can write policies that say things like ‘these four compartments, between them, can’t allocate more than 16 KiB of RAM’ or ‘this compartment may connect to my cloud back end, but nowhere else’.

Sealed handles give trivial flow isolation

Sealed capabilities are passed out and back as opaque values and can be unsealed without any global state beyond a constant (un)sealing key. A lot of compartments that use them have no global mutable state. If you pass a sealed capability to a TLS session into the TLS compartment, that compartment can unseal it and then see the state of your TLS session.

The thread operating on your TLS session, while within the TLS compartment, does not gain access to capabilities to other TLS sessions. If an attacker gains arbitrary code execution in the TLS compartment while operating on one TLS flow, that doesn’t give them the ability to attack another one.

The same applies to much simpler compartments. The message-queue compartment, which provides secure message queues and streams between compartments, has the same property. The only state that it operates over is reached via the sealed queue pointer or one of the arguments. If you dynamically compromise this compartment, you can tamper with a queue that you hold an endpoint handle for, but not any other queue.

Sentries build on sealing

CHERI also has a notion of a sealed entry (sentry) capability. These are sealed capabilities with a special otype that allows them to be used as jump targets. When you jump to a sentry, it is implicitly unsealed and installed as the program counter. PC-relative loads then let the jumped-to code retrieve capabilities inaccessible to the code that held the sentry.

In CHERIoT RTOS, all library functions, including the switcher (which handles cross-compartment calls) are provided as sentries.

Morello had several variants of sentries, including some designed for descriptors where the sentry was actually a pointer to a pair of a code and data capability. The jump would load the data capability and branch to the code capability. This underlying mechanism is very flexible and there is a lot more research to be done on how it can be used in the future on big systems.

CHERIoT provides rich sentries

CHERIoT uses sentries to control interrupts. Rather than the usual paradigm of software explicitly managing the interrupt enable status bit, we have sentry variants that explicitly enable or disable interrupts when you jump to them. This encourages a more structured style of code for interrupt management. Moreover, the interrupt status of a function is visible in the audit report, so it is possible to see which compartments are able to call which interrupt-disabling functions. This is one of the ways in which CHERIoT is very much designed for embedded systems. Implementing this feature is moderately easy on in-order pipelines, but would be much harder on out-of-order machines. This is also one of the strengths of RISC-V: we can add extensions that are desirable in part of the hardware design space, but not everywhere.

CHERIoT also differentiates forward and backwards-edge sentries. Function pointers and return addresses use different otypes. This means that you cannot trivially replace a return address on the stack with a function pointer for control-flow hijacking. This kind of attack is hard on a CHERI system anyway, but this provides some defence in depth. Credit to folks at the Microsoft Security Response Center for recommending this and to Murali Vijayaraghavan at Google for coming up with a way of adding it without invasive changes to the ISA.

When you jump to a forward-edge sentry with a jump-and-link instruction, the link register is a return sentry that captures the previous interrupt state (enabled or disabled). This means that, at least for leaf functions, we can enforce structured programming for control over interrupt state.

The forward-edge sentries are an attestation from the RTOS to the running software that this is a valid function pointer. The backwards-edge sentries are an attestation from the hardware that this is the result of executing a jump-and-link instruction.

CHERIoT uses sealing inside the RTOS

In addition to the otypes exposed directly to programmers, the RTOS reserves two for internal use. When you do a cross-compartment call, the compiler will insert a call to the switcher (via the switcher sentry). The function pointer for such a call is a sealed data capability to the target compartment’s export table. The base of this will point to the program counter and global pointer register values for the target. The address will point to the metadata describing this entry point. The switcher unseals this and so knows that this really is a cross-compartment entry point provided by the loader.

When the switcher takes an interrupt, it will spill the register file and pass the scheduler a sealed capability to the register-save area. The scheduler then returns a sealed capability of the same type and the switcher restores the register state from there. This ensures that the scheduler never sees the state of interrupted threads, it has only opaque tokens allowing it to choose the next thread to run.

Hopefully this short guided tour has given you some idea of both how powerful a mechanism sealing is, and how pervasive it is in the CHERIoT platform.

This specification defines the ISA, the CHERIoT language extensions, compilation model, relocations, and so on. The last change that we made to the ISA was in December 2024, so we are confident that this is a stable release that we can support in hardware for a long time. This specification was implemented by the 1.0 release of CHERIoT Ibex and by CHERIoT Kudu (which has not yet had an official release). These two implementations demonstrate that the ISA scales from three-stage single-issue pipelines to six-stage dual-issue pipelines, roughly the same range of microarchitectures supported by Arm’s M profile.

We at SCI have the first of our ICENI chips, which use the CHERIoT Ibex core, on the way back from the fab now and will be scaling up to mass production in the new year. I am not allowed to speak for other folks building CHERIoT silicon, but I expect 2026 to be an exciting year for the CHERIoT project!

This is a release that, both through the open-source CHERIoT Platform project and through partner companies that aim to ship CHERIoT products, we will be supporting for years to come.

Alignment with the RISC-V Y base

RISC-V International is currently standardising a CHERI base for RISC-V (tentatively named Y, alongside the existing I and E bases). We aim for this to be a common subset across all CHERI implementations, whether they are microcontroller, application cores, or domain-specific accelerators. The exact composition of this base is still under discussion, but we aim for CHERIoT 2.0 to be source-compatible with (and with equivalent functionality to) CHERIoT 1.0, but built atop this common base.

On the way to 2.0, we have been separating out the parts that are in the common core from those that are CHERIoT-specific. We also have the ct. prefix reserved for the CHERIoT vendor extensions in RISC-V and so will move any instructions that are not direct equivalents of standardised RISC-V instructions into that namespace. For compatibility, our toolchains will support both names, but we expect to transition to the RISC-V official names as the RISC-V standardisation process makes progress.

Expect to see some 1.x releases that make changes in things like assembler mnemonics (but not the underlying ISA).

Future plans beyond 2.0

No ISA specification is ever complete. Our 1.0 release is a solid foundation, not an endpoint.

We already have plans to integrate with the Zfinx and Zdinx extensions, which allow 32-bit and 64-bit floating point values to be held in integer registers on RISC-V. On CHERIoT, a 64-bit floating-point value could live in a single capability register (as, indeed, it is in the CHERIoT soft-float ABI). This avoids increasing compartment and context-switch times (which comes from a larger register file, and are why we used RV32E, not RV32I as our base), while still allowing hardware floating point acceleration.

We have two candidates for alternative bounds encodings, which each have different microarchitectural tradeoffs. They are 100% backwards compatible with the software and so we aim to add both to a future version of the specification. We have designed our ABI carefully to avoid leaking capability encoding details into the software model, which means that we can support implementers picking any of the three encodings, on the products where each makes sense, along with any vendor-specific variants that they may wish to support. Supporting different encodings does not require changes even to core RTOS code and the two proposed modifications do not require toolchain changes either, though vendor extensions may.

We’ve been exploring for a while the idea of having two integer subregisters for each capability. This would give us 30 integer registers or 15 capability registers, which would slightly reduce stack spills. This involves some complexity in the microarchitecture and in the compiler, so it isn’t an automatic win and there’s a lot more work to decide whether it’s desirable.

The current drafts for the RISC-V official specification provide a thread ID register. We’ve been considering how we could replace our use of the global-pointer register with this, to make it a compartment identifier. The specification also reserves loads and stores with the zero base register (which will always trap on a CHERI system) and so we could use these for GP-relative loads and stores.

Similarly, the current draft reserves space for a branch-if-tag-[not-]-set instruction, which would be quite commonly used in our software model. The RISC-V code-size extensions have push and pop / popret instructions, which are likely to want some adaptation to be most useful with CHERIoT.

And there are also a lot of interesting ideas in research. For example, BLACKOUT provided a way of building a data-centric constant-time programming model on top of CHERI. There’s a bit of work to see how to integrate that with CHERIoT but being able to have a good programmer model for constant-time programming would be worth it for a later release.

There are a lot of possible future directions, and we expect to be adding to the spec for a while. Having a solid core means that we can do this without breaking backwards compatibility. And, as with everything in CHERIoT today, future additions to the ISA will always be motivated by software requirements and careful consideration of the programmer model and the microarchitecture.

Sorry for the wait!

This is a slightly embarrassing post to write, because I thought we’d done a 1.0 release several months ago: The ISA has been sufficiently stable for a 1.0 stamp for a while. Unfortunately, when we initially thought about it, there was a bug in the Sail to LaTeX converter and so we postponed the release until we could generate a nice specification PDF. That bug was fixed quite quickly, but by then I’d forgotten that it had been the release blocker and thought we’d done this release back in the Spring.

Approaching a year without any ISA changes makes me even more confident than I was back in the Spring that this is a release that we can support for the next decade.