CHERIoT Programmers' Guide

David Chisnall

Table of contents

6. Compartments and libraries

Compartments in CHERIoT are somewhere between libraries and processes in mainstream operating systems. They have private code and globals and constitute a security boundary. They can export functions to be called form other compartments and can call functions exported from other compartments.

Libraries are a lightweight way of reusing code without duplicating it into different compartments. Calling a library function does not involve crossing a security boundary. Libraries contain code and read-only data but do not have mutable globals. It is possible for libraries to hold secrets but, unless library functions are written in very careful assembly, they should assume that any (immutable) globals in the library can leak to callers. Each library entry point is exposed as a sentry capability (see Section 2.4. Sealing pointers for tamper proofing) to the callers, which means that the caller cannot directly read its code or (immutable) data.

If a library traps, the error handler for the caller compartment may see the register file for the middle of the library. Similarly, the compiler may spill arbitrary values onto the stack or leave them in registers at the end of a library function. As such, you should assume that anything processed in a library written in a compiled language will leak to the caller and anything written in assembly must be very careful to avoid leaking secrets. This is not normally a problem because most libraries just exist as an alternative to compiling the same functions into multiple compartments. For example, the functions that implement locks on top of futexes (see Section 7.5. Waiting for events with futexes) are in a library to reduce overall code size, but simply copying the implementations of these functions into each caller would have no security implications.

6.1. Compartments and libraries export functions

In a UNIX-like system, a shared library can export any kind of symbol. This includes functions and global variables. In CHERIoT, compartments and libraries can export only functions as entry points. Global variables are always private to a compartment or library, unless a pointer is explicitly passed out as a function argument or return in a cross-compartment call. This design is intended to make it easier to reason about sharing between compartments.

If you declare a global in a header and define it in a library or a compartment, you may see linker errors if you try to use it in other compartments or libraries. This holds even for const globals exported from libraries. You can place a static const global in a header for a library, but that will introduce tight coupling: the value in the header may be inlined at any use site. For very large globals, this may also increase code size significantly.

As mentioned previously, (read-only) globals in a library are hidden in a software-engineering sense, but may be leaked to callers and should not be considered private in a security sense.

You can still use globals to share data but you must explicitly expose them via an accessor function. This makes CHERIoT compartments and libraries similar to Smalltalk-style objects, with public methods and private instance variables.

If you expose an interface that returns a pointer to a global, you can use CHERI permissions to restrict access. Returning a read-only pointer to a global is a common idiom for building a lightweight broadcast communication channel. The owning compartment can write to the global and other compartments can read from via their copy of the pointer, with guarantees that only the owning compartment is making changes.

To see the differences, Listing 29 shows a header that exports two functions, one from a compartment and one from a library.

/**
 * A simple example library function.
 */
__cheri_libcall void library_function();

/**
 * A simple example compartment function.
 */
__cheri_compartment(
  "compartment") int compartment_function();

Listing 29. A header defining library and compartment exports.examples/library_or_compartment/interface.h

The exported functions both contain the implementation shown in Listing 30, which uses the debug APIs (see Chapter 9. Features for debug builds) to print the capabilities in three registers. These show the stack, code, and globals regions, respectively. The compiler provides builtin functions to copy two of these (the program counter and stack capabilities) but the third, the globals pointer, requires some inline assembly. The inline assembly needs to run at the start of the function because this function does not reference any globals and so the compiler will otherwise spill this register to use it as a temporary.

	register void *cgp asm("cgp");
	asm("" : "=C"(cgp));
	// Print the stack capability from within the library.
	Debug::log("Stack pointer: {}",
	           __builtin_cheri_stack_get());
	Debug::log("Program counter: {}",
	           __builtin_cheri_program_counter_get());
	Debug::log("Globals pointer: {}", cgp);

Listing 30. A simple print function to introspect compartment state.examples/library_or_compartment/library.cc

When you run this code, you should see output something like this:

Entry: Stack pointer: 0x80000d30 (v:1 0x80000750-0x80000d50 l:0x600 o:0x0 p: - RWcgml -- ---)
Entry: Program counter: 0x80005f18 (v:1 0x80005ee0-0x80005fe8 l:0x108 o:0x0 p: G R-cgm- X- ---)
Entry: Globals pointer: 0x80006202 (v:1 0x80006200-0x80006204 l:0x4 o:0x0 p: G RWcgm- -- ---)
Library: Stack pointer: 0x80000d20 (v:1 0x80000750-0x80000d50 l:0x600 o:0x0 p: - RWcgml -- ---)
Library: Program counter: 0x80005d48 (v:1 0x80005d20-0x80005df8 l:0xd8 o:0x0 p: G R-cgm- X- ---)
Library: Globals pointer: 0x80006202 (v:1 0x80006200-0x80006204 l:0x4 o:0x0 p: G RWcgm- -- ---)
Compartment: Stack pointer: 0x80000cf0 (v:1 0x80000750-0x80000d10 l:0x5c0 o:0x0 p: - RWcgml -- ---)
Compartment: Program counter: 0x80005e20 (v:1 0x80005df8-0x80005ee0 l:0xe8 o:0x0 p: G R-cgm- X- ---)
Compartment: Globals pointer: 0x800061fe (v:1 0x800061fc-0x80006200 l:0x4 o:0x0 p: G RWcgm- -- ---)

Ignore the exact memory addresses, these may change depending on where you run the example. First, note that the program counter capability (the capability used for instruction fetch) is different in all three cases and the bounds do not overlap. In this particular built, the example compartment is placed in code memory immediately after the entry compartment, so the address 0x80005df8 is the boundary between the two.

Next, observe that the globals pointer is different between the two compartments. This example includes a single int global, to make sure that these are non-zero (two compartments may have the same zero-length capabilities for globals if they have no globals). In contrast, the library prints the same globals pointer as the caller. As previously mentioned, a malicious library can access any globals in the caller.

Finally, look at the stack pointers. The first thing to note is that the start address is 0x80000750 in all cases. Stacks grow downwards and the end of the stack is the same for each. The address of the stack pointer is different in each because they run at different depths on the stack. The initial value is very close to the top of the stack, then the library and compartment calls are deeper. A compartment call needs to save more state (two callee-save registers and the old global pointer, which are preserved across normal calls) and so entry to the compartment call is slightly (48 bytes) lower. The top of the stack is the most interesting place. In the compartment call, the top of the stack is at address 0x80000d10, 64 bytes below its location in the entry compartment and the library. Anything above that point is unreachable in the compartment before then.

This may all be easier to understand visually. Figure 4. An illustration of memory pointed to by each register in the compartment-call example shows the memory ranges that each compartment points to, in each of the three places. The lines marked with a ⓵ indicate the initial values on entry, those marked ⓶ indicate the values in the library call and those marked ⓷ show the values in the cross-compartment call.

An illustration of the memory pointed to by each compartment in the compartment and library-call example.

Figure 4. An illustration of memory pointed to by each register in the compartment-call example.

6.2. Understanding the structure of a compartment

From a distance, a compartment is a very simple construct. The core of a compartment is made of just two capabilities. The program counter capability (PCC) defines (and grants access to) the range of memory covering the compartment's code and read-only globals. This has read and execute permissions. The capability global pointer (CGP) defines (and grants access to) the range of memory covering the compartment's mutable globals. The full structure is more complex and is shown in Figure 5. The structure of a compartment.

The structure of a compartment. The PCC register points a code region that contains code, read-only globals, and an import table. The CGP register points to the read-write globals region. An export table is not directly reachable.

Figure 5. The structure of a compartment.

A future version of the ABI may move read-only globals out of the program counter capability region but this requires some ISA changes to be efficient and so will likely not happen before CHERIoT 2.0.

If a compartment didn't need to interact with anything else, these two regions would be sufficient. In practice, compartments are useful only because they interact with other compartments or the outside world. The read-only data region contains an import table. This is the only region of memory that, at system start, is allowed to contain capabilities that grant access outside of the PCC and CGP region for the compartment. The instructions for the loader to populate these are in the firmware image and are amenable to auditing.

The import table contains three kinds of capabilities. MMIO capabilities are conceptually simple: they are just pointers that grant access to specific devices. This mechanism allows byte-granularity access to device registers and so it's possible to provide a compartment with access to a single device register from a large device.

Import tables also contain sentry capabilities for library functions. A shared library has its own PCC region (like a compartment) but does not have a CGP region. Library routines are invoked by loading the sentry from the import table and jumping to it.

Finally, import tables contain sealed capabilities referring to other compartments' export tables. If a compartment exports any entry points for other compartments to call, it has an export table. This contains the PCC and CGP for the compartment and a small amount of metadata for each exported function describing:

This is all of the information that the switcher needs to transition from one compartment to another.

Extracting code and moving it to a new compartment adds a very small amount of memory overhead, on the order of a dozen words for a typical compartment.

6.3. Adding compartments to the build system

The build system makes adding compartments trivial. An xmake build file (xmake.lua) uses a declarative Lua-like syntax at the top level. This defines targets as sections. Listing 31 shows the build system logic for the example compartments and libraries from earlier. Defining a new target implicitly ends the definition of the previous one. This example implicitly ends the library and entry targets, but uses target_end() to explicitly end the compartment target.

-- An example compartment that we can call
compartment("compartment")
	add_files("compartment.cc")
target_end()

-- An example compartment that we can call
library("library")
	set_default(false)
	add_files("library.cc")

-- Our entry-point compartment
compartment("entry")
	add_files("entry.cc")
	add_deps("compartment", "library")

Listing 31. Build system code for defining compartment and library targetsexamples/library_or_compartment/xmake.lua

Inside each target definition, you can add files, dependencies on other targets, and so on. The "entry" target, for example, sets itself as a non-default target. xmake will build every default target, whether it is used or not. Marking a target as non-default allows it to be defined, but built only if it is used. This is useful for reusable components. The RTOS provides build-system logic for a set of libraries, but each is built only if it is added as a dependency of something that is built.

The compartment, library, and firmware target markers are syntactic sugar over the xmake target command. The first two simply set the default rules to build as the correct kind of target. The firmware target definition is more complex because it also implicitly instantiates core parts of the RTOS, constructs the linker script, and so on. If you are reading the xmake documentation, simply treat these as if they were target definitions.

Once you have defined the rules to build each compartment and library, they need to be combined into a firmware image. Any library or compartment that is a direct or indirect dependency of a firmware image will be built and linked into the final image. For this example, we show both forms. The entry compartment calls the library and the example compartment and so lists them as explicit dependencies. In the firmware definition (Listing 32) adds the entry compartment as an explicit dependency, which then pulls in the other two. This firmware also depends on two libraries from the core RTOS, the freestanding library, which provides the core of the C run-time environment, and the debug library which is pretty-printing the debug messages.

-- Firmware image for the example.
firmware("library_or_compartment")
	-- RTOS-provided libraries
	add_deps("freestanding", "debug")
	-- Our compartments
	add_deps("entry")

Listing 32. Build system code for adding dependencies on compartment and library targetsexamples/library_or_compartment/xmake.lua

6.4. Choosing a trust model

There are three trust models that are commonly applied to compartments:

Sandbox
A sandbox is a compartment that is used to isolate untrusted code. This model is used to protect the rest of the system. Typically, a sandbox will trust values passed to it as arguments to exported functions or return values from functions that it calls in other compartments.
Safebox
A safebox is a compartment that holds some secret or sensitive data that must be protected from the outside. For example, a safebox may be used to protect a key and perform encryption or signing on behalf of callers. A safebox does not trust any data provided from outside of the compartment, but callers may trust it to behave correctly.
Mutual distrust
Mutual distrust is the strongest model. A compartment in a mutual-distrust relationship protects itself from attacks from the outside by careful handling of inputs and expects other compartments to protect themselves from it in the same way.

This is the start of defining a threat model for your code. A compartment may simply be used for fault isolation, to limit the damage that a bug can do. You may assume that an attacker will be able to compromise some compartments (for example, those directly processing network packets) and defend yourself accordingly.

In the core of the RTOS, the scheduler is written as a safebox. It does not trust anything on the outside and assumes that everything else is trying to make it crash. The memory allocator is also written as a safebox, assuming that everything else is trying to either make it crash or leak powerful capabilities. For some operations, the scheduler invokes the allocator. The scheduler trusts the allocator to enforce heap memory safety. It does not, for example, try to check that the memory allocator is returning disjoint capabilities (it can't see every other caller of heap_allocate, and so couldn't validate this). It is; however, written to assume that other compartments may try to maliciously call allocator APIs to cause it to crash.

When thinking about trust, it's worth trying to articulate the properties that other code is trusted to enforce or preserve. For example, everything in the CHERIoT system trusts the scheduler for availability. Most things trust the allocator to enforce spatial and temporal memory safety for the heap.

6.5. Implementing a safebox

The safebox abstraction is trivial to implement in CHERIoT. Listing 33 shows a complete implementation of a simple safebox for a guess-the-number game. This generates a (very weak!) pseudorandom number using the cycle counter and allows callers to guess it.

using Debug = ConditionalDebug<true, "Safebox">;

bool check_guess(int guess)
{
	static int secret = rdcycle64() % 10;
	if (guess != secret)
	{
		Debug::log(
		  "Guess was {}, secret was {}", guess, secret);
		secret = rdcycle64() % 10;
		return false;
	}
	return true;
}

Listing 33. A safebox for a guess-the-numbers gameexamples/safebox/safebox.cc

This is called from the runner compartment, shown in Listing 34. This compartment talks directly to the outside world and so is part of the attack surface. It's quite unlikely that a compartment that simply reads individual bytes and ignores anything not in the ASCII digit range could be vulnerable but the same techniques can protect much more realistic examples.

using Debug = ConditionalDebug<true, "Runner">;

__cheri_compartment("runner") void entry()
{
	Debug::log("Guess a number between 0 and 9 (inclusive)");
	while (int c =
	         MMIO_CAPABILITY(Uart, uart)->blocking_read())
	{
		if ((c < '0') || (c > '9'))
		{
			Debug::log("Invalid guess: {}", c);
			continue;
		}
		c -= '0';
		if (check_guess(c))
		{
			Debug::log("Correct!  You guessed the secret was {}",
			           c);
		}
	}
}

Listing 34. The runner compartment for the guess-the-numbers gameexamples/safebox/runner.cc

When you run this, you will see output something like the following:

Runner: Guess a number between 0 and 9 (inclusive)
Safebox: Guess was 3, secret was 4
Safebox: Guess was 1, secret was 8
Safebox: Guess was 9, secret was 4
Safebox: Guess was 4, secret was 3
Safebox: Guess was 8, secret was 6
Safebox: Guess was 1, secret was 0
Safebox: Guess was 2, secret was 8
Safebox: Guess was 3, secret was 5
Safebox: Guess was 5, secret was 2
Safebox: Guess was 2, secret was 3
Safebox: Guess was 1, secret was 6
Runner: Correct!  You guessed the secret was 3

In this example, the extra compartmentalisation doesn't really buy us anything. You could combine these two compartments and have similar functionality. Perhaps more importantly, separating them adds little additional complexity to the code, yet respects the principle of least privilege. The code that handles I/O does not need to know the secret and the code that knows the secret does not need to be able to read via the UART.

A safebox assumes that its caller is malicious. A compartment may start malicious or become malicious as the result of a compromise. Try modifying only runner.cc in this example to leak the secret without any incorrect guesses. Hopefully, you will find it impossible.

6.6. Refining trust

It seems conceptually easy to say 'this code is trusted' and 'this code is untrusted', but that rarely tells the whole story. At a high level, components are typically trusted (or not) with respect to three properties:

Confidentiality
How does information flow out of this component?
Integrity
What how can information be modified by this component?
Availability
What can this component prevent from working?

Compartments and threads are both units of isolation in a CHERIoT system. Threads are scheduled independently and provide a building block for availability guarantees. Only a higher-priority thread or code running with interrupts disabled can prevent an unrelated thread from making progress.

The relative importance of each of these varies a lot depending on context. For example, you often don't care at all about confidentiality for encrypted data, but you would not want the plain text form to leak and you definitely wouldn't want the encryption key to leak. If you're building a safety-critical system, availability is often key. Dumping twenty tonnes of molten aluminium onto the factory floor will probably kill people and cost millions of dollars, so preventing that is far more important than ensuring that no one unauthorised can inspect the state of your control network.

This kind of model helps understand where you should put compartment boundaries. If an attacker can compromise one component, what damage can they do to these properties in other compartments and in the system as a whole?

For example, consider the simplest embedded application, which just flashes an LED in a pattern. Where should you put compartment boundaries here? You might put the piece that prepares the pattern in one compartment and the part that interacts directly with the LED in another. Doing this does not add security value. Neither compartment is exposed to an attacker and so you're just protecting against bugs. The compartment with direct access to the device is just passing a value from a function argument to the device. It is unlikely that there will be a bug in this code that can affect the rest of the system. Conversely, the code that can call this can do everything that this compartment can do and so you haven't reduced the damage that a bug can cause.

Now imagine a slightly more complex device where, rather than lighting a single LED, you are driving an LED strip that takes a 24-bit colour value for each LED in the strip, encoded as a waveform down a two-wire serial line. If you generate the wrong waveform, you'll get the wrong pattern and so there is an availability property that you can protect by moving the code that pauses and toggles a GPIO pin into a separate driver compartment. This driver routine needs to run with interrupts disabled (context switching in the middle of programming the strip would cause it to reprogram the first part twice). Running with interrupts disabled has availability implications on the rest of the system because nothing else can run while this is happening. If you put the driver in a separate compartment then you are protected in both directions:

This then gives you something to build on if you decide, for example, that you want to be able to update the lighting patterns from the Internet. Now you want to add a network stack to be able to fetch the new patterns and an interpreter to run them. What does the threat model look like?

The network stack is exposed to the Internet and so is the most likely place for an attack to start. If this needs to interact with the network hardware with interrupts disabled then you probably want to put that part in a separate network driver compartment so that an attacker can't cause the network stack to sit with interrupts disabled forever. A lot of common attacks on network stacks will simply fail on a CHERIoT system because they depend on violating memory safety but it's possible that an attacker will find novel techniques and compromise the network stack.

You will want narrow interfaces between the network stack and the TLS stack, so that the worst that an attacker with full control over the network stack compartment can do is provide invalid packets (and an attacker can do that from the Internet anyway). The TLS stack will decode complete messages and forward them to the interpreter compartment. TLS packets have cryptographic integrity protection and so anything that comes through this path is probably safe, unless the TLS compartment is compromised, but putting the interpreter in a separate compartment ensures that invalid interpreter code can provide different colours to the LEDs but can't damage the LEDs and can't launch attacks over the network.

6.7. Validating arguments

Documentation for the check_pointer function
template<PermissionSet Permissions = PermissionSet{Permission::Load},
	         bool          CheckStack  = true,
	         bool          EnforceStrictPermissions = false>
	__always_inline inline bool
	check_pointer(auto  &ptr,
	              size_t space = sizeof(std::remove_pointer<decltype(ptr)>))
	    requires(std::is_pointer_v<std::remove_cvref_t<decltype(ptr)>> ||
	             IsSmartPointerLike<std::remove_cvref_t<decltype(ptr)>>)

Checks that ptr is valid, unsealed, has at least Permissions, and has at least Space bytes after the current offset.

ptr can be a pointer, or a smart pointer, i.e., any class that supports a get method returning a pointer, and operator=. This includes Capability and standard library smart pointers.

If the permissions do not include Global, then this will also check that the capability does not point to the current thread's stack. This behaviour can be disabled (for example, for use in a shared library) by passing false for CheckStack.

If EnforceStrictPermissions is set to true, this will also set the permissions of passed capability reference to Permissions, and its bounds to space. This is useful for detecting cases where compartments ask for less permissions than they actually require.

This function is provided as a wrapper for the ::check_pointer C API. It is always inlined. For each call site, it materialises the constants needed before performing an indirect call to ::check_pointer.

If a function that is exported from a compartment takes primitive values as arguments, there's little that an attacker can do other than provide invalid values. For things like integers, this doesn't matter, for enumerations it's important to ensure that they are valid values.

Pointers are more complicated. There are a few things that an attacker can do with pointer arguments to invoke a crash:

Any of these (or similar attacks) will allow an attacker to cause your compartment to encounter a fault when it tries to use the pointer.

In general, you will want to check permissions and bounds on any pointer argument that you're passed. The CHERI::check_pointer" function helps here. It checks that a pointer has (at least) the bounds and permissions that you expect and that it isn't in your current stack region. If you don't specify a size, the default is the size of the argument type. You can use this to quickly check any pointer that's passed to you.

Checking the pointer is not the only option. A CHERI fault will invoke the compartment's error handler (see Section 6.9. Handling errors) and so it may be possible to recover. Some compartments chose to assume that their arguments are valid and just gracefully clean up if they aren't.

If a pointer refers to a heap location, there is one additional attack possible. In general, a pointer cannot be modified after it's been checked, but the memory that a pointer refers to may be freed. When this happens, the pointer is implicitly invalidated. In some cases, you may simply wish to disallow pointers that point to the heap.

You can check whether a pointer refers to heap memory by calling heap_address_is_valid. If this returns true, you can prevent deallocation by using the claim mechanism, described in Section 8.8. Ensuring that heap objects are not deallocated.

Documentation for the heap_address_is_valid function
_Bool heap_address_is_valid(const void * object)

Returns true if object points to a valid heap address, false otherwise. Note that this does *not* check that this is a valid pointer. This should be used in conjunction with check_pointer to check validity. The principle use of this function is checking whether an object needs to be claimed. If this returns false but the pointer has global permission, it must be a global and so does not need to be claimed. If the pointer lacks global permission then it cannot be claimed, but if this function returns false then it is guaranteed not to go away for the duration of the call.

Alternatively, you can use the ephemeral claim mechanism (also documented in Section 8.8. Ensuring that heap objects are not deallocated to ensure that a pointer is either a pointer that cannot be freed, or to ensure that it remains live until the next cross-compartment call. These techniques are all combined in Listing 35, which is a simple function that prints a string to the UART, defensively. This first uses a lock (see Chapter 7. Communicating between threads) to ensure that only one thread will access this function at a time. If this compartment exposed more than one function that used the UART then the lock would need to be shared between all of them.

/// Write a message to the UART.
int uart_puts(const char *msg)
{
	static FlagLockPriorityInherited lock;
	// Prevent concurrent invocation
	LockGuard g(lock);
	Timeout   t{UnlimitedTimeout};
	// Make sure that this is not going to be deallocated out
	// from under us.
	if (heap_claim_ephemeral(&t, msg) != 0)
	{
		return -EINVAL;
	}
	// Check that this is a valid pointer with the correct
	// permissions.
	if (!check_pointer<PermissionSet{Permission::Load}>(msg))
	{
		return -EINVAL;
	}
	// Get the bounds (distance from address to top) of the
	// pointer.
	Capability buffer{msg};
	size_t     length = buffer.bounds();
	// Write the data, one byte at a time.
	for (size_t i = 0; i < length; i++)
	{
		char c = msg[i];
		if (c == '\0')
		{
			break;
		}
		MMIO_CAPABILITY(Uart, uart)->blocking_write(c);
	}
	MMIO_CAPABILITY(Uart, uart)->blocking_write('\n');
	return 0;
}

Listing 35. Checks to ensure that a function does not crash.examples/check_arguments/uart.cc

Next, it uses heap_claim_fast to prevent concurrent deallocation. After this, it is safe to use check_pointer to ensure that the permissions are correct and that this pointer does not overlap the current compartment's stack. For a C string, this checks only that a single byte is readable. The function then gets the length explicitly and prints either the full length of the buffer, or the buffer up to a null terminator, whichever is shorter.

You can see how well this works with the attacks shown in Listing 36. This tries passing a string that is not null-terminated, a string without load permission, an untagged capability, and finally a valid capability.

	char unterminatedString[] = {
	  'N', 'o', ' ', 'n', 'u', 'l', 'l'};
	uart_puts(unterminatedString);
	Capability invalidPermissions = "Invalid permissions";
	invalidPermissions.permissions() &= Permission::Store;
	uart_puts(invalidPermissions);
	char *invalidPointer = reinterpret_cast<char *>(12345);
	uart_puts(invalidPointer);
	uart_puts("Non-malicious string");

Listing 36. Attempting to attack the safe UART.examples/check_arguments/hello.cc

When you run this example, you should see output like this:

No null
Non-malicious string

The string that misses the null terminator is written, but then there's no overflow. The string that has the wrong permissions and the string that is not a valid capability at all are simply not printed. Finally, the non-malicious string is printed correctly, showing that the attacker has not been able to corrupt internal state.

6.8. Ensuring adequate stack space

The stack is shared between compartments invoked on the same thread. The callee has access to the portion of the stack that its callers have not used. This means that a malicious compartment can consume almost all of the stack and then try to force a callee to trap when it tries to use the stack.

Before entering a compartment, the switcher will check the amount of stack space against the required amount in the export table. By default, the compiler will fill this value with the amount that is required by the function that serves as an entry point. This is sufficient for leaf functions, but if your function calls others (and they are not inlined) then this will be insufficient.

You can specify the stack space required by a function by using the __cheriot_minimum_stack attribute. This is a function attribute that takes a single argument, the number of bytes of stack space that the function requires. Using this attribute requires you to know how much stack space the function will use.

CHERIoT CPUs include a feature called a stack high-water mark that tracks the amount of stack that is used so that the switcher can avoid zeroing unused portions of the stack. The switcher provides a function, stack_lowest_used_address, that you can call to find the lowest address. You can then use the difference between the top of the stack capability (accessed via the __builtin_cheri_stack_get built-in function) to determine how much stack space has been used in a particular invocation of a compartment entry point.

Documentation for the stack_lowest_used_address function
ptraddr_t stack_lowest_used_address()

Returns the lowest address that has been stored to on the stack in this compartment invocation.

This helper checks the amount of stack usage of the current compartment. The switcher check is not intended to ensure that the invocation of the current compartment can succeed, only that failures are detectable and recoverable. If you want to ensure that a called compartment *also* has enough stack then you will need to add its stack requirements to those of your compartment.

The debug.hh header includes a C++ helper class, StackUsageCheck. This takes a template argument allowing it to be disabled, enabled and just log if you use more than the expected amount of stack, or enabled and trap if you use more than the expected amount of stack. This is most commonly used with a macro like this:

#define STACK_CHECK(expected)  
       StackUsageCheck<StackMode, expected, __PRETTY_FUNCTION__> stackCheck

The StackMode template argument is one of StackCheckMode::Asserting, StackCheckMode::Logging, or StackCheckMode::Disabled. Typically, you will use it in logging mode initially, then disabled mode in production. Use it in asserting mode when running representative tests in CI so that it fails if you have increased your stack requirements and not updated the caller.

It's important that the tests that you run in asserting mode have good coverage. It's typically fine for this to be function-granularity coverage: with the exception of variable-length arrays, functions stack usage does not depend on control flow within the function.

It's tempting to enable the stack checks in debug builds. This is usually a bad idea because debug builds include extra checks that increase stack usage. Enabling the stack checks in debug builds will cause you to demand more stack space than a release build actually needs, increasing overall memory pressure.

6.9. Handling errors

Asynchronous interrupts are all routed to the scheduler to wake up the relevant threads and schedule the correct thread. Synchronous faults are (optionally) delivered to the compartment that caused them. These include CHERI exceptions, invalid instruction traps, and so on: anything that can be directly attributed to the current instruction.

Compartments have two opportunities to handle these, by implementing at least one of two kinds of error handlers.

Rich
Rich error handlers that have access to the full state at the point the error occurred. These can be written in C/C++ and can do things like step over faulting instructions and resume, or provide rich diagnostic information about the interrupted context.
Stackless
Lightweight error handlers that must be written in assembly.

When a hardware exception occurs, the switcher will first look for a rich error handler and prepare a call frame for it. If there is not sufficient stack space to invoke a rich error handler, or if one is not provided by the current compartment, the switcher will then look for a stackless error handler.

The stackless variant will be passed the stack capability as it was on entry to the compartment and the exception cause, but no other information. Most users will not write these but will instead use one that the platform provides.

6.10. Writing rich error handlers

You can provide a rich error handler by implementing compartment_error_handler in your compartment.

Documentation for the compartment_error_handler function
enum ErrorRecoveryBehaviour compartment_error_handler(struct ErrorState * frame, size_t mcause, size_t mtval)

The error handler for the current compartment. A compartment may choose to implement this. If not implemented then compartment faults will unwind the trusted stack.

This function is passed a copy of the register file and the exception cause registers when a fault occurs. The mcause value will be one of the standard RISC-V exception causes, or 0x1c for CHERI faults. CHERI faults will encode the CHERI-specific fault code and the faulting register in mtval. You can decompose this into its component parts by calling CHERI::extract_cheri_mtval.

Documentation for the extract_cheri_mtval function
std::pair<CauseCode, RegisterNumber> extract_cheri_mtval(uint32_t mtval)

Decompose the value reported in the mtval CSR on CHERI exception into a pair of CauseCode and RegisterNumber.

Will return CauseCode::Invalid if the code field is not one of the defined causes and RegisterNumber::Invalid if the register number is not a valid register number. Other bits of mtval are ignored.

The error handler is called with interrupts enabled, even if interrupts were disabled in the faulting code. Latency-critical code should never depend on the error handler for meeting its timing.

If a called compartment faults and forcibly unwinds then this will be reported as a CHERI fault with no cause (zero) in mtval. You can use this to propagate faults up to callers, to track the number of times a cross-compartment call has failed, and so on.

The spilled register file does not contain a tagged value for the program counter capability. This is to prevent library functions that run with interrupts disabled or with access to secrets from accidentally leaking on faults. All other registers will be preserved exactly as they are in the register file.

Error handlers are somewhat similar to UNIX signal handlers, but with some important differences. They are invoked for synchronous faults, not arbitrary event notification. Importantly, they are required only to handle the current compartment's errors. You cannot, for example, call malloc in a signal handler because it would deadlock (or corrupt state) if the signal arrives during a call to malloc or free. In contrast, if a call to heap_allocate fails then that error will be handled in the allocator compartment. Your error handler will never be invoked in the middle of a call to the allocator and so it is fine to use error handlers to release locks and free memory.

At the end of your error handler, you have two choices. You can either ask the switcher to resume, installing your modified register file (rederiving the PCC from the compartment's code capability), or you can ask it to continue unwinding.

Error handling functions are used for resource cleanup. For example, you may wish to drop locks when an error occurs, or you may wish to reset the compartment entirely. The heap_free_all function, discussed in Chapter 8. Memory management in CHERIoT RTOS helps with the latter.

6.11. Using scoped error handling

You are unlikely to ever write a stackless error handler. Most of the time, you will want to link the one provided by the unwind_error_handler target. This can be adopted by adding add_deps("unwind_error_handler") to your compartment's target in xmake.

This implementation is intended to be used with a set of macros that provide exception-like error handling. These maintain a stack of jmp_buf structures, as defined by setjmp. The head of the linked list is stored at the top of the region of the stack that is visible to the current compartment invocation, in a gap left by the switcher. Each CHERIOT_DURING macro invocation pushes an entry onto a stack that allows returning and each CHERIOT_END_HANDLER macro pops the top entry.

Between these two, a CHERIOT_HANDLER is the equivalent of the start of a catch block. This defines the start of a region where code will run if an error is triggered. You can see this in action in Listing 37. This uses __builtin_trap, which the compiler will transform into an invalid instruction, to force a trap. This is a placeholder for anything that might (including in nested function calls) raise an error, such as a bounds violation, a use-after-free bug, a null-pointer dereference, and so on.

Documentation for the CHERIOT_DURING macro
CHERIOT_DURING

Simple error handling macros. These are modelled on the OpenStep exception macros and are similarly built on top of setjmp. Code between CHERIOT_DURING and CHERIOT_HANDLER corresponds to a try block. Code between CHERIOT_HANDLER and CHERIOT_END_HANDLER corresponds to a catch block, though no exception value is actually thrown.

Any automatic-storage values accessed in both blocks must be declared volatile.

	Debug::log("About to try something unsafe.");
	CHERIOT_DURING
	{
		Debug::log("In during block");
		if (shouldTrap)
		{
			// This will unconditionally trap.
			__builtin_trap();
		}
	}
	CHERIOT_HANDLER
	{
		Debug::log("Something bad happened!");
	}
	CHERIOT_END_HANDLER
	Debug::log("Finished unsafe block.");

Listing 37. Example of using scoped error handlingexamples/error_handling/errors.cc

If you run this example with shouldTrap set to false then it will generate the following output:

Error handling example: About to try something unsafe.
Error handling example: In during block
Error handling example: Finished unsafe block.

The code in the CHERIOT_DURING block runs, the code in the CHERIOT_HANDLER block is omitted, and control-flow resumes after CHERIOT_END_HANDLER. In contrast, if shouldTrap is true then the trap will transition into the switcher, which will then invoke the compartment's stackless error handler, which will then transfer control into the CHERIOT_HANDLER block and you'll see output like this:

Error handling example: About to try something unsafe.
Error handling example: In during block
Error handling example: Something bad happened!
Error handling example: Finished unsafe block.

This lets you do things like release locks or clean up per-call state in case of failure.

6.12. Conventions for cross-compartment calls

If a compartment faults and force unwinds to the caller then the return registers will be set to -1. This makes it easy to use the UNIX convention of returning negative numbers to indicate error codes. The value -1 is -ECOMPARTMENTFAIL and other numbers from errno.h can be used to indicate other failures.

A CHERIoT capability is effectively a tagged union of a pointer and 64 bits of data. You can take advantage of this in functions that return pointers to return either an integer or, if the result is not tagged, an error code.

To see a slightly less-contrived version of error handling, Listing 38 is a version of Listing 35, rewritten to use error handling instead of checking. The original version checked all of the properties of the string and protected itself against concurrent mutation. This version still uses check_pointer, because this also prevents information disclosure by ensuring that the string does not overlap the current compartment invocation's stack. You could omit this check if you are not worried about information disclosure. For capabilities that you write through, this check is very important because otherwise you may write over parts of your own stack and corrupt internal state by accident.

The example then simply tries to read the bytes, assuming that they are a valid null-terminated C string.

/// Write a message to the UART.
int uart_puts(const char *msg)
{
	// Prevent information disclosure, check that this does
	// not overlap with our stack region.  Check for obvious
	// errors at the same time.
	if (!check_pointer(msg))
	{
		return -EINVAL;
	}
	static FlagLockPriorityInherited lock;
	// Prevent concurrent invocation
	LockGuard g(lock);
	int       result = 0;
	// Assume this is a null-terminated string, report an
	// error on exceptions if not.
	on_error(
	  [&]() {
		  for (const char *m = msg; *m != '\0'; m++)
		  {
			  MMIO_CAPABILITY(Uart, uart)->blocking_write(*m);
		  }
	  },
	  [&]() { result = -EINVAL; });
	MMIO_CAPABILITY(Uart, uart)->blocking_write('\n');
	return result;
}

Listing 38. Using structured error handling to ensure that a function does not crash.examples/yolo_arguments/uart.cc

This version uses the C++ wrappers that take lambdas, rather than the macro versions, but the semantics are the same as described earlier. When you run this, you should see exactly the same output as the original version:

No null
Non-malicious string

The string that lacks a null terminator will now simply be written looking for one. The attempt to read one byte past the end will trigger a bounds exception. This will then invoke the second lambda passed to on_error, which will set the error code and resume. The other errors are caught by the check_pointer call. If you remove that call, you will instead see the following output:

No null


Non-malicious string

Now, each call is printing the trailing newline, even if it encountered a fault while reading the message.

6.13. Building software capabilities with sealing

The CHERI capability mechanism can be used to express arbitrary software-defined capabilities. Recall that a capability, in the abstract, is an unforgeable token of authority that can be presented to allow some action. In UNIX systems, for example, file descriptors are capabilities. A userspace process cannot directly talk to the disk or the network, but if it presents a valid file descriptor to system calls such as read and write then the kernel will perform those operations on its behalf.

CHERIoT provides a mechanism to create arbitrary software-defined capabilities using the sealing mechanism (see Section 2.4. Sealing pointers for tamper proofing). CHERIoT provides almost a few billion sealing types for use with software-defined capabilities. You can allocate one of these dynamically by calling token_key_new.

There is no mechanism to reuse sealing capabilities. As such, once you have allocated 4,278,190,079, you will be unable to create new ones. A 20 MHz core doing nothing other than allocating new sealing capabilities could exhaust this space in around a day. If untrusted code is allowed to allocate dynamic sealing capabilities then you may wish to restrict its access to this API and instead give it access to a wrapper that limits the number that it may allocate.

Documentation for the token_key_new function
SKey token_key_new()

Create a new sealing key.

This function is guaranteed to complete unless the allocator has exhausted the total number of sealing keys possible (2^32 - 2^24). After this point, it will never succeed. A compartment that is granted access to this entry point is trusted not to exhaust this resource. If you wish to allow a compartment to seal objects, but do not wish to allow it to allocate new sealing keys, then you should insert a proxy compartment that guarantees that it will call this API once and return a single key to the caller.

The return value from this is a capability with the permit-seal and permit-unseal permissions. Callers may remove one or both of these permissions and delegate the resulting capability to allow other compartments to either seal or unseal the capabilities with this key.

If the sealing keys have been exhausted then this will return INVALID_SKEY. This API is guaranteed never to block.

You can also statically register a sealing type with the STATIC_SEALING_TYPE() macro. This takes a single argument, the name that you wish to give the type. This name is used both to refer to the static sealing capability is the name that will show up in auditing reports.

Documentation for the STATIC_SEALING_TYPE macro
STATIC_SEALING_TYPE(name)

Macro that evaluates to a static sealing type that is local to this compartment.

You can access the sealing capability within the compartment that exported it using the STATIC_SEALING_VALUE() macro. You can also refer to it in other compartments, but only when constructing static sealed objects. A static sealed object is like a global defined in a compartment, but that compartment can access it only via a sealed capability.

Static sealed objects are declared with DECLARE_STATIC_SEALED_VALUE and defined with DEFINE_STATIC_SEALED_VALUE. These macros take both the name of the sealing type and the compartment that exposes it as arguments. This ensures that there is no ambiguity and that accidental name collisions don't lead to security vulnerabilities.

Documentation for the DECLARE_STATIC_SEALED_VALUE macro
DECLARE_STATIC_SEALED_VALUE(type,compartment,keyName,name)

Forward-declare a static sealed object. This declares an object of type type that can be referenced with the STATIC_SEALED_VALUE macro using name. The pointer returned by the latter macro will be sealed with the sealing key exported from compartment as keyName with the STATIC_SEALING_TYPE macro.

The object created with this macro can be accessed only by code that has access to the sealing key.

Any object created in this way shows up in the audit log. The exports section for the compartment that exposes the sealing key will will contain an entry like this:

{
  "export_symbol": "__export.sealing_type.alloc.MallocKey",
  "exported": true,
  "kind": "SealingKey"
},

This is cross-referenced with a section like this:

{ 
  "contents": "00100000 00000000 00000000 00000000 00000000 00000000",
  "kind": "SealedObject",
  "sealing_type": {
    "compartment": "alloc",
    "key": "MallocKey",
    "provided_by": "build/cheriot/cheriot/release/cheriot.allocator.compartment",
    "symbol": "__export.sealing_type.alloc.MallocKey"
  }
},

This contains the full contents of the sealed object. You can audit these in a firmware image to ensure that they are valid.

Auditing a hex string is not easy. A future version of CHERIoT RTOS will include tools to map these back to useful types. See Chapter 11. Auditing firmware images for more information about the CHERIoT Audit tool, which is designed for expressing policies over these properties.

This gives a building block that can be used to define arbitrary software-defined capabilities at system start. A compartment that performs some action exposes a sealing type and a structure layout that it expects. Static instances of this structure can be baked into the firmware image and then passed as sealed capabilities into the compartment that wishes to use them as capabilities. They can be unsealed using the token APIs described in Section 8.7. Allocating on behalf of a caller.

The token APIs look as if they're provided by the allocator, but token_obj_unseal is a fast path implemented as a library. This makes it fast to unseal objects (no cross-compartment call). It also removes any dependency on the allocator from things that rely on static sealing.

The allocator uses the static sealing mechanism to define allocation capabilities. These contain a quota that is decreased on allocation and increased on deallocation. A compartment can allocate memory only if it has an allocation capability and any allocation capability that it holds shows up in the audit report when linking a firmware image.

6.14. Sharing globals between compartments

CHERIoT supports a notion of pre-shared objects. Each pre-shared object is allocated in a dedicated region of memory and can be imported into one or more compartments. Each import can have a different set of permissions.

This model lets you define a global that is, for example, writeable by one compartment but readable from many, with no control flow between the communicating compartments.

Currently, the syntax for importing a pre-shared object is more verbose, a future version of the CHERIoT compiler will incorporate this into the type system and control imports via attributes on extern declarations.

You can import a pre-shared object with the SHARED_OBJECT(type, name) macro. This takes the type of the object and its name (which must be globally unique across the firmware image) as arguments. This evaluates to a pointer to the object. Objects imported with this macro have the full set of permissions for imported objects.

You can also disable individual permissions using the SHARED_OBJECT_WITH_PERMISSIONS macros. This takes an additional four additional boolean arguments that define the following set of permissions:

Note that load-mutable depends on both load and load/store capability permissions. You cannot load a capability that has store permission if you cannot load a capability.

Shared objects are defined in the build system, by setting the shared_objects value on a target (typically a compartment). For example:

    on_load(function(target)
        target:values_set("shared_objects", { exampleK = 1024, test_word = 4 }, {expand = false})
    end)

This is from the test suite and defines a test_word object that is a single 32-bit value and an exampleK object that is 1024 bytes. Note that the objects are defined as sizes, not as types. The type cannot be enforced by CHERI and depends on the compartment that imports the object. If a single compartment has write access to an object then that compartment forms the TCB for type safety of that object.