5. Compartments and libraries

In most conventional operating systems, you share code with shared libraries and you isolate running code with processes. Compartments in CHERIoT are somewhere between these two abstractions. Unlike a process, they do not own threads, which are an independent concept in CHERIoT, described more in Chapter 6. Communicating between threads.

Communication between compartments looks a lot more like communication between shared libraries than like inter-process communication (IPC). They can export functions to be called from other compartments and can call functions exported from other compartments. Like a shared library, they have code and globals associated with them, but a cross-compartment call crosses a security boundary, in the same way that an IPC message would.

CHERIoT shared libraries are a lightweight way of reusing code without duplicating it into different compartments. They provide a very similar programmer model to simply copying and pasting a function into every compartment that uses it, without the space overhead. Calling a library function does not involve crossing a security boundary. Libraries contain code and read-only data but do not have mutable globals. It is possible for libraries to hold secrets but, unless library functions are written in very careful assembly, they should assume that any (immutable) globals in the library can leak to callers. Each library entry point is exposed as a sentry capability (see Section 1.6. Sealing pointers for tamper proofing) to the callers, which means that the caller cannot directly read its code or (immutable) data.

5.1. Compartments and libraries export functions

In a UNIX-like system, a shared library can export any kind of symbol. This includes functions and global variables. In CHERIoT, compartments and libraries can export only functions as entry points. Global variables are always private to a compartment or library, unless a pointer is explicitly passed out as a function argument or return in a cross-compartment call. This design is intended to make it easier to reason about sharing between compartments.

If you declare a global in a header and define it in a library or a compartment, you may see linker errors if you try to use it in other compartments or libraries. This holds even for const globals exported from libraries. You can place a static const global in a header for a library, but that will introduce tight coupling: the value in the header may be inlined at any use site. For very large globals, this may also increase code size significantly.

You can still use a compartment's globals to share data but you must explicitly expose them via an accessor function. This makes CHERIoT compartments and libraries similar to Smalltalk-style objects, with public methods and private instance variables. You can also create globals that are shared between compartments (see Section 5.7. Sharing globals between compartments) but that are not part of any compartment.

If you expose an interface that returns a pointer to a global, you can use CHERI permissions to restrict access. Returning a read-only pointer to a global is a common idiom for building a lightweight broadcast communication channel. The owning compartment can write to the global and other compartments can read from their copy of the pointer, with guarantees that only the owning compartment is making changes.

To see the differences, Listing 30 shows a header that exports two functions, one from a compartment and one from a library.

/**
 * A simple example library function.
 */
__cheriot_libcall void library_function();

/**
 * A simple example compartment function.
 */
__cheriot_compartment(
  "compartment") int compartment_function();

The exported functions both contain the implementation shown in Listing 31, which uses the debug APIs (see Chapter 8. Features for debug builds) to print the capabilities in three registers. This listing shows the version in the library but the code in the compartment is identical. There are two compartments in this example, the entry compartment and the compartment that it calls.

The exported (library and compartment) functions will print the stack, code, and globals regions, respectively. The compiler provides builtin functions to copy two of these (the program counter and stack capabilities) but the third, the globals pointer, requires some inline assembly. The inline assembly needs to run at the start of the function because this function does not reference any globals, so the compiler will otherwise spill this register to use it as a temporary.

When you run this code, you should see output something like this:

Entry: Stack pointer: 0x80000d30 (v:1 0x80000750-0x80000d50 l:0x600 o:0x0 p: - RWcgml -- ---)
Entry: Program counter: 0x80005f18 (v:1 0x80005ee0-0x80005fe8 l:0x108 o:0x0 p: G R-cgm- X- ---)
Entry: Globals pointer: 0x80006202 (v:1 0x80006200-0x80006204 l:0x4 o:0x0 p: G RWcgm- -- ---)
Library: Stack pointer: 0x80000d20 (v:1 0x80000750-0x80000d50 l:0x600 o:0x0 p: - RWcgml -- ---)
Library: Program counter: 0x80005d48 (v:1 0x80005d20-0x80005df8 l:0xd8 o:0x0 p: G R-cgm- X- ---)
Library: Globals pointer: 0x80006202 (v:1 0x80006200-0x80006204 l:0x4 o:0x0 p: G RWcgm- -- ---)
Compartment: Stack pointer: 0x80000cf0 (v:1 0x80000750-0x80000d10 l:0x5c0 o:0x0 p: - RWcgml -- ---)
Compartment: Program counter: 0x80005e20 (v:1 0x80005df8-0x80005ee0 l:0xe8 o:0x0 p: G R-cgm- X- ---)
Compartment: Globals pointer: 0x800061fe (v:1 0x800061fc-0x80006200 l:0x4 o:0x0 p: G RWcgm- -- ---)

	register void *cgp asm("cgp");
	asm("" : "=C"(cgp));
	// Print the stack capability from within the library.
	Debug::log("Stack pointer: {}",
	           __builtin_cheri_stack_get());
	Debug::log("Program counter: {}",
	           __builtin_cheri_program_counter_get());
	Debug::log("Globals pointer: {}", cgp);

Ignore the exact memory addresses, these may change depending on where you run the example. First, note that the program counter capability (the capability used for instruction fetch) is different in all three cases and the bounds do not overlap. In this particular build, the called compartment is placed in code memory immediately after the entry compartment, so the address 0x80005df8 is the boundary between the two.

Next, observe that the globals pointer is different between the entry compartment and the one that is called in the last three lines of the output. This example includes a single int global, to make sure that these are non-zero (two compartments may have the same zero-length capabilities for globals if they have no globals). In contrast, the library prints the same globals pointer as the caller. As previously mentioned, a malicious library can access any globals in the caller.

Finally, look at the stack pointers. The first thing to note is that the start address is 0x80000750 in all cases. This is because the stack belongs to the thread and not the compartment. Stacks grow downwards and the end of the stack is the same for each. The address of the stack pointer is different in each because they run at different depths on the stack. The initial value is very close to the top of the stack, then the library and compartment calls are deeper. A compartment call needs to save more state (two callee-save registers and the old global pointer, which are preserved across normal calls) and so entry to the compartment call is slightly (48 bytes) lower. The top of the stack is the most interesting place. In the compartment call, the top of the stack is at address 0x80000d10, 64 bytes below its location in the entry compartment and the library. Anything above that point is unreachable in the compartment before then. The length shows this truncation another way. The original had 0x600 bytes of stack but this is reduced to 0x5c0 after the cross-compartment call.

This may all be easier to understand visually. Figure 4. An illustration of memory pointed to by each register in the compartment-call example shows the memory ranges that each compartment points to, in each of the three places. The lines marked with a ⓵ indicate the initial values on entry, those marked ⓶ indicate the values in the library call and those marked ⓷ show the values in the cross-compartment call.

Figure 4. An illustration of memory pointed to by each register in the compartment-call example.

5.2. Understanding the structure of a compartment

From a distance, a compartment is a very simple construct. The core of a compartment is made of just two capabilities. The program counter capability (PCC) defines (and grants access to) the range of memory covering the compartment's code and read-only globals. This has read and execute permissions. The capability global pointer (CGP) defines (and grants access to) the range of memory covering the compartment's mutable globals. The full structure is more complex and is shown in Figure 5. The structure of a compartment.

Figure 5. The structure of a compartment.

If a compartment didn't need to interact with anything else, these two regions would be sufficient. In practice, compartments are useful only because they interact with other compartments or the outside world. The read-only data region contains an import table. This is the only region of memory that, at system start, is allowed to contain capabilities that grant access outside of the PCC and CGP region for the compartment. The instructions for the loader to populate these are in the firmware image and are amenable to auditing.

The import table contains three kinds of capabilities. MMIO capabilities are conceptually simple: they are just pointers that grant access to specific devices. This mechanism allows byte-granularity access to device registers and so it's possible to provide a compartment with access to a single device register from a large device.

Import tables also contain sentry capabilities for library functions. A shared library has its own PCC region (like a compartment) but does not have a CGP region. Library routines are invoked by loading the sentry from the import table and jumping to it.

Finally, import tables contain sealed capabilities referring to other compartments' export tables. If a compartment exports any entry points for other compartments to call, it has an export table. This contains the PCC and CGP for the compartment and a small amount of metadata for each exported function describing:

• The location of the entry point.
• Whether interrupts are enabled or disabled when invoking this function.
• How many argument registers are used (conversely, how many are unused and should be zeroed).

This is all of the information that the switcher needs to transition from one compartment to another. Libraries have similar export-table entries, though they are used by the loader rather than the switcher. The final element, the number of registers to use, is not used for libraries because library calls are not a transition between security domains.

Extracting code and moving it to a new compartment adds a very small amount of memory overhead, on the order of a dozen words for a typical compartment.

5.3. Adding compartments to the build system

The build system makes adding compartments trivial. An xmake build file (xmake.lua) uses a declarative Lua-like syntax at the top level. This defines targets as sections. Listing 32 shows the build system logic for the example compartments and libraries from earlier. Defining a new target implicitly ends the definition of the previous one. This example implicitly ends the library and entry targets, but uses target_end() to explicitly end the compartment target.

-- An example compartment that we can call
compartment("compartment")
	add_files("compartment.cc")
target_end()

-- An example compartment that we can call
library("library")
	set_default(false)
	add_files("library.cc")

-- Our entry-point compartment
compartment("entry")
	add_files("entry.cc")
	add_deps("compartment", "library")

Inside each target definition, you can add files, dependencies on other targets, and so on. The "library" target, for example, sets itself as a non-default target. xmake will build every default target, whether it is used or not. Marking a target as non-default allows it to be defined, but built only if it is used. This is useful for reusable components. The RTOS provides build-system logic for a set of libraries, but each is built only if it is added as a dependency of something that is built.

Once you have defined the rules to build each compartment and library, they need to be combined into a firmware image. Any library or compartment that is a direct or indirect dependency of a firmware image will be built and linked into the final image. For this example, we show both forms. The entry compartment lists both the library and the example compartment as explicit dependencies because it calls both of them. The firmware definition (Listing 33) adds the entry compartment as an explicit dependency, which then pulls in the other two. This firmware also depends on two libraries from the core RTOS, the freestanding library, which provides the core of the C run-time environment, and the debug library which pretty-prints the debug messages.

-- Firmware image for the example.
firmware("library_or_compartment")
	-- RTOS-provided libraries
	add_deps("freestanding", "debug")
	-- Our compartments
	add_deps("entry")

5.4. Choosing a trust model

There are three trust models that are commonly applied to compartments:

Sandbox

A sandbox is a compartment that is used to isolate untrusted code. This model is used to protect the rest of the system. Typically, a sandbox will trust values passed to it as arguments to exported functions or return values from functions that it calls in other compartments.

Safebox

A safebox is a compartment that holds some secret or sensitive data that must be protected from the outside. For example, a safebox may be used to protect a key and expose functions that use it to encrypt or sign data on behalf of callers. A safebox does not trust any data provided from outside of the compartment, but callers may trust it to behave correctly.

Mutual distrust

Mutual distrust is the strongest model. A compartment in a mutual-distrust relationship protects itself from attacks from the outside by careful handling of inputs and expects other compartments to protect themselves from it in the same way.

This is the start of defining a threat model for your code. A compartment may simply be used for fault isolation, to limit the damage that a bug can do. You may assume that an attacker will be able to compromise some compartments (for example, those directly processing network packets) and defend yourself accordingly.

In the core of the RTOS, the scheduler is written as a safebox. It does not trust anything on the outside and assumes that everything else is trying to make it crash. The memory allocator is also written as a safebox, assuming that everything else is trying to either make it crash or leak powerful capabilities. For some operations, the scheduler invokes the allocator. The scheduler trusts the allocator to enforce heap memory safety. It does not, for example, try to check that the memory allocator is returning disjoint capabilities (it can't see every other caller of heap_allocate, so it couldn't validate this). It is, however, written to assume that other compartments may try to maliciously call allocator APIs to cause it to crash, for example by freeing memory that the scheduler is using. When thinking about trust, it's worth trying to articulate the properties that other code is trusted to enforce or preserve. For example, everything in the CHERIoT system trusts the scheduler for availability. Most things trust the allocator to enforce spatial and temporal memory safety for the heap.

5.5. Implementing a safebox

The safebox abstraction is trivial to implement in CHERIoT. Listing 34 shows the build system for a safebox. Note that there's nothing in this that indicates the trust model. From the perspective of the build, all compartments are just compartments. Trust relationships between them are a property of how you write the code in the compartments.

-- The safebox compartment
compartment("safebox")
	add_files("safebox.cc")

compartment("runner")
	add_files("runner.cc")

-- Firmware image for the example.
firmware("safebox_example")
	-- RTOS-provided libraries
	add_deps("freestanding", "cxxrt", "debug")
	-- Our compartments
	add_deps("runner", "safebox")
	on_load(function(target)
		-- The board to target
		target:values_set("board", "$(board)")
		target:values_set("threads", {
			{
				compartment = "runner",
				priority = 1,
				entry_point = "entry",
				stack_size = 0x400,
				trusted_stack_frames = 2
			},
		}, {expand = false})
	end)

Listing 35 shows a complete implementation of a simple safebox for a guess-the-number game. This generates a (very weak!) pseudorandom number using the cycle counter and allows callers to guess it.

using Debug = ConditionalDebug<true, "Safebox">;

bool check_guess(int guess)
{
	static int secret = rdcycle64() % 10;
	if (guess != secret)
	{
		Debug::log(
		  "Guess was {}, secret was {}", guess, secret);
		secret = rdcycle64() % 10;
		return false;
	}
	return true;
}

This is called from the runner compartment, shown in Listing 36. This compartment talks directly to the outside world and so is part of the attack surface. It's quite unlikely that a compartment that simply reads individual bytes and ignores anything not in the ASCII digit range could be vulnerable, but the same techniques can protect much more realistic examples.

using Debug = ConditionalDebug<true, "Runner">;

__cheriot_compartment("runner") void entry()
{
	Debug::log("Guess a number between 0 and 9 (inclusive)");
	while (int c =
	         MMIO_CAPABILITY(Uart, uart)->blocking_read())
	{
		if ((c < '0') || (c > '9'))
		{
			Debug::log("Invalid guess: {}", c);
			continue;
		}
		c -= '0';
		if (check_guess(c))
		{
			Debug::log("Correct!  You guessed the secret was {}",
			           c);
		}
	}
}

When you run this, you will see output something like the following:

Runner: Guess a number between 0 and 9 (inclusive)
Safebox: Guess was 3, secret was 4
Safebox: Guess was 1, secret was 8
Safebox: Guess was 9, secret was 4
Safebox: Guess was 4, secret was 3
Safebox: Guess was 8, secret was 6
Safebox: Guess was 1, secret was 0
Safebox: Guess was 2, secret was 8
Safebox: Guess was 3, secret was 5
Safebox: Guess was 5, secret was 2
Safebox: Guess was 2, secret was 3
Safebox: Guess was 1, secret was 6
Runner: Correct!  You guessed the secret was 3

In this example, the extra compartmentalisation doesn't really buy us anything. You could combine these two compartments and have similar functionality. Perhaps more importantly, separating them adds little additional complexity to the code, yet respects the principle of least privilege. The code that handles I/O does not need to know the secret and the code that knows the secret does not need to be able to read via the UART.

A safebox assumes that its caller is malicious. A compartment may start malicious or become malicious as the result of a compromise. Try modifying only runner.cc in this example to leak the secret without any incorrect guesses. Hopefully, you will find it impossible.

5.6. Building software capabilities with sealing

The CHERI capability mechanism can be used to express arbitrary software-defined capabilities. Recall that a capability, in the abstract, is an unforgeable token of authority that can be presented to allow some action. In UNIX systems, for example, file descriptors are capabilities. A userspace process cannot directly talk to the disk or the network, but if it presents a valid file descriptor to system calls such as read and write then the kernel will perform those operations on its behalf.

CHERIoT provides a mechanism to create arbitrary software-defined capabilities using the sealing mechanism (see Section 1.6. Sealing pointers for tamper proofing). CHERIoT provides almost a few billion sealing types for use with software-defined capabilities. You can allocate one of these dynamically by calling token_key_new.

SKey token_key_new()

You can also statically register a sealing type with the STATIC_SEALING_TYPE() macro. This takes a single argument, the name that you wish to give the type. This name is used to refer to the static sealing capability and is the name that will show up in auditing reports.

STATIC_SEALING_TYPE(name)

You can access the sealing capability within the compartment that exported it using the STATIC_SEALING_VALUE() macro. You can also refer to it in other compartments, but only when constructing static sealed objects. A static sealed object is like a global defined in a compartment, but that compartment can access it only via a sealed capability.

Static sealed objects are declared with DECLARE_STATIC_SEALED_VALUE and defined with DEFINE_STATIC_SEALED_VALUE. These macros take as arguments both the name of the sealing type and the compartment that exposes it. This ensures that there is no ambiguity and that accidental name collisions don't lead to security vulnerabilities.

DECLARE_STATIC_SEALED_VALUE(type,compartment,keyName,name)

DEFINE_STATIC_SEALED_VALUE(type,compartment,keyName,name,initialiser,...)

This gives a building block that can be used to define arbitrary software-defined capabilities at system start. A compartment that performs some action exposes a sealing type and a structure layout that it expects. Static instances of this structure can be baked into the firmware image and then passed as sealed capabilities into the compartment that wishes to use them as capabilities. They can be unsealed using the token APIs described in Section 7.7. Allocating on behalf of a caller. Often, you'll use this via a type-specific wrapper, as in Listing 37. The macro (DECLARE_AND_DEFINE_COUNTER) lets users of this API define a new counter with the sealing type that the compartment exposes.

using MonotonicCounterState = std::atomic<int64_t>;

#define DECLARE_AND_DEFINE_COUNTER(name)                   \
	DECLARE_AND_DEFINE_STATIC_SEALED_VALUE(                  \
	  MonotonicCounterState, monotonic, CounterKey, name, 0)

typedef MonotonicCounterState
  *__sealed_capability MonotonicCounter;

/**
 * Increments a monotonic counter and returns the new value.
 *
 * Returns a negative value for errors.
 */
int64_t __cheriot_compartment("monotonic")
  monotonic_counter_increment(
    MonotonicCounter allocatorCapability);

The increment function from this header is defined in Listing 38. This uses token_unseal to unseal the sealed argument. This will return a valid unsealed object, or null if the unseal fails. Note that this uses the static sealed value via the STATIC_SEALING_TYPE macro. The sealing type doesn't need to be declared, simply using it here ensures that it is exported from the compartment. Trying to use a sealing type that isn't exported from a function will result in a linker error.

int64_t
monotonic_counter_increment(MonotonicCounter sealedCounter)
{
	if (auto *counter = token_unseal(
	      STATIC_SEALING_TYPE(CounterKey), sealedCounter))
	{
		return ++(*counter);
	}
	return -EINVAL;
}

This function follows the convention of returning -EINVAL if passed an invalid value. If the unseal succeeds, this function knows that it can trust the contents of the object. You can see this being called in Listing 39.

// Declare and define a counter for this to use.
DECLARE_AND_DEFINE_COUNTER(aCounter)

void __cheriot_compartment("caller") entry()
{
	// Get a pointer to the valid counter.
	auto validCounter = STATIC_SEALED_VALUE(aCounter);
	// Create an unsealed value of the correct type
	MonotonicCounterState invalidCounterState;
	auto invalidCounter = reinterpret_cast<MonotonicCounter>(
	  &invalidCounterState);
	auto invalidSealedCounter =
	  reinterpret_cast<MonotonicCounter>(MALLOC_CAPABILITY);

	// Try the valid capability
	printf("Valid counter increment returned %lld\n",
	       monotonic_counter_increment(validCounter));
	printf("Valid counter increment returned %lld\n",
	       monotonic_counter_increment(validCounter));
	// Try the invalid ones
	printf("Invalid counter increment returned %lld\n",
	       monotonic_counter_increment(invalidCounter));
	printf("Invalid counter increment returned %lld\n",
	       monotonic_counter_increment(invalidSealedCounter));
	// Try manipulating the counter directly
	auto underlyingCounter =
	  reinterpret_cast<MonotonicCounterState *>(validCounter);
	(*underlyingCounter)++;
}

The caller starts by declaring a monotonic counter and then tries calling the increment function twice with a valid (sealed) value. It then tries again with two kinds of invalid values. The first is a value of the correct underlying type, but which is not sealed. The lack of sealing means that the callee can't trust that the rules for this type are followed. In this example, the type of a monotonic counter guarantees that the counter increments monotonically but an unsealed value would allow the caller to set it to arbitrary values. This should fail to unseal because it is not a sealed value.

The second invalid value is a sealed value, but it is not sealed with the correct type. This is the default capability used for memory allocations in this compartment. In this case, sealing should fail because the type is incorrect. This shows how sealing enforces type safety even in the presence of untrusted code.

Finally, this casts away the sealed type qualifier and tries to access the underlying type directly. The compiler will permit this (it's a reinterpret_cast, which tells the compiler that you're doing something that might be unsafe but that it should trust you) but then the hardware will prevent you from using the result.

When you run this, you should see the following output:

Valid counter increment returned 1
Valid counter increment returned 2
Invalid counter increment returned -22
Invalid counter increment returned -22
Error handler: SealViolation(0x3) error at 0x80003dd8 (v:0 0x80003d88-0x80003e48 l:0xc0 o:0x0 p: G R-cgm- X- ---) (return address: 0x80006358 (v:1 0x80006110-0x80006da0 l:0xc90 o:0x5 p: G R-cgm- X- ---)), with capability register CA0(0xa): 0x800079f0 (v:1 0x800079f0-0x80007a00 l:0x10 o:0xc p: G RWcgm- -- ---)

The first two calls worked and incremented the monotonic counter. The last two calls, with invalid objects, were detected in the caller and the errors were handled gracefully. Finally, the attempt to directly access the sealed object trapped and invoked the compartment's error handler (see Section 5.11. Handling errors), which printed a helpful message for debugging the failure.

The monotonic counter is valid as long as the counter starts at zero (overflowing a 64-bit signed counter will take an infeasible amount of time). What would happen if a malicious caller initialised a counter with a non-zero value? The full contents of any object created in this way shows up in the audit log. You can audit these in a firmware image to ensure that they are valid (see Chapter 10. Auditing firmware images). Once you've read Chapter 10. Auditing firmware images, try writing a policy for this example that ensures that all counters start at zero.

Most of the token APIs are implemented as cross-compartment calls into the allocator, but token_obj_unseal (called here via the token_unseal wrapper) is a fast path implemented as a library. This makes it fast to unseal objects (no cross-compartment call). It also removes any dependency on the allocator from things that rely on static sealing.

The RTOS' allocator provides a more detailed example of a real-world use for this feature. When you allocate memory, you pass the allocator a static sealed object that represents your allocation capability, which authorises allocating memory up to a quota. These contain a quota that is decreased on allocation and increased on deallocation. A compartment can allocate memory only if it has an allocation capability and any allocation capability that it holds shows up in the audit report when linking a firmware image.

5.7. Sharing globals between compartments

CHERIoT supports a notion of pre-shared objects. Each pre-shared object is allocated in a dedicated region of memory and can be imported into one or more compartments. Each import can have a different set of permissions.

This model lets you define a global that is, for example, writeable by one compartment but readable from many, with no control flow between the communicating compartments.

Currently, the syntax for importing a pre-shared object is more verbose. A future version of the CHERIoT compiler will incorporate this into the type system and control imports via attributes on extern declarations.

You can import a pre-shared object with the SHARED_OBJECT(type, name) macro. This takes the type of the object and its name (which must be globally unique across the firmware image) as arguments. This evaluates to a pointer to the object. Objects imported with this macro have the full set of permissions for imported objects.

You can also disable individual permissions using the SHARED_OBJECT_WITH_PERMISSIONS macros. This takes an additional four boolean arguments that define the following set of permissions:

• Load
• Store
• Load or store capabilities
• Load mutable

Note that load-mutable depends on both load and load/store capability permissions. You cannot load a capability that has store permission if you cannot load a capability.

Shared objects are defined in the build system, by setting the shared_objects value on a target (typically a compartment). For example:

    on_load(function(target)
        target:values_set("shared_objects", { exampleK = 1024, test_word = 4 }, {expand = false})
    end)

You don't need to define this on every compartment that imports the object, a single definition is all that is necessary. This example is from the test suite and defines a test_word object that is a single 32-bit value and an exampleK object that is 1024 bytes. Note that the objects are defined as sizes, not as types. The type cannot be enforced by CHERI and depends on the compartment that imports the object. If a single compartment has write access to an object then that compartment forms the TCB for type safety of that object.

5.8. Refining trust

It seems conceptually easy to say 'this code is trusted' and 'this code is untrusted', but that rarely tells the whole story. At a high level, components are typically trusted (or not) with respect to three properties:

Confidentiality

How does information flow out of this component?

Integrity

How can information be modified by this component?

Availability

What can this component prevent from working?

The relative importance of each of these varies a lot depending on context. For example, you often don't care at all about confidentiality for encrypted data, but you would not want the plain text form to leak and you definitely wouldn't want the encryption key to leak. If you're building a safety-critical system, availability is often key. Dumping twenty tonnes of molten aluminium onto the factory floor will probably kill people and cost millions of dollars, so preventing that is far more important than ensuring that no one unauthorised can inspect the state of your control network.

This kind of model helps understand where you should put compartment boundaries. If an attacker can compromise one component, what damage can they do to these properties in other compartments and in the system as a whole?

For example, consider the simplest embedded application, which just flashes an LED in a pattern. Where should you put compartment boundaries here? You might put the piece that prepares the pattern in one compartment and the part that interacts directly with the LED in another. Doing this does not add security value. Neither compartment is exposed to an attacker and so you're just protecting against bugs. The compartment with direct access to the device is just passing a value from a function argument to the device. It is unlikely that there will be a bug in this code that can affect the rest of the system. Conversely, the code that can call this can do everything that this compartment can do, so you haven't reduced the damage that a bug can cause.

Now imagine a slightly more complex device where, rather than lighting a single LED, you are driving an LED strip that takes a 24-bit colour value for each LED in the strip, encoded as a waveform down a two-wire serial line. If you generate the wrong waveform, you'll get the wrong pattern and so there is an availability property that you can protect by moving the code that pauses and toggles a GPIO pin into a separate driver compartment. This driver routine needs to run with interrupts disabled (context switching in the middle of programming the strip would cause it to reprogram the first part twice). Running with interrupts disabled has availability implications on the rest of the system because nothing else can run while this is happening. If you put the driver in a separate compartment then you are protected in both directions:

• The driver is the only thing that can touch the relevant GPIO pin, so if the code in that driver is correct, nothing can cause the strip to be incorrectly programmed.
• The driver runs with interrupts disabled but the rest of the application runs with interrupts enabled, so you can audit the driver code to ensure that it doesn't cause problems for anything else that the microcontroller is doing.

This then gives you something to build on if you decide, for example, that you want to be able to update the lighting patterns from the Internet. Now you want to add a network stack to be able to fetch the new patterns and an interpreter to run them. What does the threat model look like?

The network stack is exposed to the Internet and so is the most likely place for an attack to start. If this needs to interact with the network hardware with interrupts disabled then you probably want to put that part in a separate network driver compartment so that an attacker can't cause the network stack to sit with interrupts disabled forever. A lot of common attacks on network stacks will simply fail on a CHERIoT system because they depend on violating memory safety, but it's possible that an attacker will find novel techniques and compromise the network stack.

You will want narrow interfaces between the network stack and the TLS stack, so that the worst that an attacker with full control over the network stack compartment can do is provide invalid packets (and an attacker can do that from the Internet anyway). The TLS stack will decode complete messages and forward them to the interpreter compartment. TLS packets have cryptographic integrity protection so anything that comes through this path is probably safe (unless the TLS compartment is compromised) but putting the interpreter in a separate compartment ensures that invalid interpreter code can provide different colours to the LEDs but can't damage the LEDs and can't launch attacks over the network.

5.9. Validating arguments

If a function that is exported from a compartment takes primitive values as arguments, there's little that an attacker can do other than provide invalid values. For things like integers, this doesn't matter; for enumerations it's important to ensure that they are valid values.

Pointers are more complicated. There are a few things that an attacker can do with pointer arguments to invoke a crash:

• Provide a pointer without write permission for an output operand.
• Provide a pointer without read permission for an input operand.
• Provide a pointer without global permission that must be captured and held across calls.
• Provide a pointer with a length that is too small.
• Provide something that isn't a valid pointer at all.
• Provide a pointer that overlaps your stack as an output argument.

Any of these (or similar attacks) will allow an attacker to cause your compartment to encounter a fault when it tries to use the pointer.

In general, you will want to check permissions and bounds on any pointer argument that you're passed. The CHERI::check_pointer function helps here. It checks that a pointer has (at least) the bounds and permissions that you expect and that it isn't in your current stack region. If you don't specify a size, the default is the size of the argument type. You can use this to quickly check any pointer that's passed to you.

template<PermissionSet Permissions = PermissionSet{Permission::Load},
	         bool          CheckStack  = true,
	         bool          EnforceStrictPermissions = false>
	__always_inline inline bool
	check_pointer(auto  &ptr,
	              size_t space = sizeof(std::remove_pointer<decltype(ptr)>))
	    requires(std::is_pointer_v<std::remove_cvref_t<decltype(ptr)>> ||
	             IsSmartPointerLike<std::remove_cvref_t<decltype(ptr)>>)

If a pointer refers to a heap location, there is one additional attack possible. In general, a pointer cannot be modified after it's been checked, but the memory that a pointer refers to may be freed. When this happens, the pointer is implicitly invalidated. In some cases, you may simply wish to disallow pointers that point to the heap.

You can check whether a pointer refers to heap memory by calling heap_address_is_valid. If this returns true, you can prevent deallocation by using the claim mechanism, described in Section 7.8. Ensuring that heap objects are not deallocated.

_Bool heap_address_is_valid(const void * object)

Alternatively, you can use the ephemeral claim mechanism (also documented in Section 7.8. Ensuring that heap objects are not deallocated) to ensure that a pointer is either a pointer that cannot be freed, or to ensure that it remains live until the next cross-compartment call. These techniques are all combined in Listing 40, which is a simple function that prints a string to the UART, defensively.

/// Write a message to the UART.
int uart_puts(const char *msg)
{
	static FlagLockPriorityInherited lock;
	// Prevent concurrent invocation
	LockGuard g(lock);
	Timeout   t{UnlimitedTimeout};
	// Make sure that this is not going to be deallocated out
	// from under us.
	if (heap_claim_ephemeral(&t, msg) != 0)
	{
		return -EINVAL;
	}
	// Check that this is a valid pointer with the correct
	// permissions.
	if (!check_pointer<PermissionSet{Permission::Load}>(msg))
	{
		return -EINVAL;
	}
	// Get the bounds (distance from address to top) of the
	// pointer.
	Capability buffer{msg};
	size_t     length = buffer.bounds();
	// Write the data, one byte at a time.
	for (size_t i = 0; i < length; i++)
	{
		char c = msg[i];
		if (c == '\0')
		{
			break;
		}
		MMIO_CAPABILITY(Uart, uart)->blocking_write(c);
	}
	MMIO_CAPABILITY(Uart, uart)->blocking_write('\n');
	return 0;
}

This first uses a lock (see Chapter 6. Communicating between threads) to ensure that only one thread will access this function at a time. If this compartment exposed more than one function that used the UART then the lock would need to be shared between all of them. Next, it uses heap_claim_ephemeral to prevent concurrent deallocation. After this, it is safe to use check_pointer to ensure that the permissions are correct and that this pointer does not overlap the current compartment's stack (because of the default value of CheckStack). For a C string, this checks only that a single byte is readable. The function then gets the length explicitly and prints either the full length of the buffer, or the buffer up to a null terminator, whichever is shorter.

You can see how well this works with the attacks shown in Listing 41. This tries passing a string that is not null-terminated, a string without load permission, an untagged capability, and finally a valid capability.

	char unterminatedString[] = {
	  'N', 'o', ' ', 'n', 'u', 'l', 'l'};
	uart_puts(unterminatedString);
	Capability invalidPermissions = "Invalid permissions";
	invalidPermissions.permissions() &= Permission::Store;
	uart_puts(invalidPermissions);
	char *invalidPointer = reinterpret_cast<char *>(12345);
	uart_puts(invalidPointer);
	uart_puts("Non-malicious string");

When you run this example, you should see output like this:

No null
Non-malicious string

The string that misses the null terminator is written, but then there's no overflow. The string that has the wrong permissions and the string that is not a valid capability at all are simply not printed. Finally, the non-malicious string is printed correctly, showing that the attacker has not been able to corrupt internal state.

5.10. Ensuring adequate stack space

The stack is shared between compartments invoked on the same thread. The callee has access to the portion of the stack that its callers have not used. This is most important when a compartment is called by an untrusted caller. In this case, a malicious caller may try to consume almost all of the stack before calling a victim compartment. The victim would then trap in a place under the attacker's control.

Before entering a compartment, the switcher will check the amount of stack space against the required amount in the export table. By default, the compiler will fill this value with the amount that is required by the function that serves as an entry point. This is sufficient for leaf functions, but if your function calls others (and they are not inlined) then this will be insufficient.

You can specify the stack space required by an exported function by using the __cheriot_minimum_stack attribute. This is a function attribute that takes a single argument, the number of bytes of stack space that the function requires. Setting this attribute ensures that the switcher will not invoke the exported function unless at least the required amount of stack space is available. The malicious caller from the previous example would see a return value if -ENOTENOUGHSTACK and your code would not be invoked. Using this attribute requires you to know how much stack space the function will use.

CHERIoT CPUs include a feature called a stack high-water mark that tracks the amount of stack that is used so that the switcher can avoid zeroing unused portions of the stack. The switcher provides a function, stack_lowest_used_address, that you can call to find the lowest address. You can then use the difference between the top of the stack capability (accessed via the __builtin_cheri_stack_get built-in function) to determine how much stack space has been used in a particular invocation of a compartment entry point.

ptraddr_t stack_lowest_used_address()

The debug.hh header includes a C++ helper class, StackUsageCheck. This takes a template argument allowing it to be disabled, enabled and just log if you use more than the expected amount of stack, or enabled and trap if you use more than the expected amount of stack. This is most commonly used with a macro like this:

#define STACK_CHECK(expected)  
       StackUsageCheck<StackMode, expected, __PRETTY_FUNCTION__> stackCheck

The StackMode template argument is one of StackCheckMode::Asserting, StackCheckMode::Logging, or StackCheckMode::Disabled. Typically, you will use it in logging mode initially, then disabled mode in production. Use it in asserting mode when running representative tests in CI so that it fails if you have increased your stack requirements and not updated the caller.

It's important that the tests that you run in asserting mode have good coverage. It's typically fine for this to be function-granularity coverage: with the exception of variable-length arrays, functions stack usage does not depend on control flow within the function.

5.11. Handling errors

Asynchronous interrupts are all routed to the scheduler to wake up the relevant threads and schedule the correct thread. Synchronous faults are (optionally) delivered to the compartment that caused them. These include CHERI exceptions, invalid instruction traps, and so on: anything that can be directly attributed to the current instruction.

Compartments have two opportunities to handle these, by implementing at least one of two kinds of error handlers.

Rich

Rich error handlers that have access to the full state at the point the error occurred. These can be written in C/C++ and can do things like step over faulting instructions and resume, or provide rich diagnostic information about the interrupted context.

Stackless

Lightweight error handlers that must be written in assembly.

When a hardware exception occurs, the switcher will first look for a rich error handler and prepare a call frame for it. If there is not sufficient stack space to invoke a rich error handler, or if one is not provided by the current compartment, the switcher will then look for a stackless error handler.

The stackless variant will be passed the stack capability as it was on entry to the compartment and the exception cause, but no other information. Most users will not write these but will instead use one that the platform provides.

5.12. Writing rich error handlers

You can provide a rich error handler by implementing compartment_error_handler in your compartment.

enum ErrorRecoveryBehaviour compartment_error_handler(struct ErrorState * frame, size_t mcause, size_t mtval)

This function is passed a copy of the register file and the exception cause registers when a fault occurs. The mcause value will be one of the standard RISC-V exception causes, or 0x1c for CHERI faults. CHERI faults will encode the CHERI-specific fault code and the faulting register in mtval. You can decompose this into its component parts by calling CHERI::extract_cheri_mtval.

std::pair<CauseCode, RegisterNumber> extract_cheri_mtval(uint32_t mtval)

The spilled register file does not contain a tagged value for the program counter capability. This is to prevent library functions that run with interrupts disabled or with access to secrets from accidentally leaking on faults. All other registers will be preserved exactly as they are in the register file.

At the end of your error handler, you have two choices. You can either ask the switcher to resume, installing your modified register file (rederiving the PCC from the compartment's code capability), or you can ask it to continue unwinding.

Error handling functions are used for resource cleanup. For example, you may wish to drop locks when an error occurs, or you may wish to reset the compartment entirely. The heap_free_all function, discussed in Chapter 7. Memory management in CHERIoT RTOS, helps with the latter.

5.13. Using scoped error handling

You are unlikely to ever write a stackless error handler. Most of the time, you will want to link the one provided by the unwind_error_handler target. This can be adopted by adding add_deps("unwind_error_handler") to your compartment's target in xmake.lua.

This implementation is intended to be used with a set of macros that provide exception-like error handling. These maintain a stack of jmp_buf structures, as defined by setjmp. The head of the linked list is stored at the top of the region of the stack that is visible to the current compartment invocation, in a gap left by the switcher. Each CHERIOT_DURING macro invocation pushes an entry onto a stack that allows returning and each CHERIOT_END_HANDLER macro pops the top entry.

Between these two, a CHERIOT_HANDLER is the equivalent of the start of a catch block. This defines the start of a region where code will run if an error is triggered. You can see this in action in Listing 42. This uses __builtin_trap, which the compiler will transform into an invalid instruction, to force a trap. This is a placeholder for anything that might raise an error (including in nested function calls), such as a bounds violation, a use-after-free bug, a null-pointer dereference, and so on.

CHERIOT_DURING

	Debug::log("About to try something unsafe.");
	CHERIOT_DURING
	{
		Debug::log("In during block");
		if (shouldTrap)
		{
			// This will unconditionally trap.
			__builtin_trap();
		}
	}
	CHERIOT_HANDLER
	{
		Debug::log("Something bad happened!");
	}
	CHERIOT_END_HANDLER
	Debug::log("Finished unsafe block.");

If you run this example with shouldTrap set to false then it will generate the following output:

Error handling example: About to try something unsafe.
Error handling example: In during block
Error handling example: Finished unsafe block.

The code in the CHERIOT_DURING block runs, the code in the CHERIOT_HANDLER block is omitted, and control flow resumes after CHERIOT_END_HANDLER. In contrast, if shouldTrap is true then the trap will transition into the switcher, which will then invoke the compartment's stackless error handler, which will then transfer control into the CHERIOT_HANDLER block and you'll see output like this:

Error handling example: About to try something unsafe.
Error handling example: In during block
Error handling example: Something bad happened!
Error handling example: Finished unsafe block.

This lets you do things like release locks or clean up per-call state in case of failure.

5.14. Conventions for cross-compartment calls

If a compartment faults and force unwinds to the caller then the return registers will be set to -1. This makes it easy to use the UNIX convention of returning negative numbers to indicate error codes. The value -1 is -ECOMPARTMENTFAIL and other numbers from errno.h can be used to indicate other failures.

A CHERIoT capability is effectively a tagged union of a pointer and 64 bits of data. You can take advantage of this in functions that return pointers to return either an integer or, if the result is not tagged, an error code.

To see a slightly less-contrived version of error handling, Listing 43 is a version of Listing 40, rewritten to use error handling instead of checking. The original version checked all of the properties of the string and protected itself against concurrent mutation. This version still uses check_pointer, because this also prevents information disclosure by ensuring that the string does not overlap the current compartment invocation's stack. You could omit this check if you are not worried about information disclosure. For capabilities that you write through, this check is very important because otherwise you may write over parts of your own stack and corrupt internal state by accident.

The example then simply tries to read the bytes, assuming that they are a valid null-terminated C string.

/// Write a message to the UART.
int uart_puts(const char *msg)
{
	// Prevent information disclosure, check that this does
	// not overlap with our stack region.  Check for obvious
	// errors at the same time.
	if (!check_pointer(msg))
	{
		return -EINVAL;
	}
	static FlagLockPriorityInherited lock;
	// Prevent concurrent invocation
	LockGuard g(lock);
	int       result = 0;
	// Assume this is a null-terminated string, report an
	// error on exceptions if not.
	on_error(
	  [&]() {
		  for (const char *m = msg; *m != '\0'; m++)
		  {
			  MMIO_CAPABILITY(Uart, uart)->blocking_write(*m);
		  }
	  },
	  [&]() { result = -EINVAL; });
	MMIO_CAPABILITY(Uart, uart)->blocking_write('\n');
	return result;
}

This version uses the C++ wrappers that take lambdas, rather than the macro versions, but the semantics are the same as described earlier. When you run this, you should see exactly the same output as the original version:

No null
Non-malicious string

The string that lacks a null terminator will now simply be written looking for one. The attempt to read one byte past the end will trigger a bounds exception. This will then invoke the second lambda passed to on_error, which will set the error code and resume. The other errors are caught by the check_pointer call. If you remove that call, you will instead see the following output:

No null


Non-malicious string

Now, each call is printing the trailing newline, even if it encountered a fault while reading the message.