13. Adding a new board

CHERIoT RTOS uses a JSON file to describe the target. At a first glance, this looks similar to a flattened device tree (FDT) source file. Both contain a layout of memory and locations of devices but the CHERIoT RTOS board description file also contains a lot of information that is useful only at build time, such as the locations of header files and preprocessor defines for the platform.

When you want to create a board support package (BSP) for a new CHERIoT configuration, this is the first place to start. The CHERIoT RTOS build system allows board description files to be specified either as names in the sdk/boards directory or as file paths. Anything that has been contributed to the CHERIoT RTOS repository will use the former mechanism, anything distributed separately will use the latter.

13.1. Specifying memory layout

CHERIoT RTOS has to differentiate three kinds of memory in the configuration

• Code and read-only global data, which cannot contain pointers to revokable heap memory.
• Globals and stacks, which may contain pointers to revokable heap memory.
• The heap, which may contain pointers to revokable heap memory and may itself be revoked.

The allocator is given a capability to the revocation bitmap covering the last range. The revoker must be configured to include (at least) the latter two in its scans. The first category is safe to ignore.

The memory layout will put code then globals into memory and then the heap afterwards. In most systems, there is more code than heap and so, to reduce costs, not all memory needs to support tags.

Our security guarantees for the shared heap depend on the mechanism that allows the allocator to mark memory as quarantined. Any pointer to memory in this region is subject to a check (by the hardware) on load: if it points to deallocated memory then it will be invalidated on load. This mechanism is necessary only for memory that can be reused by different trust domains during a single boot. Memory used to hold globals and code does not require it and so an implementation may save some hardware and power costs by supporting these temporal safety features for only a subset of memory. As such, we require a range of memory that is used for static code and data ('instruction memory') that is not required to support this mechanism and an additional range that *must* support this for use as the shared heap ('heap memory'). Implementations may choose not to make this separation and provide a single memory region. At some point, we expect to further separate the mutable and immutable portions of instruction memory so that we can support execute in place.

Instruction memory is described by the instruction_memory property. This must be an object with a start and end property, each of which is an address.

The region available for the heap is described in the heap property. This must describe the region over which the load filter is defined. If its start property is omitted, then it is assumed to start in the same place as instruction memory.

The Sail board description has a simple layout:

    "instruction_memory": {
        "start": 0x80000000,
        "end": 0x80040000
    },
    "heap": {
        "end": 0x80040000
    },

This starts instruction memory at the default RISC-V memory address and has a single 256 KiB region that is used for both kinds of memory.

13.2. Exposing MMIO Devices

Each memory-mapped I/O device is listed as an object within the devices field. The name of the field is the name of the device and must be an object that contains a start and either a length or end property that, between them, describe the memory range for the device. Software can then use the MMIO_CAPABILITY macro with the name of the device to get a capability to that device's MMIO range and can use #if DEVICE_EXISTS(device_name) to conditionally compile code if that device exists.

The Sail model is very simple and so provides only three devices:

    "devices": {
        "clint": {
            "start": 0x2000000,
            "length": 0x10000
        },
        "uart": {
            "start": 0x10000000,
            "end":   0x10000100
        },
        "shadow" : {
            "start" : 0x83000000,
            "end"   : 0x83001000
        }
    },

This describes the core-local interrupt controller (clint), a UART, and the shadow memory used for the temporal safety mechanism (shadow). The UART, for example, is referred to in source using MMIO_CAPABILITY(struct Uart, uart), which evaluates to a volatile struct Uart *, giving a capability to this device.

13.3. Defining interrupts

External interrupts should be defined in an array in the interrupts property. Each element has a name, a number and a priority. The name is used to refer to this in software and must be a valid C identifier. The number is the interrupt number. The priority is the priority with which this interrupt will be configured in the interrupt controller.

Interrupts may optionally have an edge_triggered property (if this is omitted, it is assumed to be false). If this exists and is set to true then the interrupt is assumed to fire when a condition first holds, rather than to remain raised as long as a condition holds. Interrupts that are edge triggered are automatically completed by the scheduler; they do not require a call to interrupt_complete.

13.4. Controlling hardware features

Some properties define base parts of hardware support. The revoker property is either absent (no temporal safety support), "software" (revocation is implemented via a software sweep) or "hardware" (there is a hardware revoker). We expect this to be "hardware" on all real implementations, the software revoker exists primarily for the Sail model and the no temporal safety mode only for benchmarking the overhead of revocation.

If the stack_high_water_mark property is set to true, then we assume the CPU provides CSRs for tracking stack usage. This property is primarily present for benchmarking as all of our targets currently implement this feature.

13.5. Specifying clock speeds

The clock rate is configured by two properties. The timer_hz field is the number of timer increments per second, typically the clock speed of the chip (the RISC-V timer is defined in terms of cycles). The tickrate_hz specifies how many scheduler ticks should happen per second. Ticks were discussed in Section 7.3. Using the Timeout structure, they are the minimum unit of scheduling. Threads sleep integer numbers of ticks. A larger duration here means that a short sleep becomes longer, a smaller duration means that the scheduler will be invoked more often. If you have two threads at the same priority then they will be preempted at this frequency, so a 1,000 Hz tick rate will mean that each runs for 1ms. That means that the scheduler will run (at least) once every 100,000 cycles on a 100 MHz core. With a 100 Hz tick rate, the scheduler will run once every million cycles.

Higher tick rates will (usually) give lower latency, lower tick rates will (usually) give higher throughput.

13.6. Supporting conditional compilation

The defines property specifies any pre-defined macros that should be set when building for this board. The driver_includes property contains an array (in priority order) of include directories that should be added for this target. Each of the paths in driver_includes is, by default, relative to the location of the board file (which allows the board file and drivers to be distributed together). Optionally, it may include the string $(sdk), which will be replaced by the full path of the SDK directory. For example, "$(sdk)/include/platform/generic-riscv" will expand to the generic RISC-V directory in the SDK.

The driver headers use #include_next to include more generic files and so it is important to list the directories containing your overrides first.

13.7. Enabling simulation support

The RTOS has some special behaviour for simulation platforms. If you implement a provide a simulation-exit driver (or rely on the generic RISC-V one) in a platform-simulation_exit.hh file, the RTOS will exit the simulator after the last thread has exited.

The short-spin APIs also behave differently in simulation, where time may not be real. These treat short sleeps as elapsing immediately.

The fail-simulator-on-error.h header includes an error handler that is useful for debugging. It logs an error and will exit the simulator (if run in a simulator) when a CHERI trap occurs. This is useful for trying to determine where code is failing if you do not have access to a debugger.

These behaviours are all controlled by the simulation property. If simulation is set to true then this board is assumed to be a simulation platform, opting into these behaviours.

13.8. Running with xmake run

Most of the examples in this book were run with the xmake run command. This needs to know how to invoke the simulator or deploy to the device.

In addition, the run_command property can be the name of a program (or script) that can either simulate the firmware image or deploy it to a real target. This will be run from the build directory and will be passed the absolute path of the firmware image when xmake run is used. The build system will look for the simulator in the SDK directory and, failing that, in the path. Exact paths can be provided by using $sdk or $board in the name of the simulator. These will be expanded to the full path of the SDK or the directory containing the board description file, respectively.

13.9. Creating board variants

Some boards are minor variations on others. For example, the Sonata simulator is identical to the Sonata FPGA platform, but has a different run command.

Copying the original board file and creating a variation works, but (unless both are generated by some external system) incurs a maintenance burden. The build system provides built-in support for this use case. Rather than providing a .json file for the board, you provide a .patch file. This is a JSON document that uses JSON Patch to describe a set of changes.

This file must have two top-level nodes. The base node contains the board to modify. This can be another patch file or a .json file. The patch node contains an array of JSON Patch directives that are executed in sequence.

This is the entirety of the patch file for the Sonata simulator:

{
  "base": "sonata-prerelease",
  "patch": [
    {
      "op": "replace",
      "path": "/simulation",
      "value": true
    },
    {
      "op": "replace",
      "path": "/run_command",
      "value": "${sdk}/../scripts/run-sonata-sim.sh"
    }
  ]
}

This starts with the Sonata pre-release board file (which tracks the current in-development version of the Sonata FPGA design) and applies two changes. The first sets the simulation property to true, the second replaces the run command with one that invokes the simulator instead of copying the code to the FPGA.