RISC-V CPU Part 1: Scoping out

Prologue

This post will describe my adventure in making a RISC-V CPU that boots Linux into userspace. The language I will be using is SystemVerilog and C++ for Verilator testbenches. I will be setting up tests for the CPU and later running it on 2 FPGAs. This post will assume the reader is familiar with HDLs, boolean logic, computer architecture and related fields. I will cover in detail the parts that I found difficult, that aren't obvious how to do, or that aren't really covered in other resources. I will not cover the basics of RISC-V, computer engineering or HDLs.

Defining the project scope

In hindsight, one of my biggest mistakes, and why this took so long, was that I didn't define the project scope precisely. This, as you would expect, led to never-ending scope creep. This project went from creating a small RV32I CPU to a full-blown Linux-capable CPU. The lack of planning led me to spend time on things that I ultimately didn't end up using. For example, I spent a lot of time creating floating point modules which I didn't use in the end because they would have increased the CPU complexity significantly. At this point you need to define WHAT you are making:

There are more decisions like these. Each of them can be broken down further. You don't have to specify everything down to the last detail, but it has to be enough to prevent scope creep.

Why RISC-V

RISC-V is modern, modular, open-source and royalty-free. x86 is incomprehensibly complex. ARM is more approachable but closed-source. RISC-V consists of various extensions. All extensions are either referred to by letter e.g. I, or a short string e.g. Zicsr. Some extensions might require others or might be incompatible with others. This creates innumerable valid combinations. In order to make binaries compatible across CPUs, profiles are defined. Profiles define a set of extensions that a compliant device must implement, e.g. RVA23.

What extensions do I need?

That depends on what you are trying to achieve. For a simple CPU that adds and subtracts, RV32I is sufficient. For booting a standard Linux distro, you need RV64GC. With a custom build you get away with much less. The CPU I am making ends up being RV64IMAUS_Zicsr_Zifencei_Sv39. That means it supports:

Note 1: Zicsr by itself defines a lot of registers. A lot of these don't have to be implemented or can be left as a read-only 0.

Note 2: Linux can be compiled without A. However OpenSBI requires at least LR/SC which are easily implemented. You don't have to implement the AMO instructions.

You can compile Linux in NoMMU mode and run it in M-mode. This removes the need for A, because OpenSBI is no longer needed, and for S-mode, at the cost of compatibility and security.

I implemented the C extension but didn't enable it because it allows instructions (4-byte) to be on a 2-byte boundary, which requires misaligned access.

RISC-V CPU Part 2: Environment and board setup

IDE

Compared to software dev, IDE support for HDLs is much worse. You can either use some text editor or software IDE with an extension for Verilog/VHDL or use a tool like Vivado. I used VSCode with the TerosHDL extension. It's a bit clunky, but it worked. Vivado is AMD's tool to simulate and compile their FPGAs. It's a powerful but complicated tool. And it takes up a LOT of disk space (min 50 GB). For starting out VSCode + TerosHDL is great.

Which FPGA should you get?

My GOWIN/Sipeed experience

Which FPGA you should get depends on your budget and your I/O needs. I initially got a Sipeed Tang Primer 20K Pro board. It was relatively cheap but had a chip with a decent amount of resources and I/O that I wanted. This board had some problems differing in severity. The board has a USB-C port. This port exposes two interfaces, one for UART and the other for JTAG. I had issues with the ftdi_sio driver grabbing the JTAG interface making it unable to be programmed. Gowin software is janky and a child's toy compared to something like Vivado. For example, I had an issue where I hardwired a port to 0 and instead measured it floating with a multimeter. I also had an issue where it (technically) incorrectly optimized something. I could not get the simulator running so simulating a whole design was not something I could do.

The most problematic part was the hardware (HW) issue. When I finally got Linux booting, I noticed that while the Verilator sim ran perfectly, the board had random non-deterministic crashes that were similar but never exactly the same. The most obvious issue was corrupted kernel timestamps.

After I fixed all issues in the sim, I started trying to fix this issue. For about 10 days I tried everything I could think of. I set up proper testing to see if some instructions were working correctly. I tried to use the open-source toolchain for synthesis, but it didn't work. I tried to simulate the whole CPU with the memory controller in Verilator, but it didn't work.

Eventually I discovered that small changes in the oscilloscope (GAO) tool caused the CPU to crash at different places. I suspected that the place-and-route (PnR) or synthesis tools were producing incorrect code somewhere. So I tried to do equivalence checking with Yosys for the synthesis step but could not figure it out. Finally, I looked up reported HW issues for the Gowin chip and found out there is one. I loaded the test payload on my board and found that my chip was affected by this problem. Given the non-deterministic nature of the problem, figuring out exactly what caused it is not really possible.

Alinx/Xilinx

Because of this issue, I bought an Alinx 7035b board. This board had similar I/O but a Xilinx chip. On this board the CPU booted without issues. Even though Vivado is overall better, it still has some issues. The biggest one for me was licensing. The only paid feature I needed was the menu for setting up debug probes. I found this website for finding FPGA boards https://boards.fpgadeveloper.com/

RISC-V CPU Part 3: User space instructions

My CPU has a simple 5-stage "pipeline" (IF, ID, EX, MEM, WB). A RISC-V CPU does not necessarily have to conform to this pipeline; it can be longer or shorter. You can find plenty of examples for this kind of CPU so this part isn't too detailed. The CPU had no real pipelining, meaning that each stage was performed one after the other. I kept it this way to keep complexity as low as possible. I could make it a proper 5-stage pipeline, but when interrupts, virtual memory and non-constant execution time instructions are added, it's not so simple. Immediate encodings are intentionally "weird". I don't really understand why their exact layout is as it is, but make sure to get them right. (If you think these encodings are weird, then check out the encodings for C instructions.)

I set up a simple testbench that loaded a test payload into RAM and ran the CPU. The testbench was written in C++ as I was using Verilator.

The M extension includes instructions for multiplication and division. It includes a bunch of instructions for all signed/unsigned combinations. You have to implement all of them as the Linux kernel uses them. Make sure to later set up tests specifically for these instructions.

The A extension is effectively useless as this is a single-core CPU. Linux can be compiled without A, but OpenSBI requires at least LR/SC instructions.

Initially I wanted to implement the F/D extensions and even went as far as making parameterized, fully functional Verilog modules for Add/Sub/Mul/Div and SQRT (mostly functional) that cover all edge cases. When I saw all the instructions I had to implement, I shelved it for later and never got around to it. A bare-bones Linux does not need FP anyway.

After I got Linux running on the CPU, I wanted to add C instructions to reduce code size. I implemented the decoding with the help of AI, as doing it manually was painful and very error-prone. However, as the C extension allows uncompressed (4-byte) instructions to be only 2-byte aligned, this created problems with memory, as unaligned accesses are not supported.

RISC-V CPU Part 4: CSR, Timer Interrupts and Exceptions

Control and Status Registers (CSRs)

The RISC-V spec defines the CSR address space as 12 bits. The address defines the CSR's minimal privilege level and R/W permissions. A bunch of CSRs are defined by the base specification; however, you don't have to implement all of them to boot Linux.

Required CSRs for booting Linux:
Note: Some of the regs (e.g. MVENDORID) might not actually be needed, but I added them because they are trivial to implement.
Note: RO means read-only.

Linux/OpenSBI will query many other CSRs to check if they exist and which values are supported. Make sure you raise an exception when a non-implemented CSR is accessed; otherwise, Linux/OpenSBI might think some feature is implemented when it's actually not. Some CSRs are shadows of other CSRs. For example SSTATUS is a shadow of MSTATUS. You can implement SSTATUS as a separate reg or just return the value of MSTATUS with the unneeded bits masked out. These include: MSTATUS/SSTATUS TIME/MTIME CYCLE/MCYCLE INSTRET/MINSTRET MIP/SIP MIE/SIE.

Exceptions

The RISC-V spec does not neatly define what needs to happen architecturally when an exception occurs; instead, it's spread in bits of info all over it. When an exception occurs the following is done: Check if the exception is delegated to S-mode by checking MEDELEG (MIDELEG for interrupts). Then the equivalent S-mode CSRs are written to.

Interrupts and Timers

The RISC-V spec defines 3 different types of interrupts: Software, Timer and External. Each of these can be either Machine or Supervisor level. Software interrupts are used for inter-processor communication in multi-core CPUs. As this is a single-core CPU, I will ignore those. Timer interrupts are used by the scheduler, and external interrupts are for I/O. The steps taken when an interrupt is to be raised are the same as for exceptions.

When an interrupt is to be raised, first the appropriate bit in MIP is set. To actually raise an interrupt the following conditions must be met:

The MTIP register is set when MTIME > MTIMECMP. The physical addresses of these regs are arbitrary and OpenSBI/Linux learn about them from the device tree you pass it. I also implemented the Sstc extension because it is relatively simple yet reduced the timer overhead significantly. Sstc adds supervisor timer interrupts. These regs will be part of a CLINT (core local interruptor) device. The CLINT is a memory-mapped device that is part of the device tree.

PLIC

The PLIC (Platform-Level Interrupt Controller) is a device that manages interrupts from various sources. While the memory layout might look complex at first, you don't actually need all of it for a basic I/O device. A simple UART only needs S-mode interrupts.

Virtual Memory

Since my CPU is 64-bit, I implemented Sv39.

You can ignore the A/D bits initially as they are only used for page eviction, which you probably won't need for initial bring-up. Make sure to implement proper page faults as the kernel needs them for booting. Make sure you don't translate addresses in M-mode and implement the MPRV bit as it is used by OpenSBI.

Compiling Linux

Initially I wanted to just use a distro like Alpine to avoid compiling a kernel, but Alpine requires at least RV64GC support which I don't have.

I used Buildroot to make a custom kernel. I took the default config and changed it to fit my needs. I only changed the default Linux and general config. I didn't change anything else.

I compiled OpenSBI in payload mode with the custom kernel. The kernel was built with embedded initramfs. You should go through both configs and see if you can disable things you don't need, as a lot of things are enabled by default.

To make debugging easier, I used the following command-line options:

earlycon=sbi console=ttyS0 initcall_debug ignore_loglevel random.trust_bootloader=on

This enables early console output, debug information for initialization calls, ignores the log level, tells the kernel to use the RNG seed from the DT, and tells it which console to use later.

Do note that the RNG seed is set to all zeros. This is fine for testing purposes but should never be used in production.

Device Tree

I wrote the device tree mainly by looking at other ones and the docs. I intentionally did not add reserved memory for OpenSBI, as it is added by OpenSBI.

Show device tree 
/dts-v1/;

/{
    compatible = "mightycpu";
    #address-cells = <1>;
    #size-cells = <1>;
    chosen {
        stdout-path = "/serial@10000000:115200n8";
        rng-seed = /bits/ 8 <0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
                             0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
                             0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
                             0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
                             0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
                             0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
                             0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
                             0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00>;
    };
    
    memory@0 {
        device_type = "memory";
        reg = <0x00000000 0x08000000>;
    };

    reserved-memory {
        #address-cells = <1>;
        #size-cells = <1>;
        ranges;
    };

    cpus {
        #address-cells = <1>;
        #size-cells = <0>;

        timebase-frequency = <100000000>;

        cpu@0 {
            reg = <0>;
            device_type = "cpu";
            status = "okay";
            
            compatible = "riscv";
            mmu-type = "riscv,sv39";
            riscv,isa-base = "rv64i";
            riscv,isa-extensions = "i", "m", "a", "s", "u", "zicsr", "zicntr", "sstc";
            clock-frequency = <100000000>;

            cpu0_intc: interrupt-controller {
                compatible = "riscv,cpu-intc";
                interrupt-controller;
                #interrupt-cells = <1>;
            };
        };
    };
    
    
    plic: interrupt-controller@30000000 {
        #address-cells = <0>;
        #interrupt-cells = <1>;
        compatible = "sifive,plic-1.0.0";
        interrupt-controller;
        interrupts-extended = <&cpu0_intc 11 &cpu0_intc 9>;
        reg = <0x30000000 0x400000>;
        riscv,ndev = <1>;
    };
    
    
    clint@40000000 {
        compatible = "sifive,clint0";
        reg = <0x40000000 0xC000>;
        interrupts-extended = <&cpu0_intc 3 &cpu0_intc 7>;
    };
    
    serial@10000000 {
        compatible = "ns16550";
        reg = <0x10000000 0x10>;
        clock-frequency = <100000000>;
        current-speed = <115200>;
        no-loopback-test;
        interrupt-parent = <&plic>;
        interrupts = <1>;
        fifo-size = <0>;
    };
};

UART

Everything about UART you can find in this UART guide.

Memory Layout and loading the kernel

I have created a small bootloader that prints "Hello World!" to UART, sets a0 to 0, sets a1 to point to the embedded DT in it, and jumps into OpenSBI.

I build the bootloader with GCC, then use objdump on the .elf to get a binary. I extend this binary to 4KiB and append the fw_payload.bin file from Buildroot.

Then I load this file into RAM starting at 0x1FF000. This address is somewhat arbitrary, but OpenSBI and Linux have alignment requirements, so the payload can't be loaded just anywhere.

The file is sent over UART and loaded into memory.

Conclusion

At this point I had a simple Linux shell. Given that this is the most bare-bones setup possible, there wasn't much I could do with it. The next step is optimizing the CPU and adding more devices.