perilus, Part Eight

TL;DR: I think I’ve figured out how to run the RISC-V Architectural Certification Tests (ACT) on the perilus simulator. Of the 39 tests in the I set, 20 are passing. My next task will be to fix the remaining 19, which I imagine will be frustrating but enlightening.

I was happy to find the ACT suite because it’s exactly what I wanted: a canonical listing of programs that exercise every instruction in a given ISA extension. I still only know a little bit about RISC-V overall, so I have much more confidence in the correctness and completeness of the ACT suite than my own Chisel tests. If I can get perilus to pass all the tests in the I set, then I’ll consider my implementation to be as close to compliant with the spec as I can get without formal verification. That’ll be good enough for me to build on.

The design of the ACT is pretty cool. Most of the source code by raw tonnage is the assembly for the test programs themselves, and the main output of the ACT’s compilation process is a bunch of ELFs. Each binary contains known inputs and outputs for every instruction in every requested extension. To actually run the tests, you’re supposed to load those ELFs one by one and make sure they pass. But because the space of compliant RISC-V processors is pretty large, the ACT tooling has to allow the end user to specify how to report passes and failures on the particular device under test (DUT) in question. This ends up feeling a lot like a Rust trait, only with C macros instead of associated methods. ACT basically says “you tell me how to report a passing and failing test, and I’ll call that implementation where it’s needed.” There are lots of other optional macros that you can implement for, e.g., printing to the console if your DUT supports that, but the system scales way down to perilus’s level by only requiring the pass and fail macros to have meaningful implementations.

There’s much more to configure, though. The ACT system needs to know the memory layout for your DUT, so that requires a linker script, which isn’t too difficult in perilus’s case. But the really verbose complexity comes from specifying the precise capabilities of the DUT by creating a Sail model for it. This is the area that I know the least about, and I have low confidence that my current model is correct when compared with perilus’s intended capabilities. But I’ve managed to glue together something that appears to mostly work, so for now it’s okay. One benefit of having such a minimal DUT is that, when in doubt, I can favor disabling features. If I don’t know what something is by name, then it seems pretty unlikely that I implemented it by accident.

Anyway, after I got the ELFs to compile, I had to figure out how to actually run them. perilus can’t read ELFs, but I was able to add the ability to read files verbatim into memory. objcopy can convert the ELF tests into a raw memory image by essentially doing the same job as a typical operating system’s program loader, except the output is the filesystem instead of memory. I wrapped these two steps into a script and set it to iterate over all of the test ELFs generated for the I set.

But this still doesn’t quite solve the problem. Sure, the tests get loaded into the simulator, but then what? The simulator needs to run each test until it encounters either the pass or fail macros (or the program counter diverges, or it reaches the cycle count limit). I set up the pass and fail macros to write unique values to the x1 register and then run the ebreak instruction. In the new test subcommand, the simulator does not launch the interactive ratatui interface and instead simply pulses the clock until one of the conditions I mentioned is met. If the simulator sees an ebreak, then it also checks the value in x1 to get the test outcome. If one of the other failure conditions are met, then it just exits. To make it slightly more Unix-y, test failures result in a nonzero exit code.

So that’s how the tests work. Nearly all of this is wrapped up in a few just recipes and the test runner script. To debug the failing tests, I’m probably going to need to improve the simulator. More ergonomic controls in the interactive mode would be nice, but maybe it would be easier to try to write VCD files or even just a log to stdout. The first goal of any debugging technique for this problem will be finding the region of the program that’s causing problems and mapping that back to the assembly source code in the ACT repo. It has good comments and formatting that make it reasonably clear what it’s testing at any given time, so hopefully it’ll be easy to see what’s wrong in each case. For those that aren’t so obvious, I’ll have to dig deeper and potentially create my own minimum failing examples to scrutinize further.