Nanaimo: Hardware-In-the-Loop Unit Testing

A delicious Python treat that makes on-target testing sweet and satisfying.

Nanaimo is a set of utilities and plugins designed to enable real hardware test apparatuses to be integrated with unit test frameworks like pytest. This can allow on-target tests to run as part of continuous integration pipelines like Buildkite, Bamboo, or Jenkins.

Figure 1.0: Example of S32K dev boards attached to Raspberry PI CI workers running the Buildkite agent and using Nanaimo.

Nanaimo is designed to enable testing of software-defined, physical components in isolation to provide pre-integration verification of software interfaces and behavioral contracts. It adapts asynchronous control and monitoring of these components to fit familiar testing idioms (e.g. x-unit testing) using the popular python test framework, pytest.

Figure 1.1: Block diagram of Nanaimo's relationship to other components of a typical software build and test pipeline.

Nanaimo is not a simulation framework and is not designed to support the complexity of a full hardware-in-the-loop platform. Instead it's focused on testing small integrations with one or two hardware components and instruments using concepts, syntax, and frameworks familiar to software engineers. Examples of these small integrations might include verifying a SPI driver for a microcontroller or ensuring a serial bootloader's upload performance meets expected KPIs. To do this Nanaimo abstractions of instruments provide async interfaces to hardware either directly using available communication busses (e.g. serial or IP network) or by invoking a CLI provided by the instrument vendor. Because of this latter use case some instruments will require additional programs be installed on a system.

Figure 1.2: Example scenario using Nanaimo to test an I2C driver for a microcontroller.

This design is an amalgam of the TLYF (Test Like You Fly) methodology and the Swiss cheese model of failure analysis. Specifically; the goal is to encourage testing on actual or representative hardware early in the testing process of a system to make the cheese loaf less hole-y.