postmarketOS is a community project aiming to bring real mainline linux to your smartphones (and other consumer devices). What was once a far dream, has developed into a vibrant community with many smartphones receiving enough features in mainline kernels for people to daily-drive them. However, community and manual testing can only gets us so far. To be able to scale, avoid regressions, and ensure long-term reliability automated testing is a requirement.

After many years of work and multiple attempts, postmarketOS has been able to build a system to test phones directly into CI. The system is FOSS, distributed, simple enough to be deployed by a hobbyist at a hacker-space for <200€, and easy to scale-up. It uses CI-tron for all the orchestration, and a custom PCB as PDU for phone-specific needs.

In this talk, we will discuss the different approaches we took over time, lessons learned from successes and failures, current status, and a look into the future.