Ethernet bring‑up on embedded hardware is supposed to be predictable: reset the PHY, bring up clocks, negotiate the link, and move on with your life. A routine kernel upgrade on an Arria 10 platform turned into an unexpectedly deep debugging journey through PHY reset sequencing, Ethernet CRC errors, and a behavior that only surfaced reliably with PREEMPT_RT.

Along the way, I’ll introduce the basics of the Linux network stack as it applies to embedded devices: how drivers, PHYs, resets, and the kernel’s initialization paths interact, and why ordering matters.
What followed was a debugging journey that went much deeper than expected. The symptoms were loud -CRC errors, severe packet loss at boot, and a driver that only behaved after an unbind/rebind but the root cause was hiding behind assumptions we never realized we’d made. The hardware, PHY, and DMA path all appeared to check out, leaving us to suspect something subtle in the initialization order.

PREEMPT_RT made the failure reproducible, and that was the real clue. It forced us to revisit everything: PHY reset timing, misleading device‑tree reset names, and the subtle ways real‑time preemption changes how the kernel actually runs. A probe path that had always “worked” was only working because the non‑RT kernel had been quietly guaranteeing an execution order it never promised. This talk walks through the investigation, the wrong turns, and where things stand today. While we’ve narrowed the problem to initialization ordering and reset handling, the full picture is still emerging - and the ongoing work highlights how PREEMPT_RT doesn’t break your code, it just stops covering for it.