Embedded firmware
The firmware nobody tests: bootloaders, startup and clock configuration
The code that runs before your application is the code that matters most.
By Peter Sords··3 min read
The short answer
The firmware nobody tests is the code that runs before the application: the bootloader, the startup sequence and the clock configuration. It fails in the field rather than on the bench, because the bench never browns out, never loses power halfway through an update and never runs at the edges of the temperature range. Test it with repeated cold boots, power pulled at every stage of an OTA update, clock measurements across temperature, and a golden recovery image.
Your application firmware has unit tests, integration tests, maybe even CI. Good.
But the bootloader? The clock configuration? The startup sequence? Almost nobody tests that code. And it's the code that bricks devices in the field.
The invisible layer
The bootloader, the startup code and the clock configuration all run before your RTOS or your application does anything. The bootloader decides which image to trust. The startup code prepares memory. The clock configuration brings the oscillators and PLLs up to speed.
If any of it fails, your application never starts, and there is no debugger attached in the field to tell you why. A unit that hangs before main() looks exactly like a dead board.
That's why field failures are disproportionately caused by this layer. It runs on every boot, and it gets the least test time of anything in the product.
What goes wrong, and why the bench misses it
- Clock configuration that only fails at temperature extremes. A configuration with thin margin behaves at room temperature and misbehaves at the cold or hot end of the range. A bench at room temperature never sees it.
- Bootloader corruption during an interrupted OTA update. Power drops partway through writing a new image, and the device wakes up with neither the old firmware nor the new one intact.
- Startup race conditions between peripherals. A peripheral gets configured before it is ready. Most boots it wins the race. Occasionally it doesn't, and the failure never reproduces on demand.
- Watchdog misconfiguration that causes reset loops. A timeout shorter than the worst-case startup path resets the device before it finishes booting, every time.
None of these show up on a board powered from a clean bench supply, flashed over a debugger and left running. They show up when the rail browns out, when a user pulls the plug at the worst possible moment, or when a watchdog fires. The question is never "did it pass on the bench?" It's "does it come back?"
How to test the code that runs first
- Power-cycle stress testing. Cold-boot the device a thousand times with logging, and count every boot that doesn't come up clean.
- OTA interruption testing. Pull power at every stage of an update and confirm the device always comes back on a working image.
- Clock accuracy across temperature. Measure the clocks across the full operating temperature range, not just at the bench.
- A golden image and a recovery path. Keep a known-good image the device can always fall back to, and a recovery mechanism that doesn't depend on the image that just failed.
My definition of done
Compiling is not done. Tests passing is not done. A matching CRC on the flash is not done. Those tell you the code is what you think it is, not that the product survives the real world.
Done is the device booting from cold, coming up clean, running its actual job, and shrugging off a hard power cycle in the middle of doing it.
So the last test I run is the rudest one. I cut power while it's working, and I watch. When it wakes up and carries on like nothing happened, that's the green light. Not the test report.
Why this is an architecture decision
If you're shipping a product with OTA updates, your bootloader strategy is one of the most important architectural decisions you'll make. It decides whether an interrupted update is a retry or a returned unit.
It's also far cheaper to decide before the first production build than after the first field failure.
About the author
Peter Sords is the founder and Director of Engineering at Kyros Engineering, a senior-led electronics, PCB, embedded firmware and hardware product development firm in Cleveland, Ohio. More about Kyros
Frequently asked
Why do bootloader and startup bugs reach the field?
Because bench testing rarely exercises them. A board on a clean supply, flashed over a debugger, boots the same way every time. Field units see brownouts, interrupted updates, temperature extremes and watchdog resets, and that is where this layer fails.
How do you test that an OTA update cannot brick a device?
Interrupt it on purpose. Pull power at every stage of the update and confirm the device always comes back on a working image, backed by a known-good golden image and a recovery path that does not depend on the image that failed.
What does done mean for embedded firmware?
For Peter Sords, done means the device boots from cold, comes up clean, does its real job, and recovers from a hard power cycle in the middle of doing it. Compiling, passing tests and a matching flash CRC are necessary, but they are not done.