Four theories died first.
One of them was ours.
A control board froze partway through a firmware update — in the field, at a different offset every time. We did not write this firmware. Here is exactly how the fault was found, including the parts where we were wrong.
The symptom
It failed in the field, and it never failed the same way twice.
A firmware update would begin, write a portion of the image, and stop. The progress count froze at a different byte offset on each attempt, which ruled out a bad block at a fixed address and ruled in something timing-dependent. The display update, which shares much of the same path, worked every single time.
The customer had a workaround: pull the SD card before updating. It appeared to help. It was not actually the fix, and the reason it sometimes worked turned out to be part of the answer.
No debugger was attached when a unit froze, and halting the core to attach one would have destroyed the state we needed to read. That constraint shaped everything that followed.
What we tried that didn't work
Four plausible theories, tested on hardware, all wrong.
Each of these was checked on a real board, not argued about in a meeting. Ruling a theory out on hardware costs a day. Carrying an unverified theory into a redesign costs a program.
A USB-ID jumper
The board enumerated differently depending on a strap. Plausible, testable, and wrong — the freeze reproduced with the strap in either position.
CAN bus integrity
An update running while the bus was busy looked like a contention story. Bus traffic was captured during a freeze; the bus was clean.
Stack overflow
The usual suspect for a mid-operation hang, and our own first guess. It died on one line of the eventual capture: 660 bytes used of a 2 KB stack.
A bus collision on the display path
The display update always worked, which made the shared path suspicious. Isolating it changed nothing.
How it was actually found
Read the fault out of a running core.
Evidence 01
How do you read a fault without stopping the board?
The first deliverable was not a fix. It was an instrument — sixty lines of Python that attach over SWD in HotPlug mode and decode the fault registers while the core keeps running.
Evidence 02
What one live read settled
Three numbers, taken off a board that was still frozen. Together they killed every remaining theory and pointed at the one subsystem nobody had looked at.
The tell is the third block. The status struct's error field was holding the firmware image's size, and bytesWritten was holding something that was plainly not a byte count. A wild pointer does not produce that pattern. A structured write landing at the wrong destination does — which pointed at DMA, the one place nobody had looked.
Evidence 03
Two DMA masters, one region of SRAM
Nothing in this code is wrong on its own. That is precisely why reading it had never found the defect — it only existed while both masters ran at once.
This is also where the customer's workaround gets explained. Pulling the SD card seemed to help, but the logger wrote internal flash whether a card was present or not, so removing the media never actually stopped the writes. Quiescing the task is the fix. And it has to resume on every exit path — success, error, and user abort — or you ship a device that silently stops logging after one failed update.
Evidence 04
Then their auditor told us our own binary was wrong
He was right, on all three findings. A fix nobody can rebuild is not finished, so closing them was part of the job — and it turned up something worse than the original complaint.
Chasing that down surfaced a second, worse defect: a repository line-ending rule was normalising binaries, corrupting 18 of 25 committed firmware images. A fresh clone produced broken artifacts. The original mismatch had been masked by a test fixture holding an older image whose checksum happened to match the expected value — so every “PASS” had been passing against the wrong binary.
What the customer got
A fixed device, and a build they can trust.
- —The update path fixed at the cause, verified on hardware with the SD card back in: staged, verified, applied, rebooted clean.
- —A reusable fault probe they keep, so the next imprecise fault is a ten-minute read instead of a three-day argument.
- —A byte-for-byte reproducible build, and the repository defect that had been corrupting committed firmware images closed.
- —Every finding their auditor raised answered in writing, including the ones where he was right and we were not.
Stuck on firmware someone else wrote?
That is the work we do most often, and it starts with reading what you already have. Bring the symptom and the codebase; we will tell you what we would instrument first.
Schedule an Architecture CallFrequently Asked Questions
- Can you debug firmware your team didn't write?
- Yes — inherited codebases are the majority of our firmware work. We reproduce the failure on real hardware, instrument it at the register level over SWD, and deliver a written diagnosis before proposing a change.
- What is an imprecise BusFault and why is it hard to debug?
- On ARM Cortex-M, an imprecise BusFault means the offending write was buffered, so the CPU cannot report the address that caused it and BFAR is empty. The fault is reported after the fact, which is why guessing replaces evidence unless you capture the surrounding state.
- How do you debug a board that has already frozen, with no debugger attached?
- Attach over SWD in HotPlug mode, which connects without halting or resetting the core. A frozen board is still holding its own evidence — halting it to attach a debugger destroys the state you need.
- Do you publish client source code in case studies?
- No. We publish our own diagnostic tooling, its output on our bench, and patterns re-authored for publication. Client source, repository paths, customer names and logos are never published.