Over-The-Air (OTA) updates are one of the most fundamental mechanisms in any connected product. Thanks to OTAs, we can fix problems even after the product has shipped simply by downloading a new firmware over the air. In fact, most connected products perform an OTA update the first thing they do when they are installed: this ensures they always have all the latest fixes as quickly as possible.
OTAs entirely change the entire software in our system, so they are delicate: there are plenty of chances where the mechanism could go wrong. (This is why you sometimes see a message saying "do not switch off until the update is applied".)
But because the OTA mechanism is so important, it has to be completely reliable. There should be no way to break the OTA mechanism, no matter how hard we try. Even if we download only a partial OTA, or if power is cut at the worst possible moment. Nothing should break it. And we should recover as quickly as possible.
OTAs are such an important mechanism that it must be automatically tested on every change to the system. We should have our devices go through the worst possible ways we could wreck havoc to the update mechanism, and see that they still work. The update needs to either fully apply or not apply at all: if the update fails, we should be back at a known-good state.
We need to:
- Survive the interruption. The device should still be alive after an interrupted OTA update, running the original version of the code.
- Recover from the interruption. The device should redo the OTA update, thus ending up on the new version of the code.
So this we need to test: an update either finishes and applies cleanly, or it gets interrupted. The power goes out mid-download, the device reboots on its own, someone hits reset at the wrong moment. When that happens, the device has to recover on its own and finish the job anyway.
In this post we use the Groundrun system to try to crush OTAs on an ESP32-C6 system and to see that we are able to successfully recover every time. We kill the OTA in three different ways: (1) we do a software reset in the middle of the update, (2) we do a hardware reset in the middle of the update, and (3) we pull the power plug on the entire system in the middle of the update. We then ensure that the system is able to recover, restart the update, and complete it. We measure the time it is able to do so.
This is something that any connected product should do as part of its development workflow to ensure that the system is as stable as possible, after every change to the system.
How Over the air updates work
An over-the-air update pushes new firmware to a device over its own network connection, then installes the new firmware to itself. The new image comes from a cloud backend, which knows both which version each device is running and which version the device shold be updated to. Whether the device fetches it right away depends on the product: some updates apply quietly in the background, and others show up in the app first, so the person using the product decides when to accept one. Either way, the device downloads the new image, applies it, and reboots into it.
The tricky part is if the OTA process gets interrupted. In the worst case, a firmware image applied halfway through a write can leave a device unable to boot at all, which is the absolutely worst state that a connected product can end up in. This is typically known as a bricked device: a device that is about as useful as a brick. Completely dead to the world.
Any OTA mechanism must be fully able to avoid bricking the device. The common defense is to never touch the firmware a device is currently running until the new image has confirmed itself. The new image goes into its own separate storage first, gets verified before anything switches over, and the device falls back to what it was running if the new image never completes. Today, every chip vendor provides an OTA mechanism that has this level of production. In our case, the Espressif's ESP-IDF, which the ESP32 in this experiment runs, is one of them. With the default mechanism, an interrupted update cannot leave a device unable to boot.
Since the OTA mechanim protects us against bricking the device, the worst state we can end up in is a device stuck on the old version, with nothing making sure a fresh attempt ever happens. Our system must therefore be able to detect this and recover. We should automatically start a new attempt with the end goal that our device ends up on the target version.
The ESP32
The ESP32 is Espressif's family of Wi-Fi and Bluetooth microcontrollers. The ESP32 is a very common microcontroller, used across smart plugs, sensors, appliances, and plenty of other connected products. The board in this experiment is an ESP32-C6, one of the newer chips in the line, adding Thread alongside Wi-Fi and Bluetooth Low Energy on the same chip. Its firmware runs on ESP-IDF, Espressif's own development framework.
How the ESP32 handles OTA recovery
ESP-IDF implements the general pattern above with two flash partitions, ota_0 and ota_1. The device boots from whichever one its bootloader currently points at, and an update downloads to whichever one is inactive.
The board in this experiment joins Wi-Fi the way a shipped device would join a home network, and the update itself comes from an ordinary HTTPS server, the same path esp_https_ota() uses in a real deployment.
The update downloads to the inactive slot, gets verified, and only then does the bootloader switch the boot pointer and reboot into it. If the new image boots and confirms itself, the bootloader marks it valid and moves on. If it doesn't, CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE sends the device back to the slot it just came from, automatically, with nothing running on top of the firmware involved.
We interrupt the update in three ways
A clean run of this update, undisturbed, takes about ninety seconds on the rig's own network. At some point point during the download we cut the update three different ways.
- Software reset. Sends the board's own console a
force_rebootcommand mid-download, closer to a crash somewhere else in the firmware. - Reset-pin toggle. Pulses its hardware reset line directly, the same debug hardware used to flash it in the first place.
- Hard power cut. The rig's own power switch cuts current to the board, the way a brownout or someone unplugging the product would. This one is a real hardware fault, not a simulated one, and it's the rig doing it automatically as part of the scenario, not a person standing at the outlet.
Each of these methods has the same effect: they stop the OTA partway through writing the new image to the inactive slot. They interrupt before we reach the point where the bootloader would switch the boot pointer, so the device that reboots is still running the old firmware, with the same rollback mechanism ready to walk it through a fresh attempt.
They interrupt before we reach the point where the bootloader would switch the boot pointer, so the device that reboots is still running the old firmware, with the same rollback mechanism ready to walk it through a fresh attempt.
While the three methods has a similar effect, in that they all interrupt the firmware download, they could potentially break the system in different ways. We want to make sure that our system survives and recovers in every case.
How to recover from a failed update
Recovery here means one thing: the device reboots, confirms it is still on the old version, completes a fresh update to the new one, and holds that version through a second, deliberate reset. That second reset rules out a lingering PENDING_VERIFY state, where the bootloader still thinks the new image might need to roll back.
In every trial where the interruption landed, that is exactly what happened. The board came back on the old version, took a second update attempt cleanly, and stayed there. ESP-IDF's own rollback mechanism did the whole job, with nothing on Groundrun's side stepping in, in any of the eighteen trials.
But we need to do more than just confirm that the second firmware works. We need to be through: a thorough test goes one step further than confirming the device runs v2: it drives one more update, from v2 to v3, to check that v2 did not lose its own ability to take a future update along the way.
Comparison
Three interruption methods, two rollback settings, three trials each: eighteen runs total, on real hardware. The numbers are mean recovery time, from the moment the update starts to the confirmed second reset.
| Rollback enabled | Rollback disabled | |
|---|---|---|
| Hard power cut | 99.6s | 93.5s |
| Software reset | 165.7s | 155.3s |
| Reset-pin toggle | 157.5s | 158.5s |
Recovery time clustered by how the update was interrupted more than by whether rollback was enabled. A hard power cut recovered fastest, under two minutes either way, because that recovery run is a clean second attempt with no re-provisioning needed. Software reset and the reset-pin toggle took longer, around two and a half minutes, because their recovery run also rejoins Wi-Fi from scratch.
Varying when interruption occurs
To be even more thorough, we also vary at which point the interruption happens. This does not change the correctness of our experiemnt: we can still recover every time. But it changes how long recovery takes. The comparison above held that fixed, cutting every mechanism at the same instant, right as the download starts.
We used the reset-pin toggle for this one, since it the fastest of the mechamisms to apply, so we can control the timing on a millisecond level. First we measured three clean, uninterrupted downloads to get a real baseline: 89 seconds, in tight agreement across all three runs. Then we timed the reset-pin toggle to land at 25%, 50%, and 75% of that baseline, rollback enabled throughout, three trials at each point.
Every one of the nine cut-point trials recovered automatically, the same as the fixed-point comparison above. What changed was how long it took. Recovery at the earliest cut point, 25% into the download, took just over three minutes. By the time the cut landed at 75%, recovery had grown to close to three and three-quarters. The extra recovery time tracks the added delay closely: it's the download's own progress, redone twice, once before the interruption and once again after.
The extra recovery time tracks the added delay closely: it's the download's own progress, redone twice, once before the interruption and once again after.
Conclusions
Our experiment show that we are able to both survive and recover our interrupted OTA attempts. This is what we expected: we were expecting the raw OTA mechanism in the Espressif IDF to be solid. Across every configuration we interrupted, the device found its own way back through ESP-IDF's own rollback mechanism. Exercised by three different interruption paths, it recovers reliably regardless of a legitimate configuration choice like whether that mechanism is even turned on.
The value of our experiment is in the confidence the mechanism itself brings. We can now run these scenarios repeatedly and be certain that whatever we do, our most important mechanism still always continues to wokr.