We asked AI coding agents to implement six ways to update a Matter device on real hardware (all six worked)

We asked AI coding agents to develop a critical mechanism for a Matter smart home device: Over-The-Air (OTA) updates. Then make sure it worked and measure its performance on real hardware.

We gave them a hardware rig and their goal.

They worked for 29.7 hours, mostly unattended, and were able to implement and validate a set of six OTA methods, measure their performance, and even managed to find a bug in the Matter code along the way.

The task: implement, validate, and measure OTA mechanisms

OTA updates are a fundamental feature for any connected device. This is how new features are added after the device has been installed and how bugs are fixed in the field. In fact, it is so crucial that it is mandated by regulation: the EU's Cyber Resilience Act requires manufacturers to deliver security updates for their connected products.

We asked coding agents to implement OTA for a specific wireless chip (the ESP32-H2) running a specific wireless standard (Matter over Thread) and measured both how fast the updates were and how long it took for the agents to complete their task.

There are two reasons why we did this experiment with an OTA mechanism:

• OTA is a standard mechanism, so the coding agents have plenty of documentation and example code to work from.
• OTA is a critical mechanism, so we want it to be thoroughly tested and mechanically verified.

It took our coding agents 29.7 hours of working time to develop six different OTA mechanisms and to run them on a Groundrun hardware-in-the-loop rig, equipped with an ESP32-H2 devboard, a Silicon Labs EFR32MG24 devboard, an Android phone, and a backend. As the coding agent, we used Claude Code with the Claude Opus 5.5 model at medium effort level.

What are Over-The-Air (OTA) updates?

OTA updates are a critical feature of any connected product: it is the means by which we can update the product's built-in software. As we have written about before, OTA updates are a mechanism that can break, so we need to be careful with them. This is why we want them to be mechanically validated and tested.

An OTA update is a three-step process:

• An update is downloaded to the device
• The device verifies the integrity of the update
• The device reboots into the newly updated version

If the reboot fails, the device re-installs the previous version. This process has to be rock solid. If it fails, we may end up with a device that is unable to boot. This state is known as a "bricked" device: it is effectively as useful as a brick, just sitting there doing nothing. Chip vendors provide well-tested mechanisms to perform the reboot of a new version, but it is on us, the users of the chip, to make sure that the entire flow works, every time.

Because OTA updates are a standard mechanism, there is plenty of documentation and code examples that agents can learn from. Matter, the wireless standard, provides open source code examples of how to do OTAs. Espressif, the manufacturer of the wireless chip that we use in this experiment, also provides open source code examples for OTA methods on their chipsets.

What is Matter and Thread?

Matter is a smart home standard that is being used by an increasing number of smart home device vendors, including Apple, Google, Amazon, IKEA, and Samsung. Because Matter is a standard, Matter devices work together, even if they come from different vendors.

Matter uses three different wireless technologies:

• Bluetooth, to set up a new device
• WiFi, for devices that are always powered such as light bulbs
• Thread, for devices that run on batteries, such as light switches

Because it is designed for low-power systems, Thread is a much slower mechanism than WiFi. In our experiment, we used the Thread technology to test our OTA updates because of this slower speed. The idea is that with Thread, performance differences would be easier to see.

Thread, the network under Matter

Thread is an open networking protocol for small, low-power devices. It is built on the internet's own addressing (IPv6) and runs over the low-power IEEE 802.15.4 radio. Thread builds a mesh network, in which devices pass each other's messages along. The network starts at a border router, the box that connects the mesh to the home network.

The Thread radio network is slow: the raw bit rate of the radio is 0.25 Mbit/second.

The ESP32-H2 chip

For this experiment, we used the Espressif ESP32-H2 chip. It is a microcontroller with a built-in radio: a small chip that holds a processor, memory and radios on one piece of silicon. This is where our firmware runs. It is also this chip that we update.

The chip supports Thread through its built-in IEEE 802.15.4 radio, but also has a built-in Bluetooth Low Energy radio. But it has no Wi-Fi, unlike many other ESP32 chips.

The ESP32-H2 devboard has 4 MB of flash storage. That holds two slots of about 2 MB, one for the running firmware and one for an incoming update.

OTA updates in Matter

In Matter, devices download updates from an update server by downloading one chunk of data at a time. The block size is configurable on a per-download basis, and is 1024 bytes by default. Each block is written to one of the device's two flash slots. We have previously described this method when we looked at breaking an OTA update on purpose.

The update server is placed outside of the Thread network, either in the cloud or on a nearby device. The device reaches the update server over the Thread network and through the border router.

By default, OTA updates in Matter are not encrypted. The specification requires the board to check that an image is authentic and intact, using a signature from the vendor, and leaves encrypting the image itself optional and up to the vendor. The Espressif Matter code has an option for encrypting the firmware, however, which we wanted to see the effect of, so we set that up as part of our experiments.

The hardware connected to a Groundrun desk runner

We connected our ESP32-H2 devboard to a Groundrun Desk runner, a Raspberry Pi-class single-board computer, along with a Silicon Labs EFR32MG24 devboard, and an Android phone.

The Silicon Labs EFR32MG24 devboard acted as a Thread border router, so the ESP32-H2 board would have a Thread network to connect to. The Android phone was used for the non-Matter Bluetooth OTA method, which we added so that we would have a performance baseline.

The Claude Code coding agents can access the hardware connected to the Groundrun desk runner via the Groundrun CLI tool and the Groundrun coordinator.

The task we gave to our agents

We gave the agents the task to implement and measure the performance of OTA over Matter on the ESP32-H2. We did this in three steps:

• One agent surveyed the available OTA mechanisms for Matter over Thread: it read the documentation and the available code and produced a report.
• One agent read the report, and spawned a set of agents to plan a sequence of tasks, each suitable for execution by a single agent.
• A set of agents executed the planned tasks. The tasks implemented the different OTA mechanisms and validated them on hardware, and the last of them ran each mechanism on the hardware 5 times and measured the performance.

In addition to the Matter OTA mechanisms, we also asked the agents to implement a Bluetooth-based OTA method, which was provided by the official Espressif documentation.

The planning agent initially produced a set of 11 tasks, but the agent that executed the tasks found an issue with the Matter code during task 7 (read more about this below), and planned an additional task. Thus the total number of tasks was 12:

The OTA methods

In our experiment, we used the Matter example lighting-app project as the firmware build, compiled for the ESP32-H2 board. But we also needed a second version of the firmware: a slightly different firmware file that we would update the devices with. As the second version, we made a relatively small change that touched 4 files and added 97 lines of code: added a counter of how often the device switched Thread roles, as well as adding a boot counter. We wanted to have a small change so that we could see the effect of a delta-encoding-based approach.

We then varied the way to deliver the update to the board: a full-firmware download, where we varied the block size, and a delta-encoded method. By default, OTA updates in a Matter device are unencrypted, so we also wanted to look at the effect (if any) that encryption would have. And then, as a point of reference, we also wanted to see how our Matter updates compared with a Bluetooth-based update method.

In this experiment, the delta update is 3.7% of the size of the full image.

For each run the rig flashes version 1 onto the board, commissions the board onto the Matter network, downloads version 2 of the firmware onto the device, reboots the device, then reads back the version and waits for the device to rejoin its Thread network. We measure the transfer time from the first block arriving to the device, to the last block written by the device.

Results: How long does an ESP32-H2 take to update over Thread?

With the stock settings, a full 1.6 MB image takes 8 minutes 38 seconds to transfer and the board is back on the Thread network just under nine minutes after the first block. With a smaller block size the OTA takes longer. All thirty runs finished successfully. The board came back on version 2 and rejoined Thread every time.

We see that the block size affects the transfer time: the smaller the block size, the longer the transfer time. This is because of the stop-and-wait semantics of the block transfer mechanism. Each block needs to be successfully delivered before the next one is sent.

Sending a patch instead of the image

A delta update brings the board back on Thread in 41 seconds, against just under nine minutes for the full image. This is because the delta update is much smaller than sending the entire firmware file.

This is not a surprising result: we knew that a delta update would be faster. And we knew that we had made a relatively small change between our two versions.

The rest of each Matter run, outside the transfer itself, takes 13 to 14 seconds: the announcement that tells the board an update is available, the restart, the Thread rejoin, and the first secure Matter session. Five of those 13 to 14 seconds are a restart delay set by the Matter example firmware. That's rounding against a nine-minute update, and a third of the delta update's 46 seconds from the announcement to the first secure session.

Encryption and the phone route

The stock example firmware sends the plain image, which is what our baseline does. An encrypted image keeps the firmware unreadable on its way to the board, which matters when the image holds code the vendor wants to keep private. Espressif's Matter code has an option for it, so it was one of the mechanisms the survey found, and we asked the agents to measure what it costs on a slow Thread network.

The encrypted image took the same time as the plain one. In that variant the image travels encrypted and the board decrypts each block as it arrives. The median was 8 minutes 33 seconds against 8 minutes 38, with the five runs of each overlapping (509 to 537 seconds against 503 to 534).

This is the result we expected.

The baseline: the Bluetooth transfer

To have something to compare the results of the Thread OTA update, our agents implemented the stock Espressif Bluetooth OTA mechanism. This required the agents to drive the Espressif Android app through the Groundrun rig, as seen in the screenshots below.

The Bluetooth transfer speed was on par with the Thread network: it took 8 minutes 4 seconds, about 7% faster than Matter over Thread at stock settings.

This was surprising: we would have thought that the Bluetooth baseline would be much faster than the Thread network download, because the Bluetooth radio runs at 2 Mbit/second, compared to the 0.25 Mbit/second of Thread. It turns out that Espressif's phone app uses the same stop-and-wait semantics as the Matter OTA download mechanism: it sends one packet and waits for the board to acknowledge it before it sends the next. And because the Matter firmware limits each Bluetooth message to 256 bytes, the phone has to split every packet into several smaller writes, each one acknowledged by the board. So the total transfer time is similar.

Results: How long does it take for AI agents to complete this task?

It took the agents 29.7 hours of working time, from start to finish, to implement, validate, and measure all six OTA mechanisms.

The primary reason for us to run this experiment was to see (1) if the agents were even able to complete their task, and (2) how long it would take them to do this. As we saw above, the agents were able to successfully implement, validate, and measure the performance of the six Matter OTA mechanisms.

To measure the time it took for our agents to complete their tasks, we looked at the timestamps of the git commits that they produced as they were working.

Some of the tasks were run in parallel, so the total time the agents spent was a little more than 29.7 hours.

This is the breakdown for each task:

The measurements took the longest amount of time, roughly 8 hours. This is because they were done on a single rig, with the runs made back-to-back. The five 256-byte runs took almost 2 hours, whereas the quick delta runs took only some 12 minutes.

Of the implementation tasks, the delta update task took 4.4 hours, second only to the first full-image update at 4.7 hours. This was in part because the agent found a problem with the code: it was not able to validate its work on the rig. This turned out to be a bug inside the Matter code, which had already been fixed in the Matter tree, but after we had imported it. The agent was able to both figure this out and to find the bugfix in the Matter tree.

After the agent had found this bug, it brought the cause and the newer Matter version to us. We decided to move to the newer version. The agent then validated the delta transfer on the hardware rig and planned a new task that updated the Matter code and verified that the existing OTA mechanisms still worked (Task 8, running for 3.5 hours).

Looking at the breakdown of the measurement task, most of the time was spent waiting for the OTA runs to complete on the rig (75% of the total time). The rest (25%) went to recording the results after each round, writing up the measurements and the gaps between runs.

The setup, gaps, and write-up time was further broken down into:

The delta update code contained a bug and needed a newer version of the Matter code

The agent that worked on the delta update run was quickly able to figure out that it could not validate its own work: the OTA did not work. While the transfer completed, the verification of the new image failed.

The agent was able to validate the delta encoding locally, by patching the version 1 firmware and producing a correct version 2 firmware. But the firmware on the ESP32-H2 was not able to do the same.

The version of the Matter code the firmware was built on dropped 104 bytes of the rebuilt image, the ones right after the image header: its delta write callback returns as soon as it has written the header, so the rest of that buffer was never written. Everything after the header ended up 104 positions too early. The firmware printed an error saying that a segment of the new image was 0x32203832 bytes long. Read as characters, those bytes are the ASCII repressentation of the text "28 2" from the image's build date, which pointed the agents at where the fault was.

It turned out that the Matter project had fixed this in pull request 71824, commit d950721e90. The change was:

             if (!VerifyChipId(header->chip_id))
             {
+                headerDataRead = 0;
                 return ESP_ERR_INVALID_VERSION;
             }
             imageProcessor->chipIdVerified = true;
+            headerDataRead                 = 0;
 
             // Write data in headerData buffer.
-            return esp_ota_write(imageProcessor->mOTAUpdateHandle, headerData, IMG_HEADER_LEN);
+            esp_err_t err = esp_ota_write(imageProcessor->mOTAUpdateHandle, headerData, IMG_HEADER_LEN);
+            if (err != ESP_OK)
+            {
+                ESP_LOGE(TAG, "esp_ota_write failed (%s)!", esp_err_to_name(err));
+                return err;
+            }
         }
     }

The callback no longer returns after writing the header, so the rest of the buffer is written too. It also resets the static counter of header bytes, both when the header is accepted and when it is rejected. The same commit adds the delta update component to the library's requirements in CMake, which also removed a linker problem the agent had already worked around:

     list(APPEND matter_requires espressif__esp_encrypted_img)
 endif()
 
+if (CONFIG_ENABLE_DELTA_OTA)
+    list(APPEND matter_requires espressif__esp_delta_ota)
+endif()
+
 add_prebuilt_library(matterlib "${CMAKE_CURRENT_BINARY_DIR}/lib/libCHIP.a"
                      REQUIRES ${matter_requires})
 

The commit with the fix was 24 commits after the commit the original firmware was built on, d803429. After updating the code to the version with the fix, the board was able to complete its delta OTA and correctly boot up version 2.

The agents found the cause 26 minutes after the failed run. The fix was confirmed on the board 2 hours 8 minutes after it, 75 minutes of that after the decision to move to the newer version.

Conclusions

We wanted to see how well AI coding agents would do on a complex and critical task: to implement, validate, and measure the performance of a set of Over-The-Air (OTA) update mechanisms for a Matter smart-home device. The OTA update mechanism is critical in that it has to work, perfectly, every time, so we want it to be thoroughly and mechanically tested. It is also a well-documented standard mechanism which the agents should be well-equipped to complete.

The Claude Code Opus 5.5 agents worked for 29.7 hours in total, unattended apart from one decision they asked us to make, and were able to complete their goal of implementing and measuring the performance of 6 different OTA mechanisms. Along the way, they also were able to identify a bug that was present in the Matter tree at the time of cloning it, as well as find that there was a fix for the bug committed a few commits after our clone. With the bug fix in place, the OTA mechanism could be validated and measured.

Everything was validated on real hardware, by means of a Groundrun desk rig with an ESP32-H2 devboard, one Silicon Labs EFR32MG24 devboard, and one Android phone connected via USB.

The performance data on the OTA mechanisms was mostly unsurprising: OTA was slower with a smaller block size and a delta encoding method was the fastest. The baseline Bluetooth OTA performance was surprisingly slow, however: we would have expected it to be several times faster than the Thread OTA method. We will follow this up with another blog post.

Our takeaway from this work is that AI coding agents today are well-equipped to do complex, unattended work on embedded / connected products, such as a Matter device and its OTA mechanisms. Done by hand, this work would have taken weeks or months to complete, but our agents were able to do it in just over one day.

---

Explore how Groundrun works: platform overview.