Three passing test suites and a broken onboarding screen

Three green checkmarks. Zero cross-domain coverage.

A customer unpacks the device, opens the app, and gets stuck on the pairing screen. Every test suite reports success.

This is the default failure mode for connected hardware products, and it keeps happening for a structural reason. Each domain's tests cover that domain on its own. None of them test the moment the domains have to talk to each other.

What domain-specific tests cover

Firmware tests cover boot sequences, peripheral initialization, protocol state machines, low-power behavior, and memory safety. They run on hardware, or a close simulation of it, and they catch regressions inside the firmware.

The app suite covers different ground: UI flows, API contracts, navigation, data parsing. It mocks the device and catches regressions inside the app.

The backend tests cover API endpoints, database queries, authentication, and business logic. They mock the clients and catch regressions inside the backend.

So the firmware's Bluetooth Low Energy (BLE) advertising packet, the parser the app uses for it, and the device registration endpoint in the cloud are all tested. Just never together.

Where the bugs live

A cross-domain bug lives in the mismatch between two domains, so it fails none of the tests each domain runs on itself. A firmware update ships a new BLE advertising structure while the app team's build from two days ago still expects the old one. Both suites pass. The customer gets an error.

A few patterns come up again and again.

The firmware team updates the over-the-air (OTA) update mechanism while the cloud backend still writes the old binary format. Devices in the field accept the update, apply it, and brick. The cloud tests mock the device, so they never saw it. The firmware tests mock the cloud, so they never saw it either.

The app team adds a new required field to the onboarding call. The firmware already sends it and the cloud validates it, so both new suites pass. But an old build of the app, which a good fraction of the user base is still running, does not send the field. That old app was in neither test suite.

The cloud team adds rate limiting to the provisioning endpoint. Normal single-device pairing still works. But the product has a bulk enrollment feature, where a technician pairs 20 devices in a row: the first 10 succeed, the next 10 hit the rate limit. The bulk-enrollment flow was never in a test.

Each of these is a coordination failure. The two teams involved each did their job and tested it correctly. Neither tested the gap between them.

The manual pass sets the shipping pace

The usual answer is a manual pass: someone walks the flows before each release. It catches some cross-domain bugs, and for years it was a workable answer, because the release itself was the slowest step.

The pass has three structural limits.

It is slow. A thorough cross-domain walkthrough for a product with several device variants, app versions, and firmware builds can take days, and release pressure squeezes it, so the "full manual pass" ends up covering the happy path on one device, one firmware build, and the current app.

It is late. It runs when someone schedules it, so a bug from Monday's firmware commit waits for the next pass, and by then five more changes have landed on top of it and the cause is buried.

It is narrow. It covers one path through the matrix: three firmware versions in the field, two mobile platforms, a cloud that just shipped a migration, and a user base that hits every combination.

And the other side of the pass sped up. AI agents now produce changes that span firmware, app, and cloud in minutes, and every one of them asks the same question: does the product still work end to end? When the answer comes from a walkthrough measured in days, the walkthrough sets the shipping speed, however fast the code got written.

The gap between domains

Each team's test suite is that team's responsibility, so the seam between domains doesn't belong to either team and no test suite covers it.

Each team's test suite is that team's responsibility, so the seam between domains doesn't belong to either team and no test suite covers it.

That is why connected products ship with cross-domain bugs a five-minute manual test would have caught. The test infrastructure was built domain by domain. The product is a system that cuts across all of them.

What covering the gap looks like

A cross-domain test exercises the full stack: real hardware, real firmware, a real app instance, a real cloud backend. It scripts the user flow from scratch:

scenario: full-onboarding-and-ota
device: nrf52840-devkit
firmware: latest
app: v3.1.0
phone: pixel-7a / android-14

steps:
  - flash firmware onto device via debug probe
  - boot device, wait for BLE advertising
  - app: scan, discover, and pair device
  - app: complete onboarding flow
  - assert: device appears in cloud dashboard
  - push OTA update (firmware v2.4.1 -> v2.5.0)
  - assert: device reboots and reconnects within 60s
  - assert: app shows updated firmware version

The scenario either completes the flow or fails at a specific step with a specific error. These scenarios sit above each domain's suite, covering the ground between them, and they run continuously through development, from inside the loop where the change is made. An agent reworking the firmware's advertising format runs the onboarding scenario as it works, reads the failure at the pairing step, and fixes it before the change goes anywhere. Engineers decide what has to be validated, and the verification arrives at the pace changes land.

Building it takes physical hardware, orchestration to drive it, scripted app flows, and a hook into the cloud backend's staging environment. That is months of senior engineering time, which is why most teams do not have it. The return is just as concrete: validation, in the loop as the work lands, that a customer can still unpack the device, pair it, and take an update.

The three green checkmarks from your test suites tell you each domain is healthy. Whether your customers can connect is a different question. Cross-domain scenarios give the second question an automated answer too, at the pace your team makes changes.

The three green checkmarks from your test suites tell you each domain is healthy. Whether your customers can connect is a different question.

---

See how Groundrun covers the seam between firmware, app, and cloud, so a customer can still pair the device, on the platform overview.