OTA firmware updates for IoT devices: ship new firmware without bricking the fleet

The day a firmware update ships to a device in the field is the riskiest deploy your team ever does. There is no load balancer to shift traffic, no blue-green swap, and no "just roll back the container" safety net — a bad image on a thousand devices is a thousand service calls, and a bad image on a device you cannot physically reach is a brick. Over-the-air (OTA) updates are how connected products stay alive for years, but the difference between a product that updates itself quietly and one that strands its customers is design work done before the first update ever ships.

Design for the failure, not the happy path

Every firmware update fails somewhere: a corrupted download on a flaky link, a power cut mid-flash, an image that passes checksum but crashes on one hardware revision. The happy path — download, verify, flash, reboot — is the easy 10%. The other 90% is the failure model. Before writing a line of update code, write down the failure cases: what happens if the device loses power at each stage, if the server answers with a 500 mid-download, if a device reboots into a partially written image. The physical world does not behave like a test bench — as we covered when turning raw sensor samples into trustworthy product events, each answer becomes a requirement, not a wish.

Sign the manifest, then verify the image

An OTA pipeline is a supply chain, and supply chains get attacked. The device must know two things before it flashes anything: this update was built by us, and this update is intact. That means an offline signing key, a manifest the device can authenticate, and a hash check over the image itself.

The manifest is a small signed document describing the update — version, target hardware, image hash, and a rollback window. Keep verification minimal and auditable on the device side:

# Build side: sign the manifest with the offline private key
manifest = {
    "version": "2.4.0",
    "hw_rev": ["b3", "b4"],
    "image_sha256": "9f2c…a1d7",
    "min_battery_pct": 30,
    "rollback_window_hours": 72
}
signature = ed25519_sign(canonical_json(manifest), OFFLINE_KEY)

// Device side: verify before flashing
if (!ed25519_verify(canonical_json(manifest), signature, PUBLIC_KEY)) {
    reject("bad signature");   // never flash, never retry blindly
}
if (sha256(downloaded_image) != manifest.image_sha256) {
    reject("hash mismatch");
}
if (!supported_hw_rev(manifest.hw_rev)) {
    reject("wrong hardware revision");
}

The public key lives in immutable storage — burned into a write-protected region or provisioned at manufacturing time. If an attacker can rewrite the public key, the signature check is theatre. Treat your signing key the way you treat the provenance of every build artifact: offline, access-controlled, and rotated on a schedule.

Dual partitions: never ship an update you cannot back out of

The single most important hardware decision for OTA is reserving flash for two application slots. The device always runs from one partition and writes the update to the other. Only after the new image verifies does the bootloader switch slots on the next boot. If the new firmware crashes, a watchdog or a failed health check flips back to the known-good slot — automatically, with no server round-trip.

Single-partition devices can still do OTA with a small recovery bootloader, but they trade every update for a window where the device is unbootable, and the recovery story becomes a truck roll. When clients ask us to scope an IoT product on our IoT and hardware engineering service, partition layout is one of the first things we lock down, because it is nearly free to get right at design time and ruinously expensive to retrofit.

Roll out in rings, not to the whole fleet

Even a perfectly verified image can fail against reality — one hardware revision with a marginal power rail, one carrier firmware quirk, one sensor batch that drifts. Staged rollouts bound the blast radius:

• Ring 0 — the lab fleet. Your own devices, within arm's reach, running the same configurations you will see in the field. Update these first and soak for at least one full duty cycle.
• Ring 1 — friendly sites. A handful of customer devices where you have a direct line to the site contact. Ask for confirmation before proceeding.
• Ring 2 — a percentage of the fleet. Typically 5–10%, chosen randomly, watched against error-rate and battery-drain thresholds for several days.
• Ring 3 — everyone else. Only when the earlier rings are quiet.

Between rings, the update is a deployment decision, not a background task. Your rollout server needs a pause button, a per-device status view, and thresholds that halt the rollout automatically — the same instincts that make asynchronous workflows observable apply to a fleet of ten thousand radios.

The pre-launch checklist

Run this list before the first update ships, and again before every major one:

• Signing key is offline — air-gapped, with an access log and a documented rotation procedure.
• Manifest verification on-device — signature, hash, hardware revision, and minimum battery level all enforced before any flash.
• Dual partitions or a tested recovery bootloader — and you have triggered a real rollback at least once in the lab.
• Resume interrupted downloads — range requests or chunked transfer, so a dropped connection does not restart a 20 MB image.
• Delivery is authenticated and rate-limited — the update server is not a public CDN folder; control who can push and pull binaries the way you control anything else sensitive.
• Versioned update metadata in your database — schema-friendly, so rollout history survives the way zero-downtime schema changes keep the rest of your data online.
• Automatic halt conditions — crash-loop rate, connectivity-loss rate, and battery-drain anomalies each have a threshold that stops the rollout without a human in the loop.
• Device-side acknowledgements are idempotent — treat install reports like any other callback: make them safe to retry so duplicate acknowledgements cannot corrupt rollout state.

Telemetry is the second half of the update

Shipping the image is only half the job; knowing how it landed is the other half. Each device should report update-relevant telemetry for the first days after an install: boot count and crash-loop detection, connectivity success rate, battery and temperature deltas, and the versions of the peripherals it talks to. Compare these against the pre-update baseline per ring, not against the fleet average — a 2% battery-drain regression in ring 2 is the signal that saves you from bricking confidence in ring 3.

Keep that telemetry flowing back into the same dashboards the engineering team already watches. OTA updates are not an embedded-only concern; they are a product operations capability, and the custom software around the device — the rollout server, the fleet dashboard, the support tooling — deserves the same engineering rigor as the firmware itself.

A fleet that updates itself reliably becomes a compounding advantage: security patches land in days, features reach customers without a hardware swap, and the product gets better while it sits in the field. The upfront cost — signing infrastructure, dual partitions, staged rollouts — is real, but it is a fraction of one failed fleet-wide update. If you are planning a connected product and want the update story designed before the first board spins, talk to us.