Insights / IoT & Hardware
Edge vs cloud inference: where should your IoT device run its AI model?
Running an AI model on the device or in the cloud is a trade-off between latency, cost, connectivity and accuracy. Here is how to decide for a real IoT product.
Kiran Bandarupalli · 10 Nov 2026 · 5 min read

Every IoT product with an AI feature eventually faces the same question: should the model run on the device or in the cloud? The answer decides your latency, your running costs, what happens when the network drops, and how you ship model updates. Edge inference runs the model on the device or a nearby gateway; cloud inference sends data to a server and waits. Most real products land between the two, and the skill is drawing that line deliberately instead of drifting into it.
Latency: the reason most teams move inference to the edge
A cloud round trip is rarely free: even on a good connection, sending sensor data or an image to a server and back takes tens to hundreds of milliseconds, and on a congested network it can take seconds. For a vibration sensor that must shut down a machine the moment it detects a fault, that delay is the product failing.
On-device inference answers in milliseconds because the data never leaves the board. That matters most when the decision drives a physical action: stopping, alerting, sorting, or rejecting something in real time. If the output only feeds a dashboard someone reads minutes later, cloud latency is usually irrelevant.
Measure the full path from sensor reading to action, not just the model runtime. A model that runs in 8 ms on-device is no help if the sensor pipeline adds 200 ms before it. The snippet below compares the two paths on your own hardware and network:
import time
import statistics
def measure(fn, samples=100):
latencies = []
for _ in range(samples):
start = time.perf_counter()
fn()
latencies.append((time.perf_counter() - start) * 1000)
latencies.sort()
return {
"p50_ms": round(latencies[len(latencies) // 2], 1),
"p95_ms": round(latencies[int(len(latencies) * 0.95)], 1),
}
print("edge :", measure(run_on_device))
print("cloud:", measure(run_via_api))
Compare the p95, not the average — averages hide the slow tail, and the slow tail is what users notice on a bad network day.
Connectivity: what happens when the network goes away
An edge model keeps working when the connection drops. A cloud model stops. That single fact rules out cloud inference for devices in basements, rural sites, moving vehicles, and factories with flaky Wi-Fi.
Where the device must keep working offline, the model and its inputs live on the device, results are stored locally, and sync happens when connectivity returns. The same thinking applies to provisioning and giving each device an identity at scale.
Even when connectivity is reliable, think about what the device sends. Streaming raw video or high-rate sensor data costs bandwidth and money, and can raise privacy questions. An edge model can send a short verdict — "fault detected, confidence 0.94" — instead of the raw feed.
Cost: cheap per device, expensive per fleet
Cloud inference looks cheap for one device and expensive for ten thousand: every request carries compute, egress, and endpoint costs. A fleet of sensors predicting every second can quietly become the largest line on the cloud bill.
Edge inference flips the economics: you pay once, in hardware, and the per-inference cost is near zero. The catch is that the hardware has to be good enough — a model that needs a GPU in the cloud must be shrunk, quantised, or distilled to run on a microcontroller or low-power SoC, and that is real engineering. Custom IoT firmware development often includes exactly this step: fitting the model to the board without losing the accuracy the product depends on.
Do the maths for your fleet size, not your prototype: multiply per-request cloud cost by devices, by requests per day, by 365. If that exceeds the cost of better hardware plus the engineering to fit the model on it, edge pays for itself. If the model runs only a few times a day per device, the cloud is usually cheaper — you rent compute only when you need it.
Accuracy and model updates: where the cloud still wins
The cloud runs the biggest models. A small model on a microcontroller will not match a large model on a GPU cluster, and for some tasks that gap matters — reading a damaged label, classifying an unusual sound, or reasoning over a long sensor history. When accuracy decides and latency allows, the cloud is the right place.
Updates are the cloud's other advantage: a cloud model can be retrained and redeployed in minutes, with no device involved. An edge model has to reach every device in the field, which means over-the-air firmware updates that are signed, staged, and reversible — a real piece of infrastructure, and a failed model update can take a fleet offline just as surely as a failed firmware update.
Whatever you choose, evaluating a model before you trust it applies on both sides: measure accuracy on data from the real deployment, not just the training set.
The hybrid pattern most products end up with
The winning design is usually a split: the device runs a small, fast model that handles the common and urgent cases — detect the event, filter the noise, decide in milliseconds — while the cloud runs the larger model on the cases the device flags, and handles retraining, reporting, and fleet-wide history.
In practice, the edge model scores every reading locally and handles the confident ones on the spot; uncertain readings, plus a sample of confident ones, go to the cloud for a second opinion and retraining data. The cloud side should treat those uploads as background jobs with retries and idempotency, so a dropped connection never loses a sample or double-counts one.
This split also gives you graceful degradation: if the cloud is unreachable, the device keeps its local verdicts, and if the edge model is uncertain while the cloud is down, it falls back to a safe default — alert a human, hold the item, or log the event — instead of guessing.
Decision checklist
- Response time: does the decision drive a physical action within a second? If yes, start with edge.
- Connectivity: will the device ever be offline when it must decide? If yes, edge is mandatory for that decision.
- Data volume: would sending raw data cost more than a better board? Estimate per-device, per-day traffic first.
- Accuracy gap: measure the small edge model against the large cloud model on real deployment data. If the gap changes outcomes, the cloud earns its keep.
- Update frequency: will the model change weekly or yearly? Frequent changes favour the cloud.
- Fleet size: multiply cloud cost by devices and requests per day. Hardware cost is once; cloud cost compounds.
- Privacy: does raw data need to leave the site at all? An edge verdict often removes the question.
- Fallback: define what the device does when the cloud is unreachable, before you ship.
Edge vs cloud inference is a set of measurable trade-offs — latency against accuracy, hardware cost against cloud spend, independence against easy updates. Measure your own numbers, decide which failure modes you can live with, and draw the line where the evidence puts it. If you are scoping a connected product, our IoT and hardware programming and AI and web applications teams build exactly that kind of system. Talk to us about your device and we will help you place the inference where it belongs.