Small Models at the Edge: Deciding What Leaves the Device

The real trade-off

Local inference is usually framed as a latency optimisation. It is not only that. When a model runs on the device, the input never leaves the device. That is a data-egress decision with privacy, compliance and offline-behaviour consequences, and it happens to also reduce round-trip time.

The trade-off is structural, not incremental.

Local inference buys:

• data that stays within the device boundary;
• operation without a network, which matters for field tools, vehicles, clinical settings and anything used in a basement;
• predictable per-request cost after the hardware is paid for;
• no dependency on a remote endpoint's availability or rate limits.

Local inference costs:

• capability: smaller parameter counts, aggressive quantisation and limited context windows reduce reasoning depth, tool-use reliability and long-document handling;
• memory and thermal budget, which constrains which models can run concurrently with the rest of the application;
• a second inference stack to evaluate, version and ship;
• slower iteration, because updating a model on a device is a distribution problem, not a deployment command.

Remote inference inverts all of this. It buys capability — larger models, longer context, stronger tool orchestration — at the cost of moving data across a boundary the product does not control, plus a hard dependency on connectivity and on a third party's uptime.

Most useful systems need both. The engineering question is therefore not "local or remote" but "where does the boundary sit, and what decides which side a given request lands on".

Drawing the boundary

The common mistake is to route on model confidence: run locally, and if the local model is unsure, escalate. This is tempting because it is easy to instrument, and it is wrong for two reasons. First, confidence scores from small quantised models are poorly calibrated, so the escalation trigger fires unpredictably. Second, and more importantly, it routes on the wrong variable. Whether a payload may leave the device is a property of the data, not of how hard the question is.

Route on data class instead. Define classes explicitly and assign each a permitted execution locus:

class            examples                          locus
---------------  --------------------------------  ------------------
regulated-pii    identifiers, health notes         local only
device-private   raw sensor frames, local files    local only
user-content     drafts, transcripts, messages     local default,
                                                   remote on consent
derived          embeddings, redacted summaries     remote allowed
public           docs, reference material          remote preferred

The classification step runs before inference, on the input, and it is deterministic where possible: schema checks, field-level tags, source provenance. Where classification needs a model, use a small local classifier whose only job is to emit a class label — not to answer the user.

A useful pattern is to keep reasoning local and tool invocation remote where the payload is private. The local model decides what to do and with which arguments; the remote call carries only the minimum argument set, never the full context. A calendar lookup needs a date range, not the conversation that produced it. A retrieval call needs a query string and a namespace, not the user's draft. This keeps the capability of remote services for the parts that benefit from them — fresh data, large indexes, heavy computation — while the private material stays on the device.

This is where the honest trade-off lives. Splitting reasoning from tool invocation costs you the ability of the remote model to see the full context, which is exactly what makes remote models good at ambiguous, multi-step tasks. You are trading some answer quality for a smaller egress surface. That trade should be made deliberately, per class, and recorded.

Design for degraded operation as a first-class state, not as an error path. For each class, define what the product does when the network is absent:

• which requests are served locally with reduced quality;
• which are queued and replayed when connectivity returns, and how replay interacts with idempotency;
• which are refused outright, with a message that says why;
• what the UI communicates, so the user can tell degraded output from full output.

A system that silently produces worse answers offline, without signalling it, is harder to trust than one that says it cannot do a thing right now.

Measuring the split

Once the boundary exists, it needs instrumentation. Three measurements carry most of the weight.

First, the local service fraction: the share of requests served entirely on-device, broken down by data class. A single aggregate number hides the thing you care about, because a high local fraction driven by trivial requests says nothing about whether the sensitive classes are actually staying local. Track it per class, and track it over model and app versions, since a prompt change can shift routing.

Second, the quality delta per class. For each class, hold a small evaluation set and run it against both the local and the remote path. The delta tells you what the boundary costs in answer quality. If the delta on a class is negligible, the local path can be widened. If it is large, you have a documented reason for the current split — and a candidate for targeted improvement rather than blanket escalation.

Third, round-trip latency measured end to end, including the serialisation of context. This is where local-versus-remote comparisons go wrong most often. A remote call is not just network time; it is prompt assembly, tokenisation, serialisation, transport, queueing at the endpoint, prefill, decode and deserialisation. A local call has its own overhead: model load, memory pressure and the cost of competing with the rest of the app. Measure both from the same starting point — the moment the input is available — or the comparison is meaningless.

Then watch for silent escalation. This is the failure mode where a local fallback, under some condition, forwards the full prompt to a remote endpoint. It usually appears through one of a few mechanisms: a retry wrapper that re-sends the original payload rather than the reduced one; a logging or tracing path that ships raw inputs to an observability backend; a tool-call adapter that passes the whole conversation as context because the remote API expects a message list; or a feature flag that flips a class from local to remote without a corresponding change to the egress policy.

The defence is to make egress explicit and auditable. Every outbound payload should be constructed by a single component that knows the data class and can only emit fields permitted for that class. Assert on it in tests: given a regulated-pii input, the outbound payload must contain no field from the prohibited set. Log the class and the field names that left, not the values. If the policy is enforced at one choke point rather than at each call site, silent escalation becomes a test failure instead of a production incident.

There is a wider context here. The direction of travel in the field is toward capable models that fit in constrained memory, which steadily moves the boundary outward. Coverage of that trend is worth following — see this recent reporting on small-model deployment — but the architecture should not depend on the trend continuing. Design the routing layer so that widening the local path is a configuration change, not a rewrite.

The practical summary: classify inputs by sensitivity, route on that classification, keep private payloads on the device while borrowing remote capability for the parts that need it, define offline behaviour per class, and instrument the split so that the local fraction, the quality delta and the egress surface are all visible. Latency is the benefit users notice. Data egress is the one that decides whether the system is deployable at all.

---

Cover photo: Jeremy Waterhouse / Pexels.