Edge AI means running inference on the device, inside its power, memory and latency budget. That is an optimisation problem with hard constraints, not a smaller version of cloud inference. Faststream engineers the whole path: architecture selection against the budget, quantisation and pruning, operator fusion, accelerator mapping, and measurement on the real target rather than an estimate.
A cloud model can be large because the constraint is cost per query. An edge model has to fit in the memory that is physically present, complete within a cycle time set by a machine that will not slow down, and do it on the power a battery or a passive enclosure can supply.
Those constraints interact. Reducing precision saves memory bandwidth and energy, but shifts accuracy. Pruning reduces computation, but irregular sparsity often runs slower on real hardware than the dense version it replaced. A smaller architecture may be worse than a quantised larger one. None of this is settled by reading a paper; it is settled by measuring on the target.
| Class | Memory available | Typical workloads |
|---|---|---|
| Microcontroller with DSP extensions | Tens to hundreds of kilobytes | Anomaly detection, keyword spotting, simple classification, sensor signal analysis |
| Microcontroller with NPU | Hundreds of kilobytes to a few megabytes | Small vision models, low-resolution detection, always-on sensing |
| Embedded SoC with NPU or GPU | Hundreds of megabytes to gigabytes | Real-time detection and segmentation, multi-camera, in-cab monitoring |
| Industrial PC or edge server | Gigabytes | Multi-stream analytics, heavier models, several cells aggregated |
| Custom ASIC accelerator | Designed to the workload | Volume products where no available part closes the power, cost or latency budget |
Framework-reported operation counts correlate poorly with wall-clock latency on embedded targets, because the bottleneck is usually memory bandwidth rather than arithmetic. A model with fewer multiply-accumulate operations can be slower than one with more.
So the deliverable includes measured latency, measured power and measured accuracy on the actual hardware, across the input distribution the device will see — including the worst case, which is the one the cycle-time budget has to accommodate.
Edge AI runs machine learning inference on or near the device that produces the data, rather than sending data to a cloud service. It is used where latency must be bounded, where connectivity is unreliable, where data volume makes uplink impractical, or where data must not leave the site.
Useful anomaly detection and simple classification run in tens of kilobytes on a microcontroller. Real-time object detection typically needs an NPU or GPU class device. The right question is not how small a model can be, but what the smallest sensing and model combination is that clears the actual decision threshold.
Usually a little, and sometimes almost nothing. INT8 post-training quantisation with a representative calibration set and per-channel scaling is often within a fraction of a per cent for classification. Small models and regression outputs are more sensitive, and those are where quantisation-aware training earns its cost.
Yes, and if the selected part will not close the budget, that will be said plainly along with what would.
Tell us the specification, the constraint and the deadline. Programmes that cross silicon, radio, embedded and AI are where Faststream is strongest.