ModelRefs / Edge AI — AI Glossary
Edge AI — AI Glossary
Running AI models directly on-device (phone, IoT sensor, laptop) rather than sending data to a cloud inference endpoint.
Overview
Edge inference reduces latency, eliminates cloud costs, and addresses data-privacy requirements for sensitive workloads. Enabled by quantization (INT4/INT8), model distillation, and runtimes like llama.cpp, MLC-LLM, and Apple MLX. Trade-off: smaller, less capable models.
Reference details
| Topic | inference |
|---|---|
| Also known as | on-device AI, edge inference, on-device inference |
| Last reviewed | 2026-06-24 |
Related terms
Example: What you trade for privacy and latency
On-device, the audio never leaves the phone and there is no network round trip. The cost is capability: you are running a few-billion-parameter quantised model against a frontier model in the cloud. Edge wins where the data must not move or the latency budget is tens of milliseconds — not where the task is hard.
Commonly confused with
Edge AI is not the same as a small model. Edge is where inference runs; a small model is what runs there. You can serve a small model in the cloud, and a well-provisioned edge device can run a surprisingly large one. The constraint is the device, not the label.
When to use it
Reach for it when:
- Data must not leave the device for legal or contractual reasons
- The network is unreliable or absent
- Per-request cloud cost at your volume is prohibitive
Reach for something else when:
- The task genuinely needs frontier capability
- You cannot update models across a fleet you do not control
- Device thermal and battery limits would make it unusable in practice
Referenced by
This term is used by the following ModelRefs references:
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Edge AI — AI Glossary.
Frequently asked questions
What is Edge AI?
Running AI models directly on-device (phone, IoT sensor, laptop) rather than sending data to a cloud inference endpoint.
Is Edge AI the same as on-device AI?
Yes — on-device AI, edge inference, on-device inference are common aliases for Edge AI.
What concepts are related to Edge AI?
Closely related concepts include quantization, model distillation, gguf, ollama.