When AI moves out of the cloud — into your phone, your laptop, and your car

The straightforward question is: if the cloud is so much more powerful, why bother running AI on small, battery-powered devices? The answer is that running on the device beats the cloud on four things users feel right away — and these four are exactly why the NPU has become a standard part of modern devices.

• Privacy: the most sensitive data — photos, messages, voice, health — never has to leave the device to sit on someone's server. Process it on the device, and that's it. This is the main selling point of Apple Intelligence, which touts "your data stays on the device"
• Low latency: no waiting for data to round-trip to the cloud. On-device inference on a modern phone runs as fast as under 20 milliseconds — fast enough for real-time tasks like blurring a video background, or a car's emergency-brake assist that can't wait "a fraction of a second"
• Offline: it works with no internet. On a plane, underground, or in a weak-signal area, translation, document summaries, and voice commands all work normally
• Lower cloud cost: every answer that runs in the cloud has a real electricity and GPU cost. The more people use AI, the more the cloud bill swells. Moving the small, frequent tasks onto users' devices spares companies from carrying the enormous inference cost in the data center

When these four forces combine, the edge AI-hardware market is growing very fast. It's estimated to climb from about $26 billion in 2025 to ~$59 billion in 2030, at an average of roughly 17–18% a year. And that doesn't even count it gradually becoming a "standard freebie" in nearly every phone and laptop chip.

ภาพประกอบ (privacy.webp)

The simplest way to understand this node is to compare the two paths the same question takes — the cloud path that races across the world, and the on-device path that finishes inside the device itself.

The trick that makes on-device possible is the "shrunken model." A cloud model might have trillions of parameters, too many to fit in a phone. The way out is techniques like quantization (lowering the precision of the numbers in a model, say from 16-bit to 4-bit) and small language models (SLMs) — like Microsoft's Phi family or Google's Gemini Nano, squeezed small enough to run on the NPU in a phone. You trade away some of the very top-end smarts in exchange for it being "in our device" at all times.

The NPU itself is good at this job because it's purpose-built — instead of a general-purpose chip, it's packed with lots of small matrix multiply-add units that work all at once at very low power. Its speed is measured in TOPS, and that number has become the "new spec" every brand races to show off at phone and laptop launches.

This node isn't in a fight to the death with the cloud — it's a partner that splits the work. In the new AI architecture, work that's small, frequent, and privacy-sensitive comes down to the device, while the biggest and most complex work still relies on the cloud. Let's look at who it connects to.

• Complements (doesn't replace) Inference-Optimized Silicon in the cloud: when the on-device model meets work too big to handle, it "hands it off" to the cloud — the idea called hybrid AI. The tiny model on the device handles 80% of everyday work, while the inference chip in the data center takes the heavy remainder. These two layers work seamlessly together
• Uses the same technology as the most advanced Logic Chips: the NPU in a flagship phone is built on a 3nm process, on par with high-end computer chips — so this node is a key customer pulling demand into Semiconductors across the board, especially the advanced-logic side
• Directly unlocks Robotics & Physical AI: robots, drones, and autonomous machines have to "think" on the spot in real time and can't wait for the cloud — the NPU at the edge is the brain that lets them decide for themselves instantly
• Is the heart of the AI that's "closest to people": the AI models and apps in the layer above only truly go mainstream when they run in the device people are holding — this node is the layer that carries AI out of the data center and into the hands of billions of users

The wave changing the game fastest right now is the "AI PC." In 2024, Microsoft set a new standard called the Copilot+ PC, drawing a line that a machine must have an NPU of at least 40 TOPS to run Windows' on-device AI features — and that 40-TOPS line became the "bar" every chip brand had to clear immediately. The result: Intel (Core Ultra ~48 TOPS), AMD (Ryzen AI 300 ~50 TOPS), and Qualcomm (Snapdragon X Elite ~45 TOPS) raced to pack powerful NPUs into laptop chips.

The effect of this bar shows clearly in the sales numbers. Gartner estimates that AI PCs' share of the whole computer market jumped from ~16% in 2024 to ~31% in 2025 (about 78 million units) and will reach ~55% in 2026 (about 143 million units) — meaning that by 2026, more than half of all computers sold will ship with an NPU.

The phone side is accelerating just as fast. NPUs in flagship chips are growing at a startling rate — the Hexagon NPU in Qualcomm's Snapdragon 8 Elite Gen 5 (launched late 2025) hits ~100 TOPS, powerful enough to run a language model on the device without draining the battery. Apple puts a 38-TOPS Neural Engine in its A18 Pro and M4 chips, the foundation of Apple Intelligence and its on-device processing pitch. And MediaTek puts a ~50-TOPS NPU in the Dimensity 9400 — the TOPS number is becoming a spec consumers watch about as much as the camera and the battery.

What's worth watching is that the lineup of players is completely different from the cloud side. In the data center, NVIDIA holds the market in a near-monopoly. But the edge is a fragmented, open battlefield — the phone players (Qualcomm, MediaTek, Apple), the IP player that sells NPUs as designable blocks (Arm), and specialized edge experts (Ambarella in cameras/cars, Synaptics in IoT) each have their own foothold.

The first direction is the NPU becoming "standard equipment" in every device — the way cameras and Wi-Fi once went from special to something every device must have. Once more than half of computers ship with an NPU in 2026, and every flagship phone has an NPU at its heart, having AI on the device stops being a selling point and becomes something everyone simply expects.

ภาพประกอบ (slm.webp)

The second direction is shrunken models getting smart faster than expected. Research on small language models (SLMs) and model-compression techniques advances almost every month. The tiny models running on NPUs today already work close to last year's cloud models. The more this gap narrows, the more work flows down onto the device — and that's the structural tailwind for this node.

The third direction is expansion beyond phones and PCs, into cars and robots. A modern car is a "data center on wheels" that has to process camera footage from all around the vehicle in real time — work that absolutely cannot wait for the cloud. The same goes for the robots and drones in Robotics & Physical AI, which need a brain at the edge to decide for themselves. These new markets are the arena where specialized edge players like Ambarella and Nvidia (on the car-chip side) compete fiercely.

The first risk here is that the biggest, smartest models still live in the cloud. The NPU on the device is good at the small, frequent tasks, but work that needs complex reasoning, very long context, or the latest knowledge still relies on the giant models in the data center. If users keep feeling that on-device AI is clearly "dumber" than the cloud, the willingness to pay extra for a powerful NPU may not be as high as the chip brands hope.

The second risk is fierce, fragmented competition. Unlike the cloud, which has a clear market leader, the edge is an open battlefield crowded enough to elbow each other — Qualcomm, Apple, MediaTek, Intel, AMD, Arm, and many specialized edge players all fighting for the same space. Once the NPU becomes a "standard part" every chip must have, it risks becoming a feature competed on by price rather than by juicy profit.

The third risk is the "software fragmentation" problem. Each brand's NPU has its own toolkit — Qualcomm has QNN, Apple has Core ML, MediaTek has NeuroPilot. Developers who want their AI app to run well on every device have to tune it again and again. This hassle makes real-world adoption of on-device AI lag behind what the hardware is ready for — no matter how fast the chip is, if the software can't keep up, the power goes unused.

In short: this node is the story of AI "moving house" from a far-away data center into the tiny chip of the device we're holding. It makes artificial intelligence faster, more private, and genuinely reachable for billions of people — and the better shrunken models get, the more the line between "AI on the device" and "AI in the cloud" fades away.