For the past few years, adding AI to a product has usually meant sending the user's text or images to a large model in a data centre and waiting for the answer. That is starting to change. More and more of it can now run on the phone or laptop in front of the user.

The hardware is arriving quietly. Many new phones and laptops include a chip designed for machine learning, often called a neural processing unit. Microsoft's Copilot+ PCs, introduced in 2024, require one capable of at least forty trillion operations per second. Apple's own AI features use a model of around three billion parameters that runs on the device, sending only harder requests to its servers. Google runs a small model, Gemini Nano, on some Android phones, and has been building ways for web pages to use a model inside the Chrome browser.

Why does it matter? Three reasons. Privacy, because the data never leaves the device. Cost, because every request you do not send to a server is one you do not pay for. And speed and reliability, because the feature still works on a train with no signal.

The small models are not as capable as the big ones. They are good at narrow tasks. Summarising a message, suggesting a reply, sorting photos, cleaning up a transcript. They struggle with long reasoning or broad knowledge. So the design question for many features will become which part can run locally and which part needs the server.

What should you learn now? Get comfortable with the idea that a model is a file you ship, with a size, a version and an update cycle, much like any other dependency. Learn the basics of making models smaller, such as quantisation, which stores the numbers inside a model with less precision. And think about the users who have older devices. A feature that works beautifully on a new flagship phone and fails silently on a four year old one is not finished.

Also, be honest in your product. If something runs on the device, saying so is a real advantage. If it does not, do not imply that it does.

The cloud is not going away. But the choice of where your AI runs is becoming a design decision you will be expected to make.

– Asael Shinder