"Can we just run the model ourselves?" is a question that arrives the moment an AI pilot gets a real bill. The honest answer is a short piece of arithmetic, plus a few questions about the customer.

Here is the picture I use, then the sums.

Taxi or car?

  • An API is a taxi. You pay per ride, which here means per token. Use nothing and you pay nothing.
  • Self-hosting is a car. You rent GPUs (the chips that run the model) and run an open-weight model yourself. The cost is fixed each month, and extra trips are almost free.

Running a model to get an answer is called inference. The question is which side of the crossover point you are on.

There is a middle option too: a managed service that runs inside your own cloud account. Think of a rental car with a driver. The data stays with you, and someone else keeps the GPUs running.

Ask these questions in order

  1. Data rules first. If text may not leave the customer's own cloud account, a public API is out before cost matters.
  2. Then skills. Who can run GPUs in production? If nobody, lean towards a managed service in their account.
  3. Then volume. High and steady use favours the car.
  4. Not measured yet? Start on the API and measure.

The break-even sum

All numbers below are made up for the example. Use your own.

  • API price: $2 per million tokens
  • Pilot use: 800 million tokens a month, so the bill is 800 x $2 = $1,600
  • GPUs: $4,000 a month for one set
  • People (part-time engineer and on-call time): $2,000 a month
  • One set of GPUs can serve 4,000 million tokens a month

Self-hosting costs $4,000 + $2,000 = $6,000 a month, used or not.

break-even = fixed cost / API price per million tokens = $6,000 / $2 = 3,000 million tokens a month 

The pilot uses 800 million, far below 3,000. Stay on the API.

Now try ten times the use, which is 8,000 million tokens:

  • API: 8,000 x $2 = $16,000
  • One set serves only 4,000 million, and GPUs come in whole sets, so you need two: 2 x $4,000 = $8,000
  • Add $2,000 for people: $10,000

Self-hosting saves about $6,000 a month on paper. Check quality and on-call before you celebrate.

Owning the car is three layers of work

  • Runtime: the model running on a GPU.
  • Infrastructure: many copies, shared load, GPU supply.
  • Tooling: releases, watching for problems, tests, cost counting.

The weakest layer sets the limit. And someone has to be on call at 3 a.m. That person's time is a real cost.

Caveats

  • The sum is a first filter, not a quote. It ignores quality, traffic spikes and setup time.
  • Idle GPUs cost the same as busy ones. Low use means you pay for hardware that does nothing.
  • "Open-weight means free" is a mix-up. The download is free. GPUs and people are not.
  • Prices change, and tokens are priced per million. Re-run the sum with current numbers.
  • A cheaper model that fails your test questions is not cheaper.
  • You do not have to pick one for good. Many teams self-host the steady load and rent the spikes.

Try it with your own numbers

The full chapter has a decision tree, diagrams of the three layers, and a break-even calculator where you can change the price, GPU cost and volume and watch the answer move. It is free with no signup: Why run models yourself?. It is one of the free chapters of Deployed Lab, a course on AI engineering and the forward deployed engineer role.