At Glean, we tested a model that does not write text. It only makes decisions.

In our tests, it caught 99 out of 100 prompt injections hidden in documents, in about a tenth of a second, at about half the cost.

Most AI steps are decisions

One AI answer can hide many small calls. What kind of request is this: a question, a summary, or a task? Which model should handle it? Is this prompt safe? Which follow-up should we show?

Each has an answer you can list in advance: a label, a yes or no, or a score. Yet we send each one to a full LLM, wait for it to write text, and parse that text back into a label. It's like asking for an essay when all you need is a checkbox.

What a System 1 model does

A System 1 model is built to answer these bounded questions directly. The model we tested is Jev, a "System One" model from TypeSafe. You send it state, like a prompt, chat history, or JSON, plus typed questions. It returns typed answers with probabilities: yes or no, one pick from a list, or a score. There is no text to parse. Input costs $0.042 per million tokens, and output is free.

By the numbers

  • 8.1x faster routing to the right expert, with better precision on every route
  • 7x faster follow-up suggestions, with timeouts down from 10.2% to 0.8%
  • 90% of unsafe content caught by our first-pass guardrail, at about half the monthly cost
  • 0 training runs to reach 90% of a fine-tuned model's agreement

What we found

Guardrails. A fast first-pass screen checks every prompt, tool output, and response, and an LLM reviews only what it flags. With Jev as the screen, it caught 90% of unsafe content in our test sets, with about 91% precision. It did best where risk is hardest to see:

  • Prompt injections hidden in retrieved documents: 99% caught
  • Malicious code in responses: 97% caught

Across 22.5 million calls on our internal deployment, median time was 137 milliseconds. It runs alongside the LLM, so users do not wait. Monthly cost fell from about $4,000 to $2,100.

Follow-up suggestions. After each answer, a ranker shows at most one next step, or none. When a turn showed nothing, 58% of the time it was not a choice: the LLM ranker had timed out or returned an answer we could not use.

We replaced it with one Jev question whose options were the candidates plus "none." Median ranker time fell from 4.67 to 0.67 seconds, invalid answers went from 10 to 0, and turns with a suggestion rose from 83% to 91%.

Routing. On 751 labeled requests, Jev picked the right expert more often on every route. On one expert, precision went from 47% to 87%. When a request needed an expert, the call was 8.1 times faster at the median.

Query classification. This one was mixed. With no training, Jev agreed with our production labels 66.8% of the time, against 74.5% for a fine-tuned open model. Fine-tuning wins when labels are stable; no training wins when they keep changing.

What made it work

Let the model say "not sure." Every answer has a probability, so you can set a cutoff. If the top pick is 0.49 and the cutoff is 0.50, we show nothing. Showing nothing often beats a weak suggestion.

Give every call a time budget and a fallback. Our chat path allows 2 seconds, no retries. On a timeout or bad answer, we keep today's behavior.

Conclusion: A simple rule

  • Can the output be listed in advance? If not, keep a generative model.
  • If it can, benchmark a decision model.
  • If you also need text, use both: the decision model picks the tool, and the LLM writes the arguments.

The fastest LLM call is the one you never make. Generate when you must. Decide when you can.