Let an AI agent drive your Mac for a fiftieth of a cent per step with TypeSafe Computer Use

Computer-use agents built on frontier models work, but every step ships a screenshot to a big model and waits several seconds for a plan. A twelve-step task can cost close to a dollar. Most of those steps do not need a plan at all: they need one choice from a short list of buttons.

awlevin/typesafe-computer-use, called jev for short, is built on that observation. Per the README it drives a Mac toward a goal you type in plain English for about a fiftieth of a cent per step. It reads the screen deterministically, asks a small classifier which action comes next, and only calls a writing model when a text field genuinely needs free text or the classifier has stopped and the screen needs reading.

This post covers what it is good for, the install, and a safe first run.

The cost and speed claims

The README measures one decision on the same screenshot and goal against Claude Opus 5 given a bare screenshot:

It is honest about the catch: the big model read event dates off the pixels and compared them unaided, while the classifier needed date parsing written in code. Every piece of reasoning a frontier model does for free has to be rebuilt as deterministic state.

How one step works

1. The screen is read deterministically: Vision OCR on the frontmost window plus labelled controls from the accessibility tree, which sees icons OCR cannot.
2. Code adds facts: dates and how far off they are, the focused field, the app and URL, and the actions already tried on this screen.
3. One request to TypeSafe answers three choices: which kind of action, which item, which site.
4. The action runs, the loop waits, and the next capture is the only witness of what happened.
5. When the classifier stops, the writer model reads the screen and answers, or hands the run back to you. It never picks an action.

Step 1: install

You need macOS 14 or newer, Python 3.12 or newer and uv. Windows 10 and 11 are experimental.

git clone https://github.com/awlevin/typesafe-computer-use cd typesafe-computer-use uv sync cp .env.example .env # fill in the keys 

The .env variables:

• TYPESAFE_API_KEY (required): used for every decision.
• ANTHROPIC_API_KEY (optional): used for typing free text, writer-proposed URLs and the final answer. The README documents pointing the writer at another model instead, including LM Studio, Ollama or any OpenAI-compatible server.
• CLICKER_EMAIL (optional): enables the type_email action.
• CLICKER_BROWSER (optional): defaults to Google Chrome.

Then grant your terminal Screen Recording and Accessibility in System Settings, Privacy & Security. Without the first, captures are just wallpaper. Without the second, clicks are silently dropped and --act refuses to start.

Step 2: dry run first

It drives your real mouse and keyboard, so start without --act:

uv run clicker "open the Playground" # dry run: one step, prints what it would do uv run clicker-inspect "any goal" # capture and open the annotated screen + payload 

Clear the terminal before you run it. The terminal is on screen, so its text becomes OCR input.

Step 3: let it act

uv run clicker "open the Playground" --act # up to 100 steps uv run clicker "log in" --act --steps 20 --delay 3 # longer and slower uv run clicker "log in" --act --handoffs 0 # the classifier alone 

To stop a live run, press Ctrl-C or slam the mouse into the top-left corner of the screen. The loop also stops on done, low confidence, a stall or the step limit. Every run writes a runs/<timestamp>/ folder, so a stalled run can be replayed offline, and the README says a run folder attached to a bug report is the most useful thing you can send.

What people are building on it

The projects collected on the TypeSafe Computer Use page show it as a base layer:

• A desktop bookkeeping clerk that opens a supplier invoice PDF in Preview, types it into Frappe Books, saves, submits and files the PDF, with a second model rewriting its playbook every ten invoices.
• Otto, an early-alpha open-source computer-use agent for macOS and Windows that pairs the classifier with native accessibility controls and local OCR.
• Pointer, a Windows port that reads your screens on a hotkey and either points at each step or clicks through the task.
• A browser agent prototype that lets the classifier pick each action and hands hard interactions to Browser Use as the fallback.

If your task lives entirely in a browser, Browser Use's Jev Ultrafast applies the same classifier idea to a numbered table of page elements, and jev itself has a browser backend that reads the DOM over Chrome DevTools with no OCR and no screen permission.

Caveats

• It is beta. The README warns of rough edges and settings that change between 0.x releases.
• Every decision calls TypeSafe's paid API, so "a fiftieth of a cent" is still a metered cost, and you need an account.
• It acts on your real machine. Use the dry run, a low --steps value and, for anything risky, the documented Linux sandbox in a container.
• This walkthrough follows the README and has not been run end to end by us.

Source repo: awlevin/typesafe-computer-use. License: MIT.

Want the setup commands for 50 AI repos people actually ship with? Get the free AI Repos That Actually Ship pack: https://dedupedai.com/ship-pack/?utm_source=devto&utm_medium=article&utm_campaign=shippack