I saw HeyClicky and was instantly smitten. You talk to your Mac, it looks at your screen, answers out loud, and then a tiny cursor moves to point at the button you were hunting for. Just such a cool idea!
I then noticed it was a subscription. I'm already paying for Claude, and I already use Claude Code on my laptop. Claude Code can read files, run commands, search the web, and talk to my Gmail, Calendar, Drive and Obsidian. I felt silly paying for another AI just to look at my screen.
So instead I built my own! I named it Buddy, it's open source, and it hasn't cost me anything beyond the Claude plan I already had.
The catch: Claude Code handles the brains
The most expensive part of an assistant like this is the model. But Claude Code can run headless. With claude -p and stream-json input and output, I can keep one Claude session running and send it messages, screenshots included. It logs in with your Pro or Max account, not an API key, so it bills the plan you already have.
Once I realised this, it was just a case of building a body around Claude Code:
| Part | What I used | Cost |
|---|---|---|
| Brain | Claude Code (claude -p), on my existing subscription | already paid |
| Ears | whisper.cpp (small.en), running locally | free |
| Voice | Kokoro neural TTS, running locally | free |
| Eyes | A ScreenCaptureKit screenshot of the screen under my mouse | free |
Everything other than Claude runs on my Mac.
How it works
When I say "Hey Buddy", Apple's on-device recognition picks up the wake word, so no audio leaves my Mac. Whisper transcribes the question, and Buddy takes a screenshot plus a list of the buttons and fields in the front app, read through the Accessibility API. All of that goes to Claude.
Claude replies in short sentences, with little tags mixed in to point and draw. Buddy strips the tags out and speaks the words with Kokoro. As each sentence starts playing, Buddy acts on its tags: the cursor moves to the right spot, circles get drawn, and longer explanations show up in a chat panel that renders Markdown and LaTeX.
This was my favourite part! If I ask it to explain a derivation, it tells me the gist out loud and shows the full working in the panel.
Marking up my screen with a pen
Clicky can point. I wanted Buddy to mark things up the way a teacher does with a red pen: circles, arrows, boxes, highlights and little note cards. They all sit on a transparent layer you can click through, so they never get in the way, and Buddy can't see its own drawings when it takes a screenshot.
Giving it access to my computer, safely
Because it's Claude Code underneath, Buddy can actually do things, not just answer. I can tell it to open the cf folder, add something to my calendar, or publish this note. It clicks buttons by their names through the Accessibility API instead of guessing pixels, which is much faster and doesn't miss.
This is where it gets a bit scary, so I took my time with permissions:
- Looking never asks: reading files, searching and taking screenshots just happen.
- If it wants to change anything, it shows a "Needs your OK" card that says exactly what it wants to do, and I allow it with Touch ID. Nothing can be approved by voice.
- Deleting, sending, paying or force-pushing asks every single time, even after I've said yes once.
- Changing Buddy's own permissions needs Touch ID too, and the settings file is signed, so nothing (not even Claude) can sneakily relax them.
Quick shortcuts
I do a lot of competitive programming, so I added some voice shortcuts. When I say "set up this problem", it grabs the Codeforces problem on screen, creates a solution file from my template and opens it in Neovim. When I say "run the samples", it compiles my solution and runs the sample tests.
Each shortcut is a folder with a small JSON file and a shell script. Buddy reads them on every request, so I never have to touch Swift or rebuild anything. As an extra precaution, Buddy only runs a shortcut whose exact files I've approved with Touch ID, so nothing can change what one does behind my back.
What it's built with
- Swift 5 on macOS 14, with no Xcode project:
build.shcallsswiftcdirectly. - The panels and drawings are small web pages in a WKWebView, with KaTeX bundled offline for the maths.
- A tiny
buddycommand-line tool, so Claude drives the screen through Bash (buddy click,buddy circle,buddy board) while the app holds the Screen Recording and Accessibility permissions.
Was it worth it?
Yes! I haven't paid for another subscription, and I've got an assistant that fits how I work. It knows my folders, my shortcuts and my notes, and I can change anything about it in an afternoon.
To be fair to HeyClicky, it's a well-made product that installs in minutes. Buddy is a personal project that grew. You'll need a Mac, a Claude Pro or Max plan, and some patience with macOS permissions. If that sounds like you, everything is open source:



