The First Time My AI Agent Sent an Email to the Wrong Person — And Why I Finished It Anyway

"An agent that can't be interrupted mid-task is a demo. One that can — that's a product."

The first time my AI agent tried to send an email to the wrong person, I didn't know how to stop it.

I was six steps into a multi-step task. The message was already drafted. The AgenticBrain was about to hit send. I said: "No — not WhatsApp, email him. Kaif13018@gmail.com."

And it just... stopped. Replanned. Switched tools. Kept the project work it had already completed. Sent the email to the right address.

That moment taught me something no tutorial would have: an agent that can't be interrupted mid-task is a demo. One that can — that's a product.

I'm Kaif. Over the last 14 months I've been building JARVIS OS Agent — an open-source autonomous AI operating system for Windows — on a 2012 Intel i5-3210M with 4GB RAM.

This post is about how to install it on your machine. But first, I want to show you what makes it different — because installing yet another "autonomous agent" without knowing why is a waste of your afternoon.

What JARVIS Actually Does (And Why It Took 14 Months)

Most "autonomous agents" on GitHub are ChatGPT API wrappers with a fancy README. They run a plan, and if something goes wrong, you either wait or kill the process.

JARVIS is different in four specific ways:

1. The loop is genuinely interruptible.
Change your mind mid-task — the plan re-plans in place, completed steps stay intact. No restart. No hallucinated apology loop. This is what took the longest to get right.

2. Memory is a five-tier system, not a chat log.
Rolling episodic history (2 days raw + 13 days summarized), weighted semantic graph with temporal decay, hybrid BM25 + vector RAG, TTL ephemeral state, and a semantic behavioral pattern engine. All running on local embeddings — no cloud calls, no quotas, no cost.

3. It reaches out proactively — but asks before acting.
Background listeners catch emails, WhatsApp messages, PC health alerts. When something needs attention, the agent pauses, explains the context, and waits for your consent. HITL confirmation gates are enforced at the IPC layer, not the prompt — the agent cannot auto-approve its own outgoing messages.

4. Tool integration is plug-and-play via MCP.
18,000+ external servers — GitHub, Brave, SQLite, Notion, Docker. Add one with a single CLI command. Hot-reloads in 2 seconds. No restart, no code edits.

Plus: Android control via Termux, Telegram remote bot, dual-engine voice (Azure + Vosk), and four independent UI overlays (Agent Panel, Typing Popup, STT Popup, Input Popup) with a Unified Portal web dashboard at http://127.0.0.1:8000.

The Constraint That Shaped Everything

Here's the part I want to be honest about: I didn't build JARVIS despite my hardware. I built it because of it.

I had no GPU. I had no 32GB workstation. I had a laptop that struggled to load YouTube without lag.

So every architectural decision became a negotiation with that constraint:

• Cloud LLMs for reasoning — the hard thinking happens on Groq/OpenRouter, not on my CPU
• BAAI/bge-small-en-v1.5 — 33M parameter embedding model instead of 300M+ alternatives. Runs on CPU. Zero cloud calls.
• ONNX-based RapidOCR — no PyTorch dependency for GUI automation
• UIA-first detection — OCR only fires when native accessibility APIs return too few elements
• Lazy service boot — WhatsApp, Telegram, mobile bridges, MCP servers all skip startup entirely if not authenticated

The result: boots in ~50 seconds on any Windows machine from the last decade. Runs comfortably on modest hardware. Actually usable.

That's not a flex. That's just what happens when you can't throw hardware at a problem.

Prerequisites

No GPU. No Docker. No CUDA toolkit. Just Windows, a terminal, and an internet connection.

Step 1: The One-Command Install (5 Minutes)

Open PowerShell (the blue one, not CMD) and paste this:

Set-ExecutionPolicy Bypass -Scope Process -Force; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; & ([scriptblock]::Create((Invoke-RestMethod -Uri 'https://github.com/thekaifansari01/Jarvis-OS-Agent/releases/latest/download/install.ps1'))) 

Here's what my installer actually does — so you know exactly what you're running:

1. Checks if Git, Python 3.11, and Node.js are installed — if not, installs them via winget
2. Clones the repo to %USERPROFILE%\Jarvis-OS-Agent
3. Creates a Python virtual environment with CPU-optimized PyTorch (I deliberately avoid the 2GB CUDA version — most people don't have NVIDIA GPUs)
4. Installs the WhatsApp Baileys bridge via npm install
5. Prompts you for API keys interactively
6. Registers the jarvis command globally

☕ First run takes 5–10 minutes. ~40 Python packages + a 130MB embedding model. Windows might show a UAC prompt for the Visual C++ Redistributable — accept it.

Once it finishes, open a NEW terminal (the current one won't have the updated PATH) and run:

jarvis 

You should see the ASCII banner and a status line confirming voice input is enabled.

Step 2: Configure API Keys

The installer prompts you for keys at the end. If you skipped that (or want to change providers later):

notepad %USERPROFILE%\Jarvis-OS-Agent\.env 

Here's the minimum config to get started:

# FastBrain — for quick commands and chat FAST_BRAIN_API_KEY=gsk_your_groq_key_here FAST_BRAIN_MODEL=llama-3.3-70b-versatile FAST_BRAIN_ENDPOINT=https://api.groq.com/openai/v1 # Primary Brain — for complex multi-step tasks PRIMARY_API_KEY=gsk_your_groq_key_here PRIMARY_BASE_URL=https://api.groq.com/openai/v1 PRIMARY_MODEL=llama-3.3-70b-versatile # Fallback provider (different provider recommended for resilience) FALLBACK_API_KEY=your_openrouter_key FALLBACK_BASE_URL=https://openrouter.ai/api/v1 FALLBACK_MODEL=google/gemini-2.0-flash-001 # Web search TAVILY_API_KEY=tvly-your-key # Speech-to-text DEEPGRAM_API_KEY=your_deepgram_key 

💡 Using local models? Point PRIMARY_BASE_URL to http://localhost:11434/v1 for Ollama or http://localhost:1234/v1 for LM Studio. Both URLs must end with /v1 — that's the OpenAI-compatible route.

Save, restart JARVIS, and you're operational.

Step 3: Voice Setup (Optional, But Recommended)

Two wake-word engines ship with JARVIS:

If you don't want to touch Azure, skip this. Vosk works out of the box.

But if you want "Jarvis" to actually fire reliably:

1. Go to Azure Speech Studio
2. Create a Custom Keyword project (English US)
3. Train Basic (~30 min) or Advanced (~48h)
4. Download the .table file
5. Save as Data/model/azure/jarvis.table
6. Restart JARVIS

I built this dual-engine system after weeks of frustration with Vosk false triggers. The DSP conditioner (85Hz High-Pass IIR filter + speech-gated AGC) and 5-gate acoustic verifier (Dual WebRTC VAD, SNR, ZCR, duration gating) cut false triggers by ~90%.

Step 4: Add Superpowers via MCP

This is where things get fun.

MCP (Model Context Protocol) lets JARVIS plug into external tool servers — GitHub, Brave Search, SQLite, Notion, Docker, Puppeteer — without writing code.

Add a server from the CLI (JARVIS hot-reloads within 2 seconds — no restart):

# GitHub tools jarvis mcp add github npx @modelcontextprotocol/server-github -e GITHUB_PERSONAL_ACCESS_TOKEN=ghp_xxx # SQLite database queries jarvis mcp add sqlite npx @modelcontextprotocol/server-sqlite C:/data/mydb.sqlite # Brave Search jarvis mcp add brave npx @modelcontextprotocol/server-brave-search -e BRAVE_API_KEY=BSAxxx 

Then ask JARVIS: "List my open GitHub issues and summarize them."

The MCP layer is one of my favorite parts of the project — it means I don't have to hardcode every tool. Add a JSON entry, hot-reload, done.

🔗 Browse 18,000+ MCP servers at registry.modelcontextprotocol.io

Step 5: Remote Control via Telegram (Optional)

Want to trigger silent PC commands from your phone?

1. Message @BotFather on Telegram → /newbot → copy the token
2. Stop JARVIS (Ctrl+C)
3. Run:

jarvis bot --activate 

1. Paste the token
2. Restart JARVIS

Now you can message your bot: "Lock my PC" or "Take a screenshot and send it." Commands run silently in the background — your active desktop session stays untouched.

Troubleshooting the Real Issues

Full diagnostic tree: docs · Troubleshooting →

What I Learned Building This

A few honest reflections after 14 months of solo development:

1. Constraints are features. The limited hardware forced me to use cloud LLMs for reasoning and local embeddings for memory. The result is an agent that runs on hardware most people already own — not a demo that only works on a developer's workstation.

2. Interruptible agents are hard. Most "autonomous" tools are fire-and-forget. Making the loop re-plan mid-execution — without corrupting state or losing completed steps — took months of iteration. There's no tutorial for this. You just fail 40 times until it works.

3. HITL is not optional. I initially let the agent auto-send emails. Then it sent one to the wrong "Rahul" in my contacts. That's when I added the consent gate that cannot be bypassed — even if the LLM insists.

4. Local embeddings are underrated. Everyone reaches for cloud APIs. BAAI/bge-small-en-v1.5 runs on CPU with zero cost, zero quotas, zero rate limits. For personal agent memory, that's more than enough.

5. Documentation is half the product. I spent almost as long writing the docs as I did building the agent. It's the difference between "cool GitHub project" and "something people actually trust."

Honest Limitations

I'd rather you know what JARVIS doesn't do before installing:

• Windows-only. macOS and Linux users are out of luck — for now.
• Local LLM inference is slow. A 7B model on CPU runs at ~2 tokens/sec. Use cloud providers unless you're 100% offline.
• Single-user only. No team features, no multi-user support.
• No auto-updates. You manage updates via git pull.

Full limitations page: docs · Known Limitations →

📚 Resources & Links

Final Thought

I built JARVIS because I wanted an AI that actually did things — not one that just talked about doing them. It's not perfect, but it's mine, it's open source, and it runs on hardware most people already own.

If you install it and it works, star the repo. If it doesn't, open an issue — I read every one. If you build something cool with it, tag me. I'd genuinely love to see it.

⭐ Star JARVIS on GitHub · 📖 Read the Docs

— Kaif Ansari