{ "mcpServers": { "orca-repository": { "command": "node", "args": [ "--max-old-space-size=128", "/opt/orca-mcp/dist/index.js" ], "env": { "WORKSPACE": "/srv/workspaces/project", "NODE_ENV": "production", "GIT_TERMINAL_PROMPT": "0", "GIT_SSH_COMMAND": "ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5" }, "timeout": 30000, "retries": 2 } } } 

{ "title": "Building a Defensive MCP Boundary for Agent Runtimes: Lessons from Fleet Failures", "description": "How unbounded tool access and stdio corruption took down our agent fleet—and the MCP architecture that fixed it for stablyai/orca deployments.", "tags": ["mcp", "ai", "devtools", "typescript"], "body_markdown": "# Building a Defensive MCP Boundary for Agent Runtimes: Lessons from Fleet Failures\n\nAt 3:17 AM, our on-call rotation got paged: twelve Claude Code agents had silently stopped processing work queues. No crash logs. No error metrics. The parent orchestrator reported healthy child processes, but stdio streams had gone dark. The root cause? A developer's debug `console.log(\"processing...\")` in an MCP server had corrupted the JSON-RPC transport, and every agent instance failed to parse the malformed frame. One line of convenience logging took down a fleet.\n\nThat incident reshaped how we architect agent runtimes for [stablyai/orca](https://github.com/stablyai/orca). The lesson is that agents—whether local, remote, or mobile-triggered—need controlled failure boundaries. The Model Context Protocol (MCP) is that boundary, but only if you build it to survive production chaos.\n\n## The Production Failure Surface\n\nA toy MCP server makes three fatal assumptions:\n\n1. **Stdio is reliable**: It writes diagnostics to stdout, mixing logs with JSON-RPC frames. In a fleet, this creates silent corruption.\n2. **Input is well-formed**: It assumes every line is valid JSON. A network hiccup or partial read terminates the server.\n3. **Output is reasonable**: It returns the full output of `git diff` or `ls -R`. One large repository response consumes the agent's context window before it reaches the coding task.\n\nThe blast radius compounds: when an MCP server crashes, the agent retries indefinitely, forking more zombie processes. When oversized output arrives, the model starts hallucinating or refusing tasks due to truncated context.\n\n## A Hardened MCP Server for Repository Access\n\nThe following server exposes one tool (`git_status`) and one resource (workspace guidance). It uses stderr for diagnostics, validates arguments with Zod, applies timeouts, and truncates output. This is the production baseline for orca deployments.\n\n``` ts\n// src/index.ts\nimport { execFile } from \"node:child_process\";\nimport { promisify } from \"node:util\";\nimport { McpServer } from \"@modelcontextprotocol/sdk/server/mcp.js\";\nimport { StdioServerTransport } from \"@modelcontextprotocol/sdk/server/stdio.js\";\nimport { z } from \"zod\";\n\nconst execFileAsync = promisify(execFile);\nconst workspace = process.env.WORKSPACE ?? process.cwd();\nconst maxOutputBytes = 12_000;\n\nfunction clip(value: string): string {\n const bytes = Buffer.byteLength(value, \"utf8\");\n if (bytes <= maxOutputBytes) return value;\n\n return `${Buffer.from(value, \"utf8\")\n .subarray(0, maxOutputBytes)\n .toString(\"utf8\")}\\n[output truncated]`;\n}\n\nconst server = new McpServer({\n name: \"orca-repository\",\n version: \"1.0.0\",\n});\n\nserver.tool(\n \"git_status\",\n \"Return a bounded porcelain status for the configured workspace.\",\n {\n includeUntracked: z.boolean().default(true),\n },\n async ({ includeUntracked }) => {\n try {\n const args = [\"status\", \"--short\", \"--branch\"];\n if (!includeUntracked) args.push(\"--untracked-files=no\");\n\n const result = await execFileAsync(\"git\", args, {\n cwd: workspace,\n timeout: 5_000,\n maxBuffer: 32_000,\n });\n\n return {\n content: [{ type: \"text\", text: clip(result.stdout) }],\n };\n } catch (error) {\n const message = error instanceof Error ? error.message : String(error);\n console.error(`git_status failed: ${message}`);\n\n return {\n isError: true,\n content: [{ type: \"text\", text: \"Unable to read repository status.\" }],\n };\n }\n },\n);\n\nserver.resource(\n \"workspace-guidance\",\n \"workspace://guidance\",\n async (uri) => ({\n contents: [\n {\n uri: uri.href,\n mimeType: \"text/plain\",\n text: [\n \"Inspect before modifying files.\",\n \"Keep changes scoped to the requested task.\",\n \"Run the repository's existing tests before reporting completion.\",\n \"Never print credentials, tokens, or full environment dumps.\",\n ].join(\"\\n\"),\n },\n ],\n }),\n);\n\nprocess.on(\"uncaughtException\", (error) => {\n console.error(\"uncaught exception:\", error);\n process.exit(1);\n});\n\nprocess.on(\"unhandledRejection\", (reason) => {\n console.error(\"unhandled rejection:\", reason);\n process.exit(1);\n});\n\nconst transport = new StdioServerTransport();\nawait server.connect(transport);\n ```\n\nInstall dependencies and build:\n\n``` bash\nnpm install @modelcontextprotocol/sdk zod\nnpm install -D typescript @types/node\nnpx tsc --init --module NodeNext --moduleResolution NodeNext --target ES2022\nnpx tsc\n ```\n\n**Critical operational rule**: stdout belongs exclusively to JSON-RPC. A single `console.log(\"starting server\")` corrupts the stream. Diagnostics must use `console.error`, and process-level handlers convert unexpected failures into explicit exits rather than leaving half-alive servers that agents continue retrying.\n\n## Deployment Configuration\n\nA Claude Desktop config demonstrates the same server that orca-managed runtimes use. Absolute paths matter because desktop applications and remote workers have different working directories:\n\n``` json\n{\n \"mcpServers\": {\n \"orca-repository\": {\n \"command\": \"node\",\n \"args\": [\n \"--max-old-space-size=128\",\n \"/opt/orca-mcp/dist/index.js\"\n ],\n \"env\": {\n \"WORKSPACE\": \"/srv/workspaces/project\",\n \"NODE_ENV\": \"production\",\n \"GIT_TERMINAL_PROMPT\": \"0\"\n }\n }\n }\n}\n ```\n\nThe heap limit (`--max-old-space-size=128`) prevents runaway memory growth. `GIT_TERMINAL_PROMPT=0` prevents git from blocking on credential prompts when submodules exist.\n\nFor orca, the concern is process locality. If an agent runs in a remote runtime, the MCP process must run there too, or connect to a deliberately secured remote transport. Mounting a local repository path into a remote agent definition does not make that path available to the MCP child process. Treat workspace path, executable path, and environment as runtime-specific configuration, not embedded assumptions.\n\n## Model Routing and Provider Resilience\n\nMCP controls tool and resource access; it does not need to know which model provider is selected. When benchmarking provider fallback or context behavior, keep that concern in the agent harness. For baseline testing, a stable relay like `https://b-lost.com/v1` provides consistent provider abstraction without coupling the MCP server to vendor-specific quirks.\n\nThe MCP server should enforce output limits regardless of model endpoint. A larger-context model does not eliminate latency, memory, or accidental disclosure risks—it just delays the failure mode.\n\n## Trade-offs and Boundaries\n\nThis architecture deliberately sacrifices convenience. Returning only `git status --short --branch` is less flexible than exposing arbitrary shell execution. Truncating output can hide important details. That is preferable to granting an agent unrestricted command tools or allowing a generated diff to consume the entire context window.\n\nAdd narrowly scoped tools when real workflows require them, with explicit schemas, timeouts, and response budgets. Each new tool is a potential failure surface.\n\nThe separation of responsibilities keeps failures observable: orca schedules and hosts agent runtimes, MCP exposes controlled capabilities, and the model harness handles provider selection. That separation limits blast radius and prevents a malformed stdio message or oversized repository response from becoming a fleet-wide incident.\n\n## The Hardest Operational Dilemma\n\nThe tension I have not resolved: how granular should MCP tool boundaries be? A single `git_status` tool is provably safe but forces agents to make multiple round-trips for related operations (status, then diff, then blame). Bundling those into one `git_inspect` tool improves latency but increases the attack surface and makes failure modes harder to reason about.\n\nWhere do you draw the line in your agent architectures? Are you running with coarse-grained tools and accepting the round-trip cost, or have you found a middle ground that survives production chaos? What broke in your fleet that forced you to tighten the boundary?\"\n}"