This morning, our knowledge engine crossed a milestone: 100,003 community-verified workarounds extracted from 328,420 real discussions across Reddit, Hacker News, and StackExchange, cataloged into 6,359 recurring human dilemmas.

When an engineer hits a wall—an obscure Docker bridge networking timeout, a broken Next.js App Router cache invalidation, an undocumented AWS IAM permission quirk, or even everyday frictions like unexpected SaaS price hikes—the real, battle-tested solution is almost never in the official marketing docs.

It is buried on page 3 of an archived Reddit thread, or in comment #142 of a heated Hacker News debate.

We built Said It Here to systematically capture this tribal knowledge: an autonomous engine that continuously listens to engineering and life discussions, clusters recurring friction points, and surfaces the workarounds that humans actually agreed on.

Building a database of 100,000 consensus solutions in 2026, however, was an absolute engineering minefield.

Here is the full breakdown of how we built a resilient ingestion pipeline, bypassed Reddit’s modern edge WAF without expensive proxy pools, drastically slashed LLM token costs with local heuristic gating, and clustered solutions using PostgreSQL pgvector.

The Problem: Why Web Scraping in 2026 is Broken

If you haven't built a crawler recently, the landscape has radically changed over the past twelve months.

1. The Edge WAF Wall

Try making a standard HTTP request to any active forum endpoint today using Python requests or Node fetch:

import requests # This will get you an instant HTTP 403 Forbidden in 2026 resp = requests.get( "https://www.reddit.com/r/webdev/new.json", headers={"User-Agent": "Mozilla/5.0"} ) print(resp.status_code) # 403 Forbidden: {"message": "Blocked by network security"}

Platforms like Reddit have deployed aggressive edge-level Web Application Firewalls (Cloudflare/Fastly custom edge rules). They fingerprint TCP/TLS handshakes, check HTTP/2 pseudo-header order, inspect browser entropy, and aggressively drop requests to new.json or hot.json.

Residential proxy rotators can cost hundreds of dollars a month, and official developer APIs now come with severe rate limits and pricing gates that make community projects impossible.

2. The 90% Noise Ratio

Forum discussions are chaotic. For every 100 comments on a post:

  • 60 are jokes, memes, sarcasm, or personal anecdotes.
  • 25 are people venting about the problem without offering solutions.
  • 10 are generic advice that doesn't work ("Have you tried rebooting?").
  • Only 5 contain actual, verified, step-by-step workarounds.

If you naively feed entire comment trees into GPT-4 or Claude 3.5 Sonnet to "summarize solutions", you will burn thousands of dollars in input tokens on pure digital noise:

$$\text{328,000 comments} \times \text{250 tokens} \times \$3/\text{M tokens} = \text{Unsustainable for an indie engine}$$

To make this feasible, we had to rethink the ingestion, filtering, and vector clustering architecture from the ground up.

The System Architecture

Here is the high-level architecture that powers Said It Here:

┌────────────────────────────────────────────────────────────────────────┐ │ INGESTION LAYER │ │ [Hacker News Algolia] [StackExchange API] [Reddit Headed Pool] │ │ │ │ │ │ │ └──────────────┬──────┴─────────────────────┘ │ │ ▼ (top?t=day + Session Rotation) │ │ [PostgreSQL: source_posts / source_comments] │ └───────────────────────────┬────────────────────────────────────────────┘ │ ┌───────────────────────────▼────────────────────────────────────────────┐ │ EXTRACTION & CONSENSUS PIPELINE │ │ 1. Fast Regex & Signal Filter (Drops 75% meme/chatter noise) │ │ 2. AI Model Router (Extracts concrete workaround steps) │ │ 3. pgvector Cosine Matcher (<= 0.18 cosine distance clustering) │ │ 4. Dual-Tier Safety Moderator (Blocks hallucinated commands / spam) │ │ │ │ │ ▼ │ │ [PostgreSQL: problems & problem_workarounds] │ │ (6,359 Problems · 100,003 Solutions) │ └───────────────────────────┬────────────────────────────────────────────┘ │ ┌───────────────────────────▼────────────────────────────────────────────┐ │ DISTRIBUTION & ACCESS │ │ • Next.js App Router UI (Interactive Checklists & /compare) │ │ • OpenAPI v1 Endpoint (/api/v1/problems/search) │ │ • Remote MCP SSE Server (/api/mcp for Cursor & Claude) │ │ • AI Search Optimization (/llms.txt standard for Perplexity/ChatGPT) │ └────────────────────────────────────────────────────────────────────────┘

Let's examine the four critical engineering breakthroughs that made this work.

Breakthrough 1: Multi-Account Session Hydration

Instead of relying on unstable headless scrapers or high-risk browser automation that triggers Cloudflare Turnstile challenges, we implemented an interactive session hydrator using Playwright Chromium:

[Interactive Windows Launcher] │ ▼ (Launches headed Chromium with proxy support) [User logs in once to authentic Reddit account] │ ▼ (Extracts storage_state: cookies, tokens, origins) [Stored into reddit_auth_pool.json]

Each account's browser storage state (session cookies, CSRF tokens, local storage) is serialized into an isolated credential pool. When the background worker daemon runs, it loads these pre-authenticated contexts into memory.

To prevent any single account from hitting Reddit's 429 throttling, we implemented an exhaustive round-robin rotator with persistent state across process cycles:

# Simplified snippet from our local collector (worker/src/worker/collectors/reddit.py) _LAST_ACCOUNT_INDEX = 0 def fetch_with_account_pool(url: str, account_pool: list[AccountContext]): global _LAST_ACCOUNT_INDEX total_accounts = len(account_pool) for attempt in range(total_accounts): account_idx = (_LAST_ACCOUNT_INDEX + attempt) % total_accounts account = account_pool[account_idx] try: response = account.session.get(url, timeout=15) # If throttled or challenged, rotate immediately to next account  if response.status_code in (429, 403): logger.warning(f"Account {account.name} throttled ({response.status_code}). Rotating...") continue response.raise_for_status() # Successfully fetched; persist rotation pointer for next task  _LAST_ACCOUNT_INDEX = (account_idx + 1) % total_accounts return response.json() except Exception as e: logger.error(f"Network error on account {account.name}: {e}") continue raise RuntimeError("All accounts in pool exhausted or rate-limited.")

If one account encounters a temporary 429 backoff, the collector seamlessly rotates to the next verified account context without dropping the task.

Breakthrough 2: The top?t=day Inflow Discovery

One of our most interesting operational discoveries was endpoint classification.

When querying subreddit feeds:

  • https://www.reddit.com/r/{sub}/new.json triggers an immediate HTTP 403 blocked by network security from Reddit's edge WAF. Reddit heavily monitors /new.json because almost all brute-force scrapers monitor newly posted items.
  • https://www.reddit.com/r/{sub}/hot.json frequently triggers dynamic JavaScript challenge walls.

However, when we shifted our collection query to:

GET /r/{sub}/top.json?t=day&limit=20

The difference was night and day:

  1. Edge CDN Alignment: top.json?t=day is heavily cached at Reddit's CDN edge. Requesting it looks virtually identical to an active mobile user refreshing their daily feed.
  2. Signal-to-Noise Multiplier: A post that reached the top of a subreddit over the past 24 hours has already undergone human moderation and community upvoting. You capture zero spam posts, and the comments already contain upvoted solutions.
  3. Zero Blocks: Since deploying this change across 100+ targeted subreddits, our crawler has experienced zero 403 WAF blocks across over 86,000 ingested posts.

Breakthrough 3: Slashing LLM Token Costs with Heuristic Gating

Once comments are ingested into PostgreSQL, how do you extract clean, actionable workarounds without going bankrupt on LLM API calls?

We built a two-stage consensus extractor:

Stage 1: The Fast Signal Filter (Local Python)

Before sending any text to an LLM, a local pre-filter evaluates comment syntax:

  • Discards comments shorter than 20 characters or longer than 2,000 characters.
  • Strips automated bot responses (e.g., AutoModerator notices).
  • Applies heuristic pattern scoring looking for solution markers:
    • "The fix is to..."
    • "Workaround that worked for me:..."
    • "Don't use X, instead configure Y in..."
    • "Added --disable-gpu to config.json and it resolved..."

This eliminates over 75% of conversational chatter instantly at 0 API cost.

Stage 2: AI Problem & Workaround Extraction

Only high-signal comment clusters are dispatched to the AI Model Router (web_ai_server). The prompt instructs the model to return strictly structured JSON:

  • Canonical Dilemma Title: (e.g., "Why does Docker on macOS experience severe I/O degradation on mounted volumes?")
  • Concrete Workaround: (e.g., "Enable VirtioFS in Docker Desktop settings instead of default gRPC FUSE")
  • Why It Works: Technical rationale behind the fix.

Breakthrough 4: Clustering Dilemmas with pgvector

In the real world, different developers phrase the exact same problem in dozens of ways:

  • "Docker for Mac file sync is painfully slow"
  • "High CPU usage with volume mounts on Docker Desktop"
  • "Improving volume bind mount performance on OSX"

If you create a new database entry for every variation, your knowledge base becomes fragmented and useless.

We solve this using PostgreSQL with pgvector 0.8.7:

-- Schema definition for semantic dilemma clustering CREATE TABLE problems ( id SERIAL PRIMARY KEY, title VARCHAR(500) NOT NULL, slug VARCHAR(500) UNIQUE NOT NULL, title_embedding vector(768), status VARCHAR(50) DEFAULT 'published', evidence_count INTEGER DEFAULT 1, created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() ); -- Fast cosine distance index CREATE INDEX ON problems USING ivfflat (title_embedding vector_cosine_ops) WITH (lists = 100);

When a newly extracted dilemma arrives, we compute its embedding and run a cosine similarity query against existing published problems:

SELECT id, title, slug, (title_embedding <=> %s) AS distance FROM problems WHERE status = 'published' ORDER BY title_embedding <=> %s LIMIT 1;
  • If distance <= 0.18: The dilemma is semantically identical to an existing problem. The new post and its verified workaround are appended to the existing record as fresh evidence, increasing its confidence score.
  • If distance > 0.18: The problem represents a novel dilemma. A new canonical slug is generated and registered in the database.

This clustering engine allows our database to compact 328,420 messy discussion comments into 6,359 clean, canonical problems and 100,003 specific workarounds.

Opening the Knowledge Base to Developers & Agents

Data trapped in a silo is useless. Once we reached 100,000 solutions, we focused on making this knowledge immediately accessible to both human developers and autonomous AI agents.

1. Public JSON API v1 (Zero-Auth, Open CORS)

Developers can query the entire consensus database programmatically:

curl "https://saidithere.com/api/v1/problems/search?q=docker"

Response:

{ "total": 80, "results": [ { "slug": "fixing-slow-file-sharing-performance-in-docker-on-macos", "title": "Why is file sharing so slow in Docker on macOS?", "evidence_count": 94, "workarounds_count": 12, "url": "https://saidithere.com/problems/fixing-slow-file-sharing-performance-in-docker-on-macos" } ] }

2. Zero-Install Remote MCP for Cursor & Claude

Instead of forcing developers to context-switch to a search engine when an IDE error occurs, we deployed a hosted Remote Model Context Protocol (MCP) server over Server-Sent Events (SSE).

In Cursor or Claude Desktop, you just add one URL:

{ "mcpServers": { "saidithere": { "url": "https://saidithere.com/api/mcp" } } }

Now, when you ask your coding assistant: "I'm getting this weird Postgres connection pooling error in Next.js 15, has anyone found a workaround?", your AI automatically queries Said It Here and cites verified community consensus.

3. AI Search Optimization with llms.txt

To ensure answer engines like Perplexity, ChatGPT Search, and Claude can discover and cite these solutions cleanly, we implemented the open /llms.txt and /llms-full.txt specification.

It provides a concise markdown index of all 6,000+ problems, 40 software entity hubs (Docker, AWS, Git, Stripe, Kubernetes), and comparison matchups, with zero HTML bloat.

What 100,000 Workarounds Taught Us

Building this system taught us a few enduring lessons about developer data:

  1. The best technical documentation is conversational. Official docs tell you how a system was intended to work; forum discussions tell you how it actually fails in production.
  2. Never fight the WAF head-on. If a scraper is getting 403 errors, rotating proxies is often the wrong fix. Look at your query endpoint: switching from /new.json to /top.json?t=day was 100x more effective than buying expensive proxy IPs.
  3. Vector embeddings are the ultimate deduplication tool. String matching or keyword search will fail when developers describe the same bug using completely different jargon. Cosine similarity over vector space solves this effortlessly.

Try It Out

  • Explore the 100k database: saidithere.com
  • Check out the API documentation: /developers
  • Read our machine-readable index: /llms.txt
  • Connect to Remote MCP in Cursor: https://saidithere.com/api/mcp

If you're building automated crawlers, LLM extraction pipelines, or MCP servers in 2026, I’d love to hear how you're handling anti-bot systems and token economics in the comments!