If you are building AI agents, RAG (Retrieval-Augmented Generation) pipelines, or feeding web content into vector databases (Pinecone, Chroma, Qdrant), you already know the big problem with web scraping: Raw HTML is a token killer.

A typical news article or documentation page weighs 500 KB to 1 MB. Most of that weight is bloated navigation menus, ads, tracking scripts, and cookie banners. Ingesting raw HTML directly into an LLM context window burns tens of thousands of tokens and confuses the model.

Here is a clean, lightweight way to parse any URL directly into clean, noise-free GitHub Flavored Markdown (GFM) in Node.js while cutting token consumption by ~90%.

The Problem with Sending Raw Web Content to LLMs

Consider fetching a long Wikipedia or news article:

// Raw HTML fetch const html = await (await fetch("[https://en.wikipedia.org/wiki/Artificial_intelligence](https://en.wikipedia.org/wiki/Artificial_intelligence)")).text(); console.log(html.length); // ~2,200,000 characters (~570,000 tokens!)

Sending 570k tokens into a model like GPT-4o or Claude 3.5 Sonnet costs real money and easily blows past standard context windows.

The Solution: Extraction via Mozilla Readability & Turndown
To solve this for AI pipelines, I built MarkRAG API: a microservice in Node.js that fetches the page with realistic browser header emulation, extracts the core content using Mozilla's Readability engine (the same engine behind Firefox Reader View), and converts it into clean Markdown with tables and code blocks preserved.

It also calculates exact token metrics so you know how much context you saved:

import express from "express"; const app = express(); app.use(express.json()); app.post("/api/rag/ingest", async (req, res) => { const { url } = req.body; try { const response = await fetch("[https://markrag-web-to-clean-markdown-and-rag-parser.p.rapidapi.com/v1/extract](https://markrag-web-to-clean-markdown-and-rag-parser.p.rapidapi.com/v1/extract)", { method: "POST", headers: { "Content-Type": "application/json", "X-RapidAPI-Key": process.env.RAPIDAPI_KEY, "X-RapidAPI-Host": "markrag-web-to-clean-markdown-and-rag-parser.p.rapidapi.com" }, body: JSON.stringify({ url: url, options: { strip_links: true, // Flattens links to plain text to save extra tokens cache: true // Sub-2ms response for recurring URLs } }) }); const { data } = await response.json(); console.log(`Title: ${data.title}`); console.log(`Tokens saved: ${data.metadata.tokensSaved}`); console.log(`Context reduction: ${data.metadata.savingsPercentage}%`); // Ingest clean data.markdown directly into your vector store or LLM prompt return res.status(200).json({ title: data.title, markdown: data.markdown, tokens: data.metadata.estimatedTokens }); } catch (error) { console.error("Extraction error:", error); return res.status(500).json({ error: "Failed to extract content" }); } });

Benchmark: Raw HTML vs Clean Markdown
Testing the API against the Wikipedia Artificial Intelligence article yielded these real numbers:

Raw HTML Size: ~2.2 MB (~572,000 tokens).

Clean Markdown Size: ~201 KB (~50,300 tokens).

Tokens Saved: 521,693 tokens.

Savings Percentage: 91.2%.

It strips all sidebar clutter, footers, and redundant scripts while keeping markdown headings (#, ##), bullet points, and code blocks intact.

Try it out for your AI agents
I published the microservice on RapidAPI with a free tier of 100 requests/month for developers and side projects:

👉 MarkRAG API on RapidAPI

rapidapi.com



If you are building LangChain, LlamaIndex, or custom RAG workflows, test it out with your target URLs and let me know how it handles them!