Most retrieval-augmented generation (RAG) demos over the news stop at search results: a title, a URL and a two-line snippet. That is fine for "what happened?" and useless for "what exactly did they announce, and who disagreed?". The answer is in paragraph nine, and the snippet never had it.

This guide builds a small pipeline that works on the full text of recent articles:

  1. Find recent articles on a topic.
  2. Fetch each article's readable body, without menus, ads and comment sections.
  3. Clean and chunk it by paragraph, keeping the source and date with every chunk.
  4. Retrieve the best chunks for a question.
  5. Build a numbered context block a model can cite.

Everything runs in one Python file with requests only. You can swap the retrieval step for embeddings later without touching the rest.

Setup 

pip install requests export NESTDADDY_API_KEY="your_key"

The examples use the NestDaddy API: news/search for recent articles and extract for the full text of any URL. A free 7-day key (5,000 calls, no card) is enough to follow along.

Step 1: find recent articles 

import math import os import re from collections import Counter import requests API = "https://nestdaddy.com/api/v1" KEY = os.environ["NESTDADDY_API_KEY"] HEADERS = {"X-API-Key": KEY} def find_articles(query, days=3, country=None, limit=5): """Recent news articles about a topic: title, url, source, date.""" params = {"q": query, "days": days, "limit": limit} if country: params["country"] = country r = requests.get(f"{API}/news/search", headers=HEADERS, params=params, timeout=30) r.raise_for_status() return r.json().get("results", [])

Keep days small for questions about current events. A wider window brings in older articles that confidently contradict newer ones.

Step 2: fetch the full text 

def full_text(url): """The readable body of one article, without menus, ads or comments.""" r = requests.get(f"{API}/extract", headers=HEADERS, params={"url": url}, timeout=30) r.raise_for_status() data = r.json() return (data.get("text") or "") if data.get("success") else ""

Some pages will not give you much: paywalls, consent walls and pages that build their content with JavaScript. Step 4 skips anything shorter than 500 characters, which catches most of them.

Step 3: clean and chunk by paragraph

Fixed-size chunks cut sentences in half and split a quote from the person who said it. Chunking by paragraph keeps meaning together. Each new chunk also starts with the last paragraph of the previous one, so a fact that spans two paragraphs is never lost.

BOILERPLATE = re.compile( r"sign up for|newsletter|follow us on|subscribe to|read more:|all rights reserved|" r"click here|share this article", re.I) def chunk(text, size=800): """Pack whole paragraphs into ~size-character chunks. Each new chunk starts with the last paragraph of the previous one, so a fact that spans two paragraphs is never cut in half. """ paras = [p.strip() for p in text.split("\n") if len(p.strip()) > 40 and not BOILERPLATE.search(p)] chunks, cur = [], [] for p in paras: if cur and sum(len(x) for x in cur) + len(p) > size: chunks.append("\n".join(cur)) cur = cur[-1:] cur.append(p) if cur: chunks.append("\n".join(cur)) return chunks

The BOILERPLATE filter is small on purpose. Publishers drop newsletter prompts and social links in the middle of articles, and those lines are noise to retrieval. Add patterns as you meet them.

Step 4: build the corpus with metadata

Every chunk carries its title, URL, source and publish date. That metadata is what lets the model cite, and lets you filter by date or source later.

def build_corpus(query, **kw): corpus = [] for a in find_articles(query, **kw): text = full_text(a["url"]) if len(text) < 500: # paywall stub or failed extraction: skip it continue for i, c in enumerate(chunk(text)): corpus.append({ "text": c, "title": a["title"], "url": a["url"], "source": a.get("source_domain"), "published": a.get("published_timestamp"), "chunk": i, }) return corpus

Step 5: retrieve the best chunks

To keep the example dependency-free, here is BM25, the keyword ranking behind most search engines. For a few dozen chunks from fresh news it works well, because questions about events usually share exact words with the answer (names, places, products).

WORD = re.compile(r"[a-z0-9]+") def bm25_top(corpus, question, k=5, k1=1.5, b=0.75): """Tiny BM25 so the example has no dependencies; swap in embeddings later.""" docs = [Counter(WORD.findall(d["text"].lower())) for d in corpus] avg = sum(sum(d.values()) for d in docs) / max(len(docs), 1) terms = set(WORD.findall(question.lower())) df = {t: sum(1 for d in docs if t in d) for t in terms} n = len(docs) scores = [] for i, d in enumerate(docs): length = sum(d.values()) s = 0.0 for t in terms: if t not in d: continue idf = math.log(1 + (n - df[t] + 0.5) / (df[t] + 0.5)) s += idf * d[t] * (k1 + 1) / (d[t] + k1 * (1 - b + b * length / avg)) scores.append((s, i)) return [corpus[i] for s, i in sorted(scores, reverse=True)[:k] if s > 0]

When your corpus grows or questions get vaguer, add embeddings and combine both scores (hybrid search). Keyword matching stays useful for names and numbers, which embeddings often blur.

Step 6: a context block the model can cite 

def context_block(chunks): """Numbered sources the model can cite as [1], [2]...""" lines = [] for n, c in enumerate(chunks, 1): lines.append(f"[{n}] {c['title']} ({c['source']}, {c['published']})\n{c['url']}\n{c['text']}") return "\n\n".join(lines) if __name__ == "__main__": corpus = build_corpus("AI", days=3, country="GB", limit=4) print(len(corpus), "chunks from full articles") top = bm25_top(corpus, "What are employers being told to prepare for?") print(context_block(top)[:1500])

Put that block in your prompt under an instruction such as:

Answer using only the sources below. Cite them as [1], [2]. Mention the date of each fact. If the sources do not answer the question, say so.

On a test run, four full articles became about 50 chunks, and the top result for the question above was the paragraph that actually quoted the minister's guidance, not the headline.

Things that matter in production

  • Deduplicate before chunking. The same story appears on several sites; keep one copy per URL and consider near-duplicate titles.
  • Cache the full text. Articles rarely change after the first hours; store the extracted text by URL and skip the fetch next time.
  • Keep dates in the prompt. Models are bad at knowing today's date. "As of 12 November" in the context prevents stale answers.
  • Respect copyright. Use full text to ground answers and quote short passages with a link, not to republish articles.
  • Check what you skipped. Log URLs that returned under 500 characters. If one source always fails, drop it from your query.

Use it with an agent instead

If your RAG pipeline is really an agent, you can skip most of this code: the NestDaddy MCP server exposes news search, page reading and research papers as tools, and news tools can return full article text directly. Setup for Claude, Cursor and local models: Claude, Cursor, Ollama.

One All-in-One key covers news search, page extraction, web search and the MCP server, from $9.99 a month for 10,000 calls after the free 7-day trial: get a key.

Originally published on the NestDaddy blog. NestDaddy is an independent search engine with its own index: nestdaddy.com.