Series: Designing an Enterprise RAG Platform from Scratch (Part 1 of 5)

1. How This Journey Started: Being Real About RAG

I’ve been working as a software developer for 3 years. Over the last year, like many engineers, I watched the generative AI wave take over tech.

Everywhere I looked, I saw tutorials titled "Build a Production RAG App in 5 Minutes!". But when you open them up, they are almost always the same thing:

  • A 25-line Python script.
  • An in-memory vector database (like ChromaDB or FAISS) running locally on a laptop.
  • A single hardcoded API key making synchronous calls.

As a developer working on real-world systems, I kept asking myself:

"Is this really how companies run RAG in production?"

The answer is no. In real enterprise environments:

  1. Companies build on cloud ecosystems like AWS, utilizing managed services, IAM security, and decoupled messaging.
  2. Backends are predominantly built on enterprise frameworks like Java / Spring Boot rather than transient notebook scripts.
  3. You can't just dump a 50MB architecture PDF or an Excel sheet into a web thread without crashing your API.
  4. You can't spend \$10,000 on cloud OCR APIs like Textract for everyday documents.
  5. You can't tolerate hallucinations—if an engineer asks about an architectural decision (ADR) or incident policy, the answer must be 100% grounded and verifiable.

I had an AWS account with around \$200 in promotional credits expiring in a few months. Instead of letting those credits go to waste on idle EC2 instances, I decided to challenge myself:

Can I design and build a universal, decoupled, enterprise-grade RAG architecture from scratch that is accurate, fail-safe, and truly production-ready?

That was the birth of DocLab.

2. What Does "Production-Ready" Actually Mean for RAG?

Before writing a single line of Terraform or Java, I defined the non-negotiables for what this platform had to achieve:

  • Universal Multi-Format Ingestion (Zero Cloud OCR Fees): Real companies don't just use clean .txt files. They have complex PDFs, Word .docx documents, multi-tab Excel .xlsx spreadsheets, and raw JSON service catalogs. The platform had to parse all of them 100% in-memory with \$0 extra AWS OCR fees.
  • Asynchronous Decoupling: Ingestion had to use the enterprise Claim-Check Pattern over SQS queues so that heavy document parsing never blocks user-facing HTTP threads.
  • Two-Stage Retrieval (Beyond Simple Cosine Similarity): Vector search alone is notorious for missing exact technical keys or confusing negations. The system needed hybrid search (BM25 + pgvector RRF) paired with a real-time Cross-Encoder on AWS SageMaker to achieve >99% retrieval confidence.
  • Fail-Closed Governance: If a user from Marketing queries an Engineering ADR, or if no authorized documents exist, the system must fail closed immediately—consuming zero Bedrock tokens and producing zero hallucinations.
  • Cost Conscious: The entire infrastructure had to run efficiently within my \$200 cloud credit budget without accidental billing surprises.

3. The 5-Pillar High-Level Architecture

Here is the high-level design I arrived at:

HLD of DocLab

The system is separated into 5 clear pillars:

  1. Client & Edge Access Layer: A Spring Boot 3.3.5 platform providing REST endpoints, Server-Sent Events (SSE) streaming, distributed Redis rate limiting, and session memory.
  2. Asynchronous Ingestion Plane: Built with Amazon S3 and Amazon SQS using the Claim-Check Pattern, ensuring file intake is decoupled from processing.
  3. 100% In-Memory Universal Parsers: Pluggable extractors (Apache PDFBox, Apache POI, Jackson) that transform PDF, DOCX, XLSX, and JSON into a normalized Canonical Intermediate Representation (IR) with \$0 Textract costs.
  4. Data Vault & Hybrid Storage: PostgreSQL 16 on Amazon RDS with pgvector (HNSW indexing) and native tsvector BM25 lexical search, protected by database-level RBAC pre-filtering.
  5. Two-Stage AI Serving Engine: Reciprocal Rank Fusion (RRF) recall paired with an AWS SageMaker real-time endpoint running BAAI/bge-reranker-base cross-encoder, synthesizing grounded responses via Amazon Bedrock (Nova Micro).

4. The Plan for This Series

Building this taught me more about real distributed systems, cloud economics, and AI engineering than any tutorial ever did. In this 5-part series, I am sharing the exact architectural decisions, math, and benchmarks:

  • Part 1 (This post): The Origin Story & High-Level Architecture.
  • Part 2: The Ingestion Factory: Decoupling uploads with the SQS Claim-Check Pattern & In-Memory Universal Parsers (PDF, Word, Excel, JSON).
  • Part 3: The Storage & Hybrid Vault: Dense pgvector + Sparse BM25 + Mathematical Reciprocal Rank Fusion (RRF).
  • Part 4: The AI Brain: Why Bi-Encoders Fail and How AWS SageMaker Cross-Encoder Reranking Achieves 0.997 Confidence.
  • Part 5: Enterprise Governance: Fail-Closed Guardrails, Zero-Trust Caching, and Real Empirical AWS Benchmarks.

If you’ve tried moving RAG from a local prototype to the cloud, what was the biggest roadblock you encountered? Let’s discuss in the comments!