You have a working chat interface in an afternoon. Turning it into a product users pay for takes months. The gap isn't the model: it's conversation history, citations, cost guards, and the boring auth and billing layer nobody talks about.
What a Real Chatbot Product Has to Handle
Streaming Chat UI
Token-by-token rendering is the baseline. You need SSE or fetch streaming, Markdown rendering mid-stream with code blocks and syntax highlighting, a stop button, regenerate, edit-and-resend, and scroll-to-bottom that respects where the user scrolled. The Vercel AI SDK handles all of this. Don't roll streaming yourself.
Conversation History
Not a single chat: a list of chats with a sidebar, new-chat button, rename and delete, folders or categories, search across conversations, pinning, and optional public sharing. Storage is a conversations table with messages linked by conversation_id, indexed by user and updated-at.
Multi-Model Support
Production chatbots route across OpenAI GPT-5 for reasoning, Anthropic Claude for long context, Gemini for multimodal, self-hosted Llama or Mistral for privacy, and fine-tunes for specific tasks. The UI should expose model selection or route automatically.
RAG (Retrieval-Augmented Generation)
When the bot answers questions about your data:
- Document ingestion for PDFs, web pages, uploads
- Semantic chunking beats fixed-size in 2026
- Embeddings via OpenAI text-embedding-3-large or Voyage AI
- Vector storage in pgvector, Pinecone, Qdrant, or Weaviate
- Hybrid retrieval (keyword + semantic) with reranking
- Citation rendering showing which sources the answer uses
- Clickable source previews
Without RAG your bot is generic ChatGPT. With RAG it knows the customer's data.
Tool Calling
Real agents take actions:
- Function definitions in JSON schema
- Tool call rendering in chat ("Looking up order #1234...")
- Approval flows for destructive actions
- Multi-tool sequences where the model picks the right order
- Error handling when tools fail
The Vercel AI SDK and OpenAI's tool calling API handle mechanics. Product work is choosing tools and UX patterns.
Citations and Source Attribution
The trust layer:
- Inline citations
[1] [2]clickable to sources - Source list at bottom of each answer
- Source preview on hover showing content snippet
- Confidence indicators when unsure
- Refusal handling when data doesn't support an answer
Without clickable citations users can't verify anything important.
Feedback Capture
For improving the product:
- Thumbs up/down per response
- Optional comment explaining the rating
- Inline issue tags ("inaccurate", "irrelevant", "harmful")
- Aggregation dashboard showing top issues
- Per-prompt feedback rollups to identify failing prompt patterns
Rate Limiting and Cost Control
Critical and often skipped:
- Per-user rate limits (N messages per hour)
- Per-account quotas (M messages per month for free tier)
- Per-model cost tracking with daily/monthly caps
- Hard stop at cost threshold to prevent runaway bills
- Token usage dashboard for admins
A single bug or abusive user can run up a five-figure bill overnight. Build cost controls before launch.
Deployment and Auth
The boring critical path:
- Auth via Clerk, Auth.js, or Supabase Auth
- User accounts with conversation history per user
- Subscription tiers with rate limits per tier
- Onboarding flow so first-time users understand the bot
- Marketing site that drives signups
This is where most "ChatGPT for X" products fail: working chatbot, no auth, no billing, no real product around it.
Tech Stack Decisions
Streaming framework: Vercel AI SDK. Standard in 2026. Handles streaming, tool calls, structured outputs, multi-model.
Model APIs: OpenAI SDK + Anthropic SDK, or AI SDK abstractions for one interface.
Vector database: pgvector if Postgres-based (cheap, easy). Pinecone for managed. Qdrant for self-hosted.
Embeddings: OpenAI text-embedding-3-large or Voyage AI voyage-3.
Real-time: Built into streaming layer; no separate WebSocket needed.
File parsing: Unstructured.io or LlamaParse for PDF, DOCX.
Background jobs: Inngest for async document ingestion.
Database: Postgres + Drizzle. Conversations + messages tables.
Build Path 1: From Scratch (6 to 10 Weeks)
Week 1-2: Foundation. Next.js, auth, basic chat UI with Vercel AI SDK, conversation history.
Week 3-4: Multi-model + tool calling. Wire OpenAI + Anthropic. Build a few tools.
Week 5-6: RAG if needed. Document ingestion, embedding, vector storage, retrieval, citations.
Week 7-8: Cost controls and rate limiting. Per-user limits, per-account quotas, cost dashboard.
Week 9-10: Polish and deployment. Auth tiers, marketing site, onboarding, Stripe wiring.
Most teams take 10 to 14 weeks. RAG alone is routinely 3-4 weeks if done well.
Build Path 2: Fork a Starter, Buy the Interface (3 to 5 Weeks)
Split the problem. Fork Vercel AI Chatbot or LibreChat for auth, conversation persistence, model routing, streaming plumbing. Both are open source and functional day one. What they don't give you is an interface anyone would pay for: that's where remaining weeks go.
The work that remains:
- Wire your model provider through Vercel AI SDK
- Build retrieval if you need RAG, typically pgvector
- Add rate limiting and per-user cost caps
- Connect Stripe if charging
Common Pitfalls
Rolling streaming yourself. Use Vercel AI SDK. Edge cases (reconnection, partial messages, stream cancellation) are solved.
Single-model lock-in. OpenAI pricing changes. Anthropic launches better models. Without an abstraction layer, switching is a refactor. Use provider-agnostic interface from day one.
No cost controls. A single bug or abusive user spends $10k overnight. Build per-user, per-account, and global daily cost caps before launch.
RAG without reranking. Top-K retrieval alone misses the right chunk often. Add reranking (Cohere Rerank, OpenAI rerank) for production quality.
Citations as decoration. If citations aren't clickable to the source, users can't verify. Build source-preview UX from day one.
Storing prompts as code constants. Prompts change weekly. Pull them into config or database so non-developers can iterate.
No evals. You change the prompt and have no way to know if quality went up or down. Build a basic eval suite (10-50 test cases) before the first prompt change.
Underbuilding the conversation sidebar. Without history every chat starts cold. Users hate it. Build the sidebar in v1.
This is a shorter version of a guide I wrote at Frontkit Studio. The full post has the complete walkthrough.
