How to Build Durable Chat Memory for RAG Using ScyllaDB and LangChain

How to replace LangChain’s in-memory chat history with ScyllaDB — so your RAG chatbot retains context across restarts and scales across replicas

This post demonstrates how to integrate ScyllaDB Vector Search into your LangChain project for RAG use cases, as well as how to use ScyllaDB as a durable conversation memory within LangChain.

Background

Large language models are trained on a fixed snapshot of the world. RAG patches that gap by retrieving relevant documents at query time and injecting them into the prompt. The pipeline has two phases:

Index: load documents, split them into chunks, embed each chunk, store embeddings in a vector store.
Retrieve & Generate: enrich and embed the user’s question, find the nearest vectors (ANN search), pass the matching chunks as context to the LLM.

But RAG alone is not enough. You still need to keep track of all inputs provided by the user.

Persistent Chat Memory

LLMs are stateless. Every call starts with a blank slate unless you replay the conversation history. LangChain’s BaseChatMessageHistory abstraction lets you plug in any backend as the storage layer for that history. The default in-memory implementation vanishes on process exit. A database-backed implementation survives restarts, scales across replicas, and lets you inspect or audit conversations later.

That’s where ScyllaDB comes in.

ScyllaDB + LangChain

ScyllaDB is a NoSQL database optimized for high-throughput, low-latency workloads. Combined with LangChain, you can build reliable and always-on AI applications:

High availability: data is automatically replicated across nodes, so there is no single point of failure. The cluster continues serving reads and writes even if a node goes down.
Predictable P99 latency: ANN queries return results fast enough that retrieval doesn’t dominate your chain’s total latency.
Horizontal scalability: add nodes to the cluster to increase throughput without schema changes or downtime.
Built-in vector search: a native vector<float, N> type and HNSW index are created automatically in ScyllaDB.

In this example, embeddings are generated locally with sentence-transformers (all-MiniLM-L6-v2, 384 dimensions). For production use, you can swap in OpenAI embeddings, Cohere, or any other provider supported by LangChain.

In each conversation turn, the chatbot is reading data from ScyllaDB and then writing back into it using LangChain.

Setup Example

With the release of ScyllaDB 2026.2, you can now integrate LangChain with ScyllaDB seamlessly by reusing the existing Cassandra connector.

Install Dependencies

pip install langchain langchain-community \ sentence-transformers langchain-groq \ langchain-text-splitters cassio \ scylla-driver python-dotenv

Environment Variables

Copy demo/.env.example to demo/.env and fill in your credentials:

# ScyllaDB Cloud SCYLLADB_CONTACT_POINTS=node-0.your-cluster.cloud.scylladb.com SCYLLADB_DATACENTER=AWS_US_EAST_1 SCYLLADB_USERNAME=scylla SCYLLADB_PASSWORD=your-password-here SCYLLADB_KEYSPACE=demo # Groq (LLM) GROQ_API_KEY=gsk_...

Retrieve the contact points, datacenter name, username, and password from the Connect tab of your cluster in ScyllaDB Cloud.

Connect to ScyllaDB Cloud

from cassandra.cluster import Cluster from cassandra.auth import PlainTextAuthProvider from cassandra.policies import DCAwareRoundRobinPolicy import cassio cluster = Cluster( contact_points=["node-0.your-cluster.cloud.scylladb.com"], auth_provider=PlainTextAuthProvider( "scylla", "your-password" ), load_balancing_policy=DCAwareRoundRobinPolicy( local_dc="AWS_US_EAST_1" ), ) session = cluster.connect() cassio.init(session=session, keyspace="demo")

The cassio.init() call registers the session and keyspace globally so all downstream integrations (the vector store and the chat history) pick it up without further configuration.

Although cassio was originally built as a Cassandra integration, it is session-agnostic. It works with whatever driver session you hand it, so passing a session created by scylla-driver works just as well.

Integrating ScyllaDB with LangChain Vector Store

As an example, consider a chatbot that ingests articles, answers questions using retrieved context, and preserves conversation history in the database.

from langchain_community.document_loaders import WebBaseLoader from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_community.embeddings import HuggingFaceEmbeddings from langchain_community.vectorstores import Cassandra loader = WebBaseLoader([ "https://docs.scylladb.com/stable/get-started/scylladb-basics.html", "https://docs.scylladb.com/stable/get-started/data-modeling/query-design.html", ]) docs = loader.load() chunks = RecursiveCharacterTextSplitter( chunk_size=500, chunk_overlap=50, ).split_documents(docs) embeddings = HuggingFaceEmbeddings( model_name="all-MiniLM-L6-v2" ) vectorstore = Cassandra( embedding=embeddings, table_name="rag_docs" ) vectorstore.add_documents(chunks) retriever = vectorstore.as_retriever( search_kwargs={"k": 4} )

WebBaseLoader fetches and parses each URL into a Document.

RecursiveCharacterTextSplitter then breaks each document into 500-token chunks with a 50-token overlap so sentences are not severed at boundaries.

Cassandra(table_name="rag_docs") creates the table on first use, including a vector<float, 384> column (matching all-MiniLM-L6-v2’s output dimensions) and an HNSW index. Subsequent runs reuse the existing table; you only pay the embedding cost once unless you call add_documents again.

Persistent Chat Memory Implementation

from langchain_community.chat_message_histories import CassandraChatMessageHistory def get_chat_history(session_id: str) -> CassandraChatMessageHistory: return CassandraChatMessageHistory( session_id=session_id, table_name="chat_history", )

CassandraChatMessageHistory stores each message as a row keyed on session_id. Because rows are written to ScyllaDB, the history survives process crashes, server restarts, and horizontal scaling. Any replica that opens the same session_id sees the same history.

Change session_id to start a fresh conversation. Keep it the same to resume where you left off — that is the entire persistence mechanism.

Putting It Together

from langchain_groq import ChatGroq from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder from langchain_core.output_parsers import StrOutputParser from langchain_core.runnables import RunnablePassthrough from langchain_core.runnables.history import RunnableWithMessageHistory llm = ChatGroq(model="llama-3.3-70b-versatile") ablePassthrough from langchain_core.runnables.history import RunnableWithMessageHistory llm = ChatGroq(model="llama-3.3-70b-versatile") prompt = ChatPromptTemplate.from_messages([ ("system", "You are a helpful assistant. Answer the user's question using the " "retrieved context below.\n\nContext:\n{context}"), MessagesPlaceholder(variable_name="chat_history"), ("human", "{question}"), ]) chain = ( RunnablePassthrough.assign(context=lambda x: "\n\n".join( d.page_content for d in retriever.invoke(x["question"]) )) | prompt | llm | StrOutputParser() ) chain_with_history = RunnableWithMessageHistory( chain, get_chat_history, input_messages_key="question", history_messages_key="chat_history", ) config = {"configurable": {"session_id": "user-abc-session-1"}} answer1 = chain_with_history.invoke( {"question": "What is the difference between a partition key and a clustering key in ScyllaDB?"}, config=config, ) print(answer1) answer2 = chain_with_history.invoke( {"question": "How does using both keys affect the sort order of the data within a partition?"}, config=config, ) print(answer2) # References the prior turn via memory

RunnableWithMessageHistorywraps the chain and automatically loads prior turns from get_chat_historybefore each call, then appends the new turn after. Both storage operations hit ScyllaDB.

The second question drills deeper into clustering columns introduced in the first answer. Without persistent memory, the LLM would have no context for what was already explained; with it, the conversation flows naturally across turns.

ScyllaDB Schema

rag_docs, the vector store:

CREATE TABLE demo.rag_docs ( row_id text PRIMARY KEY, attributes_blob text, body_blob text, metadata_s map<text, text>, vector vector<float, 384>, ); CREATE CUSTOM INDEX idx_vector_rag_docs ON demo.rag_docs (vector) USING 'vector_index'; CREATE INDEX eidx_metadata_s_rag_docs ON demo.rag_docs (ENTRIES(metadata_s));

chat_history, the message store:

CREATE TABLE demo.chat_history ( partition_id text, message_id timeuuid, body_blob text, PRIMARY KEY (partition_id, message_id) ) WITH CLUSTERING ORDER BY (message_id DESC);

Next Steps

The full demo code is available on GitHub. You can run it once to ingest and embed the articles, then run it again with the same SESSION_ID to verify that prior conversation turns are loaded from the database.

Resources: