Index cost is the constraint that changes this design. For a financial research dashboard that also aggregates game listings from several feeds, I would keep sensitive fields out of the shared semantic index, attach an explicit access scope to every chunk, and retrieve in two stages: cheap metadata filtering first, embeddings second.
Short answer: build a policy-aware retrieval layer around a small, partitioned index, then prove its behavior with adversarial privacy tests before tuning recall. A giant all-purpose vector store is easy to demo and expensive to govern.
The result is less glamorous than a single search call. It is easier to explain to a reviewer, though, and the explanation is part of the product.
Start with a data contract, not an embedding model
The first implementation mistake is treating a document as the security boundary. A quarterly filing, a broker note, and a game listing can share a text format while having completely different readers. The unit that needs a policy is the chunk, and the policy must survive ingestion, indexing, retrieval, and citation rendering.
I use a narrow record contract. It keeps the raw text in a controlled store and puts only the searchable representation plus labels in the index.
from dataclasses import dataclass from typing import FrozenSet @dataclass(frozen=True) class Chunk: chunk_id: str source_id: str text: str tenant: str scopes: FrozenSet[str] contains_personal_data: bool published_at: str def can_read(chunk: Chunk, user_scopes: set[str], tenant: str) -> bool: return ( chunk.tenant == tenant and chunk.scopes.issubset(user_scopes) and not chunk.contains_personal_data ) That last boolean is not a complete privacy system. It is a forcing function: the ingestion job must make a decision instead of silently indexing everything. In production, a separate redaction service can replace or enrich it, with an audit record linking the redacted version to the source object.
For the game-listing feed, tenant can identify a publisher account and scopes can distinguish public catalog data from unreleased launch dates. For the dashboard, the same fields can represent a fund, desk, or subscription tier. The shape stays stable while policy values change.
How should retrieval handle privacy controls, citations, and index cost?
A useful query path has four observable checkpoints: authenticate the caller, apply structured filters, run semantic search over the remaining candidates, and authorize every citation before it leaves the service. Logging those checkpoints gives an operator a reason when a result is absent.
Here is a deliberately boring interface for the middle two steps. It is plain Python so the policy can be tested without a hosted search dependency.
from collections.abc import Iterable def metadata_candidates( chunks: Iterable[Chunk], *, tenant: str, user_scopes: set[str], as_of: str, ) -> list[Chunk]: return [ chunk for chunk in chunks if chunk.tenant == tenant and chunk.published_at <= as_of and chunk.scopes.issubset(user_scopes) and not chunk.contains_personal_data ] def citation_payload(chunk: Chunk, user_scopes: set[str], tenant: str) -> dict[str, str]: if not can_read(chunk, user_scopes, tenant): raise PermissionError("citation is outside the caller scope") return {"source_id": chunk.source_id, "chunk_id": chunk.chunk_id, "text": chunk.text} The semantic ranker runs only on metadata_candidates. That ordering matters for privacy, but it also controls index cost at scale: a cheap filter can prevent an embedding search from touching unrelated tenants and old versions. Keep the filter dimensions low-cardinality and measurable. If every document gets a unique, high-cardinality label, the filter itself becomes a slow secondary index.
The retrieval-augmented generation paper describes combining a parametric generator with a non-parametric memory, which is a useful mental model here: retrieval is a separate memory system with its own freshness and access rules, not a prompt field that can be patched after generation. The model should receive only authorized passages. Do not ask it to hide passages it was already given.
A small evaluation harness catches the expensive mistakes
Before choosing chunk sizes or a database, make a fixture with three classes of queries: an analyst asking for public revenue data, a desk user asking for restricted research, and a cross-tenant query that should return nothing. Add game-listing queries with unreleased metadata to make the same boundary concrete outside finance.
I keep expected outcomes as data, then run them on every ingestion change.
def evaluate(cases: list[dict], chunks: list[Chunk]) -> dict[str, float]: allowed = denied = 0 for case in cases: found = metadata_candidates( chunks, tenant=case["tenant"], user_scopes=set(case["scopes"]), as_of=case["as_of"], ) got = {item.chunk_id for item in found} expected = set(case["expected_ids"]) if got == expected: allowed += 1 if not (got & set(case.get("forbidden_ids", []))): denied += 1 total = len(cases) or 1 return {"exact_scope_rate": allowed / total, "denial_rate": denied / total} Exact matching is intentionally strict for this fixture. A production harness can add recall, citation precision, latency percentiles, and token counts, but privacy leakage is a binary failure. I also test stale records: a query with an as_of date must not surface a later filing or a future game launch. In one fixture, the public analyst sees two annual-report chunks and the desk user sees those plus a restricted channel-check note; the cross-tenant request sees zero. That tiny matrix catches a surprising number of regressions because a developer can change a default filter, refresh a snapshot, or re-embed a document without touching the answer prompt. I keep the expected IDs in version control and require the same policy version in the test report as in the index metadata, so a passing score cannot be detached from the rules that produced it.
One practical trap is measuring only answer quality. A fluent answer assembled from a forbidden chunk is still a failed request. Track policy-denial counts, empty-result rates by scope, and the number of citations rejected at the final gate. Those metrics tell you whether a recall change actually altered exposure risk.
Where this architecture does not fit
The catch is operational complexity. Partitioned indexes and a second authorization check cost engineering time, and they can reduce recall when metadata is incomplete. This approach is not suitable when the corpus is entirely public, tiny, and disposable; a conventional full-text index may be clearer. Stick with a simpler pipeline until you have a real tenant boundary or a retention requirement.
It is also a poor fit for workloads that need millisecond, globally replicated search with no tolerance for a policy lookup. In that case, choose a search system with native document-level security and validate its enforcement independently. Your mileage may vary because provider-specific filter semantics and replication guarantees differ; the contract and tests should remain yours.
The notebook version should expose the same four checkpoints as the deployed service, with fixtures standing in for the source feeds. Package the policy function, candidate filter, ranker adapter, and citation gate as separate modules. That makes a code review about decisions rather than a wall of prompt strings.
For observability, emit a request ID, tenant, policy version, candidate count, ranked count, denied-citation count, and token estimate. Never log raw restricted text by default. Retain enough identifiers to replay a decision against a quarantined copy of the source.
Finally, set a budget in the evaluation harness. If a new chunking strategy doubles indexed tokens without improving scope-safe recall, reject it before it reaches the dashboard. The right retrieval design is the one whose privacy behavior is testable, whose citations are explainable, and whose index bill grows for a reason.
