One of the most complex architectural conflicts in modern distributed backend systems is the collision between immutable data structures (Kafka event logs, event sourcing, append-only ledgers, blockchain) and Article 17 of the GDPR (The Right to Erasure).

In a traditional relational database, fulfilling a Data Subject Access Request (DSAR) for erasure is conceptually trivial:

DELETE FROM users WHERE id = $1;

In an event-sourced architecture, deleting an event from an immutable segment breaks consumer offset tracking, invalidates cryptographic checksums, and violates the fundamental premise of append-only auditability.

Here is how modern production systems handle the erasure requirement without corrupting event streams.

The Naive Approach: Compaction and Tombstoning

Many teams start by relying on Kafka topic log compaction: emitting a tombstone record (a message with the user key and a null payload).

The failure modes of log compaction for GDPR compliance:

  1. Unpredictable deletion timelines: Log compaction is asynchronous. Until segment cleaning actually triggers and active segment files roll over, raw user PII remains plainly readable in older log segments on disk for days or weeks.
  2. Backup persistence: Backups of event logs to cold storage (e.g. S3 tiered storage or Iceberg tables) freeze the historical PII indefinitely.
  3. Downstream read models: Emitting a tombstone requires every downstream microservice and consumer projection to implement custom logic to purge local caches and materialized views. If one microservice fails to handle the tombstone, non-compliance persists.

The Robust Pattern: Cryptographic Erasure (Crypto-Shredding)

Instead of mutating the immutable event stream, Crypto-Shredding decouples the event payload from the cleartext identity using per-user encryption keys.

Architecture Walkthrough

  1. Per-Entity Key Generation: When a user registers or data is ingested, a dedicated cryptographic encryption key (e.g. AES-256-GCM) is generated for that specific data subject and stored in a secure Key Management Service (KMS) or encrypted Key Store: 
Key Store: user_uuid_1048 -> aes_key_77b...
  1. Payload-Level Encryption: When an event is written to Kafka or the immutable event store, all personal identifying attributes (PII) are encrypted with the user's specific key before serialization: 
{ "event_id": "evt_99182", "event_type": "ORDER_PLACED", "timestamp": "2026-11-06T08:15:00Z", "user_id": "user_uuid_1048", "encrypted_payload": "enc_GCM_8f91a27b3d...", "non_pii_metadata": { "currency": "EUR", "total_cents": 4900, "sku_count": 2 } }
  1. Executing Article 17 Erasure: When an authenticated erasure request is validated:
  2. You do not touch Kafka.
  3. You do not re-write petabytes of historical S3 log segments.
  4. You perform an atomic, audited deletion of user_uuid_1048's encryption key from the Key Store. 
DELETE FROM key_registry WHERE user_id = 'user_uuid_1048';

Once the key is destroyed, the ciphertext in all historical event logs, cold backups, analytical warehouse replicas, and streaming replay buffers is mathematically unrecoverable. European Data Protection Authorities (including the CNIL and UK ICO) recognize cryptographic shredding as compliant erasure, provided key destruction is audited and the encryption algorithm meets modern cryptographic standards.

Key Takeaways for DSAR Engineering

  1. Keep Metadata Unencrypted: Aggregate metrics, financial audit amounts, and event IDs must remain unencrypted so analytics and accounting ledgers continue to balance without decrypting PII.
  2. Key Store Availability: Because every event read requires key retrieval, maintain a low-latency, read-through in-memory cache (with strict TTLs) for active keys.
  3. Automate Continuous Verification: A deletion request is only complete when verified across all peripheral caches, backups, and error logs. Building automated discovery pipelines that scan for orphaned PII prevents statutory compliance penalties.

How does your team handle data deletion across event streams and streaming consumers? Let's discuss in the comments below.

Run an instant conversion, SEO & leak audit on your website free: https://leaksift.pages.dev/?src=devto