Why deletes reach bronze but never reach silver when every hop runs in one Lakeflow Spark Declarative Pipeline, how I found the cause, and a small change that makes them flow all the way to gold.
A colleague sent me a message on Slack.
They had a multi-hop pipeline: landing, bronze, silver. Every hop used AUTO CDC, and every table had Change Data Feed (CDF) turned on.
Updates worked everywhere. Deletes went from landing to bronze. The bronze change feed even showed the delete.
But silver never deleted the row.
They also found a community thread with the same problem. A Databricks employee answered that it is supported, and listed the usual checks. The author checked all of them on a recent runtime, and it still failed. The thread ends with "it seems to work for single-hop only".
I wanted to understand why, so I built the smallest pipeline that shows the problem.
This guide walks you through what I found. We build the pipeline, we watch the delete disappear, we find where it goes, and then I show you the workaround that works for me.
I may be wrong about some internal details. If you know more, please tell me in the comments.
The pipeline
Everything in this guide uses one pipeline with four tables:
landing -> bronze -> silver -> gold \-> silver_naive landing is a plain Delta table. I only append rows to it. Each row says what happened to a key:
| column | meaning |
|---|---|
id | the key |
name | the data |
op | UPSERT or DELETE |
seq | the order of events for one key |
bronze reads landing with AUTO CDC:
from pyspark import pipelines as dp from pyspark.sql.functions import col, expr @dp.temporary_view(name="landing_feed") def landing_feed(): return spark.readStream.table("landing") dp.create_streaming_table("bronze", table_properties={"delta.enableChangeDataFeed": "true"}) dp.create_auto_cdc_flow( target="bronze", source="landing_feed", keys=["id"], sequence_by=col("seq"), apply_as_deletes=expr("op = 'DELETE'"), except_column_list=["op"], stored_as_scd_type=1, ) silver_naive reads the bronze change feed with the pattern you find in the docs and in most answers online. It drops update_preimage rows, and it deletes when _change_type = 'delete':
@dp.temporary_view(name="bronze_cdf_naive") def bronze_cdf_naive(): return ( spark.readStream.option("readChangeFeed", "true") .table("bronze") .filter("_change_type != 'update_preimage'") ) dp.create_streaming_table("silver_naive") dp.create_auto_cdc_flow( target="silver_naive", source="bronze_cdf_naive", keys=["id"], sequence_by=col("_commit_version"), apply_as_deletes=expr("_change_type = 'delete'"), except_column_list=["_change_type", "_commit_version", "_commit_timestamp"], stored_as_scd_type=1, ) silver and gold use the workaround. We get to it later.
Watching the delete disappear
I ran three steps. After each step I ran the pipeline and listed the rows in every table.
| step | bronze | silver_naive |
|---|---|---|
| insert alice, bob, carol | 1, 2, 3 | 1, 2, 3 |
| delete bob, update carol | 1, 3 v2 | 1, 2, 3 v2 |
| insert bob again, delete alice | 2 v3, 3 v2 | 1, 2 v3, 3 v2 |
Bronze is correct. silver_naive got carol's update, but it kept bob after the delete, and it kept alice after the second delete.
This is the exact problem from the thread.
So far, nothing new. Now let us find where the delete goes.
Looking at the change feed from outside
First, I read the bronze change feed with SQL, from a notebook:
SELECT _change_type, _commit_version, id, name FROM table_changes('my_catalog.my_schema.bronze', 1) ORDER BY _commit_version The result for the delete of bob:
delete 4 2 bob This looks perfect. There is a delete row. So the problem must be in silver... right?
Looking at the change feed from inside
Then I added one more table to the pipeline. It does nothing smart. It only saves the bronze change feed, exactly as a flow inside the pipeline reads it:
@dp.table(name="bronze_cdf_seen_from_inside") def bronze_cdf_seen_from_inside(): return spark.readStream.option("readChangeFeed", "true").table("bronze") The result for the same delete of bob:
update_preimage 4 2 bob No delete row. Only one update_preimage row, and no update_postimage row after it.
The same commit, read from two places, gives two different answers:
| reader | what the delete looks like |
|---|---|
SQL table_changes | delete |
| a second pipeline | delete |
| a flow in the same pipeline | update_preimage, nothing else |
I also checked the silver MERGE in DESCRIBE HISTORY. For that commit, numSourceRows was 1. Only carol's update reached the MERGE. The delete was lost in the read, before AUTO CDC could do anything.
Why this happens
This is my understanding. I could not find it in the docs, so please read it as my best guess.
AUTO CDC does a soft delete. It does not remove the row from the table at once. It marks the row with a hidden tombstone, so it can handle events that arrive out of order. You can see this in DESCRIBE HISTORY bronze: the MERGE uses a column called __DeleteVersion, and later a separate DELETE cleans up old tombstones.
A reader outside the pipeline gets a clean view. Databricks turns the tombstone into a delete row.
A flow inside the same pipeline sees the change as an update of the row: the live row becomes a tombstone row. The pipeline hides the tombstone row, so the update_postimage disappears. Only the update_preimage is left.
And the usual pattern drops every update_preimage row.
So the delete is lost.
This also explains the comment in the thread. With a single hop, the reader is a different pipeline, so it gets a real delete row.
The workaround
The idea is simple: inside the same pipeline, an update_preimage with no update_postimage is a delete.
I do not want state or joins to find "a preimage with no postimage". AUTO CDC can do it for me if I give it the right order.
I change three things:
| usual pattern | workaround | |
|---|---|---|
| rows to keep | drop update_preimage | keep all rows |
| what is a delete | _change_type = 'delete' | _change_type IN ('delete', 'update_preimage') |
sequence_by | _commit_version | struct(_commit_version, 0 for delete, 1 for upsert) |
Here is silver:
IS_DELETE = "_change_type IN ('delete', 'update_preimage')" @dp.temporary_view(name="bronze_cdf_for_silver") def bronze_cdf_for_silver(): return spark.readStream.option("readChangeFeed", "true").table("bronze") # keep all rows dp.create_streaming_table("silver", table_properties={"delta.enableChangeDataFeed": "true"}) dp.create_auto_cdc_flow( target="silver", source="bronze_cdf_for_silver", keys=["id"], sequence_by=expr(f"struct(_commit_version, CASE WHEN {IS_DELETE} THEN 0 ELSE 1 END)"), apply_as_deletes=expr(IS_DELETE), except_column_list=["_change_type", "_commit_version", "_commit_timestamp"], stored_as_scd_type=1, ) Gold reads silver in the same way.
Why it works:
- A real update has an
update_preimageand anupdate_postimagein the same commit. The preimage gets order(version, 0)and the postimage gets(version, 1). AUTO CDC keeps the latest event for a key, so the upsert wins. Nothing is lost. - A soft delete has only the
update_preimage. It is the latest event for that key, so AUTO CDC deletes the row. - A real
deleterow is still a delete. If Databricks changes this behaviour in the future, the workaround keeps working.
The same three steps, now with every table:
| step | bronze | silver | gold | silver_naive |
|---|---|---|---|---|
| insert alice, bob, carol | 1, 2, 3 | 1, 2, 3 | 1, 2, 3 | 1, 2, 3 |
| delete bob, update carol | 1, 3 v2 | 1, 3 v2 | 1, 3 v2 | 1, 2, 3 v2 |
| insert bob again, delete alice | 2 v3, 3 v2 | 2 v3, 3 v2 | 2 v3, 3 v2 | 1, 2 v3, 3 v2 |
Silver and gold now match bronze, including a key that comes back after a delete.
Positives
- One pipeline. No extra pipelines, no state, no joins.
- Updates and deletes both flow through every hop.
- It also accepts real
deleterows, so it does not depend on the bug staying.
Negatives
- It depends on behaviour that I could not find in the docs. A runtime upgrade can change it.
- You must keep
update_preimagerows in the view. If someone adds the usual filter back, deletes break again, and nothing fails loudly. - Every upstream update becomes a removal plus an upsert downstream, so each hop does more work per change.
- I only tested SCD type 1. I did not test SCD type 2.
The other option: one pipeline per hop
If you prefer to stay on the documented pattern, put each hop in its own pipeline. Then silver reads bronze from outside, and it gets real delete rows.
Positives
- The documented pattern works as written.
- No special rules in your code.
Negatives
- More pipelines to deploy, schedule, and monitor.
- You need to order the runs, so bronze finishes before silver starts.
Two things the workaround does not cover
Tombstone purge. AUTO CDC keeps a tombstone for about 48 hours after a delete. Then a separate DELETE in the table history purges it. Setting pipelines.cdc.tombstoneGCThresholdInSeconds as a table property does not make the purge run sooner, so the purge needs a long-running test. Nobody has shown yet what the in-pipeline stream delivers at purge time.
Physical erasure. A delete removes the row from the table, not from the files. Older data files and change feed files keep the row until the retention period ends and VACUUM runs. If you build this for GDPR, design the file retention and VACUUM schedule as a separate step.
What I tested, and what I did not
I tested a small example on serverless, CURRENT channel, in November 2026: insert, update, delete, and a delete followed by an insert of the same key, through bronze, silver, and gold.
I did not test classic compute, large volumes, SCD type 2, or the tombstone purge.
Wrapping up
When a delete stops between two hops, do not trust only table_changes(...). It shows you what an outside reader sees, not what your flow sees.
Add a debug table that saves the change feed inside the pipeline, and look at it. Then check numSourceRows on the MERGE in DESCRIBE HISTORY.
If you see an update_preimage with no update_postimage, try the workaround above, and run your delete test again after every runtime upgrade.
If someone from Databricks reads this: is this the expected behaviour? I would love to know the official way to do it.
