Iceberg Position Deletes vs Equality Deletes in CDC Pipelines
Engine support, not storage efficiency, should decide your CDC delete format.

Iceberg position deletes, equality deletes, and deletion vectors distribute CDC costs across writing, reading, and maintenance. The right choice starts with engine compatibility, then writer capability and maintenance capacity. For teams embedding data synchronization into products, Prequel offers a managed integration approach, while teams operating their own pipelines must carry those trade-offs through the storage and query layers.
Why deleting a row from an immutable file is a harder problem than it looks
With copy-on-write, deleting one row from a large, immutable Parquet file on object storage can require rewriting one gigabyte of data. The writer cannot remove the row in place, so it rewrites the affected file without that row and commits the replacement. Reads remain clean because the replacement contains only live rows, with no deletion artifacts to apply.
Keeping reads simple this way leads to write amplification when small changes come often. A CDC commit may have inserts, updates, and deletes for single rows, so whole-file rewrites cost a lot for the amount of data changed. Recording deletions separately and letting readers apply them defers that work. Iceberg’s shared metadata scaffolding, including snapshot sequence numbers, delete manifests, and partition scoping, governs which deletions apply to which data. The physical form of those deletion artifacts determines how much work remains for the reader.
How the three Iceberg delete mechanisms work physically
A position delete file is a valid Iceberg Parquet file containing pairs of file_path and pos. Each pair identifies a deleted row by its data file and row position. These entries must be sorted by file path and position, and one delete file can reference many data files. Before writing an entry, the writer must locate the row through a prior read or an index lookup. The reader then skips that row ordinal as it scans, with no need to compare column values.
An equality delete records values to match rather than a physical address, such as a predicate that deletes any row where customer_id = 12345. This fits a CDC UPDATE represented as DELETE + INSERT because the writer can record the old row’s primary key and append its replacement without locating the old row. But readers must check the equality condition against rows in the data files where it may apply. Applying it therefore resembles a filter or join.
A v3 deletion vector uses a Roaring bitmap to store the positions of deleted rows in exactly one data file. The bitmap is encoded as a deletion-vector-v1 blob inside a Puffin sidecar file. A data file can have at most one deletion vector per snapshot. When more rows are deleted, the writer reads the existing bitmap, adds their positions, and writes a replacement for the next snapshot. The reader applies the bitmap in memory, skipping any row whose corresponding bit is set.
Where the cost actually lands across writing, reading, and maintenance
Position deletes place lookup work on the writer, while equality deletes defer matching work to readers and resolution work to maintenance. Deletion vectors keep positional read work small and prevent deletion artifacts from accumulating for an individual data file. As a rule, vectors are fastest to apply, position deletes come next, and equality deletes add the most work to reads. That ordering describes read-path performance.
The write-side difference matters especially in streaming CDC. A batch writer can scan data to find target positions before producing deletes, but a streaming writer needs either a lookup for each event or a pre-built index. Those lookups introduce latency and can reduce throughput under concurrency. Equality deletes avoid that cost at write time by using the key the event already has. Their low write cost therefore transfers an obligation to readers and maintenance.
That obligation compounds when commits keep arriving. A CDC feed committing once per minute and touching a handful of rows across ten data files in a hot partition can accumulate hundreds of delete artifacts within a day. Equality-delete readers must match against the accumulated delete data, so the same underlying files become more expensive to query over time. Position delete files can also accumulate, although their application remains positional. Vectors instead replace the existing bitmap, bounding the number of deletion artifacts a reader encounters per data file.
Why CDC pipelines default to equality deletes, and what that costs over time
Equality deletes are a natural landing format because a CDC DELETE event has a primary key but lacks an Iceberg file path and row ordinal. Producing a position delete from that event requires information the source database does not provide. Choosing equality deletes is therefore not evidence of careless engineering. It is a direct response to the gap between a logical database key and a physical lakehouse address.
The Siemens case shows how that reasonable write-side choice can become a downstream constraint. Real-time CDC data was written directly into Iceberg, with all deletes materialized as equality delete files. Snowflake external tables could not read those deletes, so the data was not correctly accessible through that path until the delete files were cleared. Siemens periodically ran a Spark job to resolve the problem before Snowflake access. That added latency, compute cost, and architectural complexity while reducing real-time visibility.
This is a predictable failure mode when ingestion and consumption are designed separately. The writer can write a deletion that the reader cannot apply. For product-embedded synchronization, where Prequel’s white-label integration approach offers infrastructure for scalable data export and import, that boundary also determines how much custom deletion logic product teams must manage. High-frequency syncing across multiple destinations makes the downstream read contract as important as ingestion throughput.
Engine compatibility is the hidden constraint that overrides format preference
“Supports Iceberg deletes” is not a sufficiently precise compatibility requirement. Reading position deletes, reading equality deletes, reading deletion vectors, and writing each format are separate capabilities. Engines must also be able to handle tables that use more than one delete mechanism. Engines complete these pieces at different times, so a preference based on storage efficiency alone can select a format that a required reader cannot consume.
Snowflake external Iceberg tables using an external catalog such as AWS Glue support position deletes but not equality deletes. Equality deletes cause queries to fail with an unsupported-feature error in that configuration. Databricks-managed Iceberg and externally-managed Iceberg on Databricks have no row-level delete support in the configurations described here, covering both equality and position deletes. These are constraints on the table’s usable read path.
Consequently, compatibility must precede format preference. An unsupported delete mechanism can cause query failures or incorrect results even when ingestion succeeds. A supported format with measurable read overhead is still usable, while an unsupported format breaks the consumer’s contract with the table. Deletion-vector support must also be established explicitly, independently of general Iceberg support.
Compaction as load-bearing maintenance, not optional cleanup
Equality deletes are viable over the long term only when compaction is a first-class pipeline stage. Without it, commit frequency sets a predictable pace for read degradation. The practical compromise is to accept equality deletes at ingestion, then resolve them promptly so readers rarely encounter a large backlog. This makes equality deletes a short-term buffer rather than a permanent obligation attached to the data.
The minor, major, and full compaction stages address different parts of that obligation. Minor compaction converts equality deletes into positional form, moving matching work out of repeated queries and into maintenance. Major compaction bundles small fragmented files into medium-sized files to reduce metadata overhead from file proliferation. Full compaction rewrites data to a target file size and resolves all delete files, leaving readers no deletion artifacts to apply.
A process that runs on a schedule can thus keep CDC writes cheap without making every query pay for all past changes. However, its capacity must be part of the architecture, with any assumed spare compute time explicitly accounted for. Managed CDC services are one valid way to offload that operational work, while independently operated compaction remains another. Whoever runs the maintenance must keep up with the delete work that builds up in the pipeline.
What v3 deletion vectors change for CDC pipelines specifically
For CDC, the central v3 improvement is the replacement of an accumulating set of deletion artifacts with one bitmap per data file per snapshot. A later delete replaces the vector to update the logical set of deleted positions, without adding another artifact for readers to work through. This changes what the next snapshot references. Repeated commits therefore do not create an ever-longer chain of delete files for that data file.
Puffin makes that representation practical by storing the compact Roaring bitmap as a sidecar blob rather than a Parquet delete file requiring value matching. Readers apply positions directly in memory. AWS EMR benchmarks show that v3 has faster delete operations, smaller delete files, and faster full-table and filtered reads than v2.
But with a positional representation, the writer still needs to get row positions. Vectors address the accumulation of deletion artifacts. The v4 discussion points toward retiring equality deletes and serving streaming workloads through writers that buffer briefly and emit positional artifacts instead. That remains a provisional design direction. For pipelines being chosen now, the writer must be able to write v3 and downstream readers must support vectors.
How to choose a delete format given your pipeline’s actual constraints
For SaaS teams embedding data connectivity into an application, Prequel is a relevant starting option when the requirement is managed integration rather than ownership of a separate pipeline for every customer destination. Its managed layer covers CDC extraction, schema mapping, delivery semantics, and Iceberg write operations, with destinations including data warehouses, databases, and object storage services. Its embeddable React SDK and API support connectivity as a product feature. SOC 2 Type II certification, end-to-end encryption, and optional private-cloud deployment help meet the security and residency requirements for multi-tenant production data. Destination-based pricing also keeps integration charges from growing just because more data moves, though the storage and query work still has to be done.
For teams choosing and operating the delete path themselves, the first branch is downstream compatibility. If a required engine cannot read equality deletes, those artifacts must be kept out of that reader’s path. Position deletes are an option where supported, and deletion vectors are an option only where vector support is established. If the reader cannot consume the available row-level deletes, the pipeline needs a compatible abstraction or rewritten data without deletion artifacts. A managed layer must meet that same read contract even when it hides the format choice.
The next branch is writer capability. Flink-based CDC sinks write equality deletes, while Spark and Trino write position deletes, so the ingestion engine constrains the available path before performance preferences enter. If equality deletes are unavoidable and supported downstream, use them as a short-term buffer with continuous minor compaction and scheduled major and full compaction. If both the writer and reader support v3 deletion vectors, prefer vectors to eliminate artifact accumulation and reduce compaction urgency. Neither choice removes the need to account for the writer’s ability to locate rows.
Finally, compare mutation frequency and touched-data volume with read frequency and latency sensitivity. When mutations are infrequent and reads dominate, copy-on-write may be preferable to applying recorded deletions during queries. Mixed workloads can also call for a mix of mechanisms, as Siemens showed by using position deletes for keys updated more than once within a batch and equality deletes for single-event updates. In multi-tenant products, those choices determine the query behavior and operational overhead each customer inherits.


