High-Throughput Data Export Architecture at 100M Rows per Sync
Columnar formats, parallel chunking, and incremental capture make nine-figure syncs routine.

Exporting 100 million rows in a single sync is no longer an exceptional engineering feat reserved for the largest data platforms on earth. It is a published, documented ceiling that an ordinary enterprise integration is expected to hit, and the architecture required to hit it reliably rests on decisions including file format, chunking strategy, change-capture pattern, and transfer path.
Exporting 100 million rows per sync as a routine engineering requirement
Salesforce's Bulk API 2.0 formally documents a limit of up to 100 million records per 24-hour period for query jobs. A major CRM vendor does not encode a number like that unless its customers are routinely running into it. Enterprise SaaS has crossed a threshold where nine-figure-row syncs are a published, operational ceiling rather than a stretch goal, and that fact alone should reset how engineering teams scope export work.
The approaches that fail at this scale all share one structural flaw: they treat export as a sequential operation that holds the full job in memory or in a single thread. Row-by-row API calls run into daily API caps long before they reach the volume of a full enterprise export, and the arithmetic forecloses that option before a single optimization gets considered. Single-threaded serialization creates a CPU bottleneck that scales linearly with row count, and adding storage capacity does nothing to relieve it, because the constraint is in compute, not disk. Memory-bound processing, the pattern of loading a full result set before writing anything out, runs into heap limits that are fixed by the architecture itself, not by a configuration flag someone forgot to set. None of these failure modes are edge cases triggered by unusually large customers. They are the default outcome of applying tooling designed for thousands of rows to a job that now routinely runs into the hundreds of millions.
Columnar file formats as the load-bearing decision in a high-throughput export stack
Once row-by-row serialization is off the table, the next decision determines almost everything downstream: what file format the export actually writes. Apache Parquet stores data in a compressed, column-oriented layout, and predicate pushdown combined with column pruning means a query touching only a handful of columns reads a fraction of the physical bytes on disk. For production tables, the optimal file size is between 256 MB and 512 MB: smaller files let file-open overhead dominate the runtime, and larger files degrade the parallelism that the next section depends on. It's a direct consequence of how object storage and parallel readers interact at scale.
The deeper case for columnar formats is interoperability. Snowflake and BigQuery convert ingested data into closed, proprietary internal formats (micro-partitions and Capacitor, respectively), which makes that data expensive to use anywhere outside those platforms. Exporting into Parquet or ORC instead means a customer's Databricks cluster, Redshift Spectrum layer, BigQuery external table, or ClickHouse instance can query the files directly, with no ETL step and no format conversion standing between the export and the query. That property, files that are immediately legible to the entire downstream ecosystem, is what makes the storage-native transfer patterns discussed later in this piece possible.
Apache Iceberg extends this further by wrapping Parquet files in a metadata layer that changes the underlying economics of large-scale export. Scan planning in Iceberg runs in constant time against its metadata rather than scaling linearly against the number of partition directories, and at the scale of thousands of files, that distinction separates a query that plans in milliseconds from one that times out. Iceberg also tracks column-level statistics, minimum, maximum, and null count, for each data file, which lets an export engine skip files that cannot satisfy a given predicate without ever opening them. Arc's September 2026 announcement of zero-copy Apache Iceberg export put this architecture into production: the same files that constitute the export also function as a standard lakehouse table that the downstream ecosystem reads natively, with no proprietary API sitting in the query path and no duplicated storage footprint. That is an architectural property, not a convenience feature: it means the export pipeline and the query engine are reading the identical physical bytes.
Parallelized chunking as a horizontally scalable throughput problem
A columnar file format makes an export efficient to read. Chunking is what makes it finish on schedule. Breaking a 100-million-row export into parallel chunks is the mechanism, not an optional tuning step, that lets the job complete inside a realistic sync window, because a single-threaded export run at any reasonable per-row throughput would stretch across many minutes once serialization, network latency, and destination write time are all accounted for. Parallelism compresses that runtime roughly in proportion to how many workers can operate at once.
Chunk boundaries need to sit on a stable, indexed column, typically a primary key or a monotonic timestamp, so that each chunk is independent, non-overlapping, and resumable without forcing a re-read of the entire table. Chunk sizing is not a decision made in isolation from the file-format choice covered above: at a target of 256 to 512 MB per Parquet file, the chunk size directly determines how many files the job produces and how many writers can run against the destination in parallel. Get the chunk size wrong and the file-size target from the previous section becomes unreachable regardless of how much parallel capacity is available.
Effective chunking also requires the export engine to understand how the underlying data is distributed, beyond just how many rows exist. Chunking strictly by row count produces badly skewed file sizes whenever the data itself isn't uniformly distributed. A chunk that happens to span a high-activity time window can end up many times larger than an adjacent chunk covering a quiet period. Histogram-based or sample-based splitting, which samples the key distribution before cutting chunk boundaries, produces more even physical file sizes and materially better parallelism across the whole job.
Resumability belongs in the same category as chunk sizing: a first-class design requirement built in from the start. At 100-million-row scale, a mid-job failure without checkpoint state means restarting the entire export from zero, a regression that can cost hours. Salesforce's Bulk API 2.0 illustrates exactly this failure mode: a batch that fails and retries repeatedly can appear as a failed job hours into the run, and without chunk-level checkpointing, the entire sync window is lost. Each chunk should write to a staging location atomically before it is committed to the destination, so that a restart resumes from the last committed chunk boundary rather than from the beginning of the table.

Incremental change capture as the default sync pattern
Full-table re-export on every sync cycle is not a sustainable approach. The correct design emits only the delta since the last snapshot, after an initial full load establishes the baseline.
Iceberg v3, released in May 2025, introduced Row Lineage, a mechanism for tracking row-level identity across snapshots, and this is what lets CDC pipelines run directly on Iceberg tables without a separate change-capture system sitting alongside them. Row Lineage lets the export engine compare two snapshots and emit only the inserts, updates, and deletes between them instead of re-reading the full dataset on every run. That comparison is the mechanism that keeps sync windows short on an ongoing basis: the first load moves everything, and every run after that moves only what changed.
Deletes and updates are where many incremental implementations quietly fail. Append-only export pipelines drop deletes silently. The destination keeps accumulating rows that no longer exist at the source, a failure that corrupts downstream analytics and can violate compliance obligations. Relationship data and lookup fields in Salesforce exports show how this plays out in practice: a stale or missing relationship can break a report filter without any visible error, and the only sign something went wrong is a dashboard quietly producing the wrong number.
The destination itself has to be able to absorb a high volume of upserts for any of this to hold up under real load. Apache Pinot 1.4.0's pauseless streaming ingestion is built specifically for mutable, real-time event data, supporting native upserts for late-arriving records without stalling the pipeline. Destinations built around append-optimized storage, where applying an update requires rewriting an entire table, turn into bottlenecks the moment the CDC stream carries a high proportion of updates rather than pure inserts. Choosing a destination for a sync is, in practice, choosing how well that destination tolerates mutation, and that choice has to be made deliberately rather than inherited from whatever warehouse was already in place.
Storage-native transfer patterns that eliminate the application layer from the data path
Everything up to this point assumes data still has to pass through an application tier that reads, serializes, and transmits it. The most aggressive optimization available removes that tier from the path. Every major warehouse has re-architected itself around object storage as its primary persistence layer: Snowflake, Databricks, and Redshift all persist data in cloud object storage, while BigQuery stores data in Colossus, Google's proprietary distributed file system. When source and destination both sit on object storage, an export becomes a file copy operation instead of a query-serialize-transmit-deserialize cycle, and the compute cost of the job drops by an order of magnitude as a result. Arc's zero-copy Iceberg export, announced in July 2026 and shipped in August 2026, is the clearest expression of this pattern in production: the export files are the lakehouse table itself, with no duplicated storage and no interference with the underlying write path.
S3 API compatibility is what makes this pattern portable rather than a one-off integration with a single vendor. S3 compatibility has become a baseline expectation across the object storage landscape, and vendors now differentiate on things like zero-egress cost models (Cloudflare R2, Wasabi), edge integration (Cloudflare, MinIO), and AI-native data services (AWS S3, Google Cloud Storage). An export engine that writes Parquet files to an S3-compatible endpoint can target any of these destinations without changing protocols, which turns the choice of destination into a configuration parameter rather than a separate architectural branch that has to be built and maintained independently.
Once compute is separated from storage at the destination, multiple independent compute clusters can read the same dataset at the same time without contending for the same resources, so a data science workload and a BI dashboard can query the identical exported files without competing for capacity. Engines built as pure compute layers over open formats represent the furthest extension of this idea, operating directly against existing Parquet or Iceberg files: Iceberg wraps Parquet files with a metadata layer that changes the economics of large-scale export, with no migration step required, so a storage-native export becomes queryable the moment it lands.
Destination-aware concurrency controls that protect shared sync performance
Every optimization covered so far increases how fast a single export can run. None of them matter if that speed comes at the expense of every other job sharing the same destination. The most common production failure at this scale is one large enterprise sync quietly starving every other tenant's workload because the pipeline treats every available connection as fair game to use simultaneously.
Tenant isolation has to be enforced at the infrastructure level, not just in application logic, because without it a single job can saturate a shared compute pool or connection budget that other tenants depend on. Salesforce's Bulk API makes this constraint explicit rather than implicit: a fixed ceiling of 25 concurrent ingest jobs and a separate ceiling of 25 concurrent query jobs, both shared across Bulk API 1.0 and 2.0. An export engine that opens more simultaneous jobs than that ceiling allows doesn't run faster. It starts queuing or failing.
A destination-aware export engine treats these limits as a known input to its scheduling logic, not something discovered through trial and error in production. Different destinations carry different concurrency ceilings by design: a Snowflake virtual warehouse enforces a maximum concurrent query limit, a Redshift cluster is bounded by its connection pool, and a BigQuery project operates under slot quotas. Snowflake's step-jump scaling model compounds the risk here, because exceeding a warehouse's concurrency ceiling causes queries to queue rather than fail outright, so an export engine that isn't aware of this ceiling produces a silent latency spike instead of a visible error that anyone would notice and investigate.
The more durable fix is to give the export engine a per-destination connection budget and have it distribute chunk writes within that budget, treating whatever capacity remains as a shared resource across every tenant targeting that destination type. Back-pressure propagation, letting saturation at the destination signal the source to slow down, is what keeps the whole system stable under sustained load. Without it, a pipeline that writes faster than the destination can absorb accumulates in-flight chunks, exhausts its own memory buffers, or triggers destination-side throttling that is expensive to recover from once it starts. Adaptive rate limiting, which reduces parallelism when write latency crosses a defined threshold, holds up better than a static concurrency cap, because it reacts to how the destination is actually behaving rather than to assumptions set at configuration time. An export pipeline built at this scale is responsible for the health of the destination it writes to, beyond how fast it can move its own rows.


