The big table is the whole problem
Etlworks discovers the key and reads one large table as balanced parallel ranges, in ETL flows and in CDC snapshots. Threads are the ceiling. The multiplier is the queue.
Ten small tables parallelize themselves. The 800-million-row orders table does not, and it decides when the load finishes.
Until recently the fix was writing Partition SQL by hand, plus a source query tokenized to match. It still exists for custom boundaries, but most of the time nobody should need it.
Etlworks now discovers the key, calculates balanced ranges, and reads them in parallel: in ETL flows and CDC snapshots, on relational sources and MongoDB.
Threads are the ceiling, the multiplier is the queue
Maximum Number of Parallel Threads on the flow is the real concurrency limit. Automatic partition multiplier sets how many ranges are planned per thread. Requested ranges are the two multiplied.
The multiplier never adds threads and never opens more connections. It creates more, smaller work units so a thread finishing early has something to pick up. Start at 1 and raise it when ranges finish at noticeably different times. Setting it while threads stay at 1 does nothing.
Ranges are balanced by row position
The planner reads the ordered primary key and picks boundaries by row position, not by dividing the key space into equal widths. A table whose ids run 1 to 900 then jump to 40,000,000 still produces even ranges. Duplicate boundaries are dropped, so a low-cardinality key cannot produce empty ones.
The first range is open at the bottom and the last is open at the top, so a row inserted above the maximum seen during planning still belongs somewhere. Composite keys work, using lexicographic predicates with typed bind variables.
Turn it on with Automatically partition source tables by primary key, shipped in 9.7.11. No Partition SQL, no tokens in the source query; {table} works in wildcard flows.
It does not partition everything, and says so
The checkbox is a request, not a guarantee. A table with no primary key runs as one transformation, and so does a view with no key metadata, a source with Max Records set, one with Before Prepare Source Query, or a query whose token resolves only at runtime. Each falls back safely and writes the reason to the flow log. The feature is off by default and existing flows are unchanged. Flows on an Integration Agent need the Agent upgraded too, or they keep running as one transformation.
CDC snapshots, same idea
A full snapshot has the same problem: table-level parallelism reads several tables at once, which does nothing for the one that dwarfs the rest.
Chunk large tables across snapshot threads splits one keyed table into non-overlapping ranges sharing the existing worker pool. Snapshot Max Threads is still the ceiling and Snapshot Thread Multiplier does the same job as in ETL. Shipped in 9.7.1 for Oracle, SQL Server, MySQL, PostgreSQL, and AS400/IBM i; DB2 LUW and legacy MongoDB ignore it.
This is not Debezium incremental snapshot, which runs alongside streaming and chunks on its own.
MongoDB, since 9.8.5
ETL gets Automatically partition MongoDB collections by _id, planning typed ranges. A JSON filter is combined with each range using $and, so it keeps its meaning; shell-style queries fall back to one cursor. CDC gets Chunk large collections across snapshot threads for modern MongoDB.
Sharded clusters stay unchunked unless you opt in, because an _id range can scatter across every shard. ETL opts in with automatic.partition.sharded.enabled=true in Other Parameters; CDC has Allow chunked snapshots for sharded MongoDB.
Reading faster only helps if the destination keeps up
Partitions load concurrently as well as read concurrently, so the destination is part of the tuning. Snowflake takes a wildcard COPY INTO then a MERGE; Databricks combines staged files into one COPY INTO.
File destinations need one thing: every partition must resolve to a unique file. Configure Add Suffix When Creating Files in Transformation; Flow Inspection flags it if you forget.
Two limits, stated plainly. Parallel ranges use separate connections, so this is not a consistent point-in-time snapshot; take a maintenance window when an exact one matters. And none of it helps unless reading the source is your bottleneck and source, Agent, and destination have capacity to spare.
Full detail in the docs: Automatic partitioning, Snapshot Management, Optimizing CDC Snapshots for Large Databases, and ETL with MongoDB.