Do I need Kafka or Debezium installed?

No. Etlworks's CDC engine is built in and runs inside the platform. It uses the same Debezium-compatible binlog reading approach but doesn't require a separate Debezium installation, Kafka cluster, or schema registry. If you already use Kafka, you can publish CDC events to it as a destination — but it's not a prerequisite.

What's the latency from source change to destination?

Sub-second in most production deployments. Actual latency depends on source database load, network, and destination throughput, but the engine itself processes change events as they're emitted by the database log. Lag metrics are visible per-pipeline in monitoring.

How does Etlworks handle schema changes on the source?

Schema evolution is automatic for additive changes (new columns, larger types). Destructive changes (dropped columns, incompatible type changes) trigger a pipeline alert and require explicit confirmation before propagating. Schema history is retained for audit.

Can I replicate from on-prem databases to cloud destinations?

Yes. Etlworks's hybrid agent runs inside your network, reads the database log locally, and streams change events out through an outbound HTTPS connection. No inbound ports, no VPN required. Used in production for petabyte-scale on-prem-to-Snowflake / BigQuery / Redshift pipelines.

What happens if the destination is temporarily unavailable?

CDC pipelines persist offsets and resume from the last committed position. Source database log retention is the only constraint — as long as the log is still available, no events are lost. Default retention is 7 days; configurable per source.

How is this different from Debezium?

Etlworks's CDC engine is Debezium-compatible at the protocol level — same binlog-reading approach, same change event format. The differences: it runs inside Etlworks (no separate Kafka Connect / Zookeeper / schema registry), includes a managed UI, supports transformations and routing in the same pipeline, and includes destinations beyond Kafka topics (warehouses, files, APIs). If you want raw Debezium for fan-out to many Kafka consumers, that's still a great fit. If you want managed CDC with destinations and transforms, Etlworks is faster to deploy.

What about pricing for high-volume CDC?

Etlworks pricing is per-tier, not per-row or per-event. CDC throughput is included in every paid plan. High-volume customers (hundreds of millions of changes per day) typically run on Enterprise plans for HA and dedicated support, but there's no per-event surcharge.

dbt integration

Run dbt on the connections you already have.

dbt is a first-class flow type in Etlworks. Point it at an existing warehouse connection and the engine builds the dbt profile at run time — no second credential store, no wrapper scripts. Or trigger a dbt Platform job and watch it in the same history.

Start free trial Talk to us

10: Warehouses and databases
0: Duplicate credentials
2: Execution modes
Git: Pinned projects

The problem

dbt only does the T.

Something has to land the data before dbt can model it, and something has to schedule dbt, hold its credentials, pin its project version, and tell you what happened. That is usually a second tool, a third credential store, and a wrapper script nobody wants to own.

Why a shell step is not an integration

Calling `dbt run` from a script is the easy part.

Plenty of tools can execute a command. What they leave you with is a warehouse password copied into a second place, a project on whatever revision the box happened to have, logs in a file nobody reads, and no answer to “which commit produced this table, and did the tests pass?” The work is in the connection handling, the project provenance, and the results — not in launching the process.

Capabilities

What’s built in.

Your warehouse connection, projected

Pick an existing Etlworks warehouse connection. At run time the engine projects it into a dbt profile on the execution node. You never keep a second copy of the credentials.

A real flow type, not a shell step

dbt is its own flow type, alongside JavaScript, CLI, and SSH flows. It nests inside Composer and workflows, so extract, load, dbt, and downstream delivery run as one governed execution.

The commands you actually use

build, run, test, seed, snapshot, compile, debug, list, plus source freshness and docs generate. Selectors, excludes, variables, target, threads, full refresh, and fail-fast.

Git-pinned projects

Point a flow at a Git repository, ref, commit, and subdirectory. Every run checks out an immutable revision into a private workspace and records the exact commit in execution history.

deps, state, and defer

dbt deps honours your lockfile with bounded, cached package acquisition. State and defer can select an authorised prior Etlworks run or an uploaded bundle, with version and project identity validated first.

Results in flow history

manifest, run_results, catalog, and freshness artifacts are captured and stored per tenant. Model and test outcomes, timings, warnings, and downloads sit in the same history as every other flow.

Specifications

Supported connections.

An approved compatibility matrix, not a generic adapter claim. Each row is a connection type and authentication mode validated against a pinned dbt Core and adapter pair. Anything not listed fails validation with a clear error rather than at run time.

Analytics warehouses

Snowflake

password, PKCS#8 key pair, Azure AD OAuth service principal · dbt-snowflake

Google BigQuery

OAuth2 token, JSON service-account key · dbt-bigquery

Databricks

personal access token, OAuth service principal · dbt-databricks

Amazon Redshift

username/password, SSL options · SSH tunnel · dbt-redshift

Azure Synapse Analytics

username/password, Entra, service principal, MSI · dbt-synapse

Microsoft Fabric Warehouse

Entra service principal, interactive OAuth · dbt-fabric

Databases

PostgreSQL

username/password, SSL files and mode · SSH tunnel · dbt-postgres

Amazon Aurora PostgreSQL

username/password, SSL files and mode · SSH tunnel · dbt-postgres

Microsoft SQL Server

username/password, Entra, service principal, OAuth, MSI · SSH tunnel · dbt-sqlserver

ClickHouse

username/password · SSH tunnel · dbt-clickhouse

Runtime

dbt Core

Pinned runtime, validated against the installed adapter before a run starts

dbt Platform

Dedicated connection · trigger an existing job, poll, cancel, collect logs and artifacts

FAQ

Common questions.

Do I have to re-enter my warehouse credentials for dbt?

No, and that is the point of the integration. A local dbt flow selects an existing Etlworks warehouse connection. The engine resolves it and writes an execution-scoped dbt profile on the node that runs the job. Secrets are never written into the flow definition, never sent to the browser or the CLI, and the workspace is destroyed when the run finishes.

Does this replace dbt, or run it?

It runs it. dbt Core executes as a real process against your project, so your models, tests, macros, and packages behave exactly as they do locally. Etlworks supplies the connection, the project revision, the schedule, the permissions, and the history around it.

Can I use dbt Cloud instead of running dbt Core?

Yes. A separate flow type triggers an existing dbt Platform job through a dedicated connection, then polls it, surfaces the run URL, supports cancellation, and pulls logs and artifacts into Etlworks history. The Platform job keeps its own warehouse profile; Etlworks never copies those credentials.

Which warehouses are supported?

The ones in the table above, at the authentication modes listed there. This is an approved compatibility matrix rather than a generic JDBC claim: a connection or authentication mode that is not listed is unsupported, even where a community adapter exists. Unsupported combinations fail validation with an actionable error instead of failing mid-run.

How does Etlworks know which version of my project ran?

A dbt flow can take its project from an administrator-allowlisted local path or from Git. With Git, you give a repository, ref, commit, and optional subdirectory, and each run materialises that exact revision into a private per-run checkout. The resolved commit is recorded in execution history, so a run is reproducible after the fact.

Can dbt run as part of a bigger pipeline?

That is the normal case. A dbt flow nests inside Composer and standard workflows, so a single execution can land raw data, run dbt, test it, and push the result downstream. Scheduling, retries, cancellation, permissions, logging, and history are the same mechanisms every other Etlworks flow uses.

Start your trial

14 days. No card. Real workloads.

Connect your warehouse, point a flow at your dbt project, and run it on a schedule with the rest of your pipeline.