All posts

A shell step is not a dbt integration

Any tool can run dbt run. That leaves you a second copy of your warehouse password, a project on whatever revision the box had, and no answer to which commit produced this table.

Plenty of platforms claim dbt support. Open the documentation and it is a shell step with dbt run in it.

That does execute dbt. It also leaves you a warehouse password copied into a second place, a project on whatever revision the build agent checked out, logs nobody reads, and no answer to the question that matters after something breaks: which commit produced this table, and did the tests pass?

Launching the process is the easy part. The work is everything around it.

Your warehouse connection, not a second one

A dbt flow selects a warehouse connection you already have. At run time the engine projects it into a dbt profile on the node that runs the job. Nothing is copied into the flow definition, nothing is sent to the browser or the CLI, and the workspace is destroyed when the run ends.

The existing Script to run before connect hook still fires, and SSH tunnels still open for one execution and close before the workspace is deleted. If you use them today, dbt uses them too.

An approved matrix, not a generic adapter claim

Each supported combination is a connection type and authentication mode validated against a pinned dbt Core and adapter pair. The runtime is dbt-core==1.12.3, with dbt-snowflake==1.12.0, dbt-bigquery==1.12.0, dbt-databricks==1.12.5, dbt-redshift==1.11.1, dbt-postgres==1.11.0, dbt-sqlserver==1.11.1, dbt-synapse==1.8.5, dbt-fabric==1.9.10, and dbt-clickhouse==1.10.3.

A combination that is not on the list is unsupported, even where a community adapter exists. Those fail validation with a clear error before dbt starts, rather than halfway through a run against production.

The three Microsoft adapters reach the warehouse through Microsoft ODBC Driver 18 and pyodbc, so the execution host needs unixODBC and that driver installed.

We reject instead of guessing

Redshift's load and unload IAM role is not a database login credential, so rather than infer one from unrelated fields we reject IAM database authentication with an error that names the field. Databricks takes a personal access token or an OAuth service principal, exactly one, and a mixed configuration is refused. BigQuery service-account mode takes a JSON key; legacy P12 material is not a valid dbt-bigquery key file, so it is rejected.

Guessing produces a run that fails later and further from the cause.

Reproducible, or it does not count

Point a flow at a Git repository, ref, commit, and subdirectory. Every run materializes that revision into a private per-run checkout and records the resolved commit in execution history. There is no mutable shared working tree, so two runs cannot quietly disagree about what they executed.

dbt deps honours your lockfile with bounded, cached package acquisition. State and defer select an authorized prior run or an uploaded bundle, with version and project identity validated before anything runs.

Results live in the same place as everything else

manifest, run_results, catalog, and freshness artifacts are captured per tenant. Model and test outcomes, timings, warnings, sources and freshness, selected-node lineage, and authorized downloads sit inside the existing execution history. Defaults are 100 MiB per artifact, 250 MiB per run, a 10 GiB per-account quota, and raw artifacts kept 30 days with the latest 10 artifact-bearing runs per flow preserved. All configurable.

One rule governs that subsystem. If artifact capture or the quota fails, the result of a completed dbt transformation does not change. Storage trouble must never report a successful warehouse run as a failure. Losing artifacts quietly is not acceptable either, so you get the result plus a warning naming what went missing.

Where it stops

dbt Core runs as a real process against your project, so your models, tests, macros, and packages behave as they do on your laptop. Prefer your dbt Platform jobs? A separate flow triggers one, polls it, surfaces the run URL, supports cancellation, and pulls logs and artifacts back. That job keeps its own warehouse profile; we never copy those credentials.

Some matrix rows passed live acceptance against real accounts on 15 September. The rest are covered by automated adapter-contract testing only, and we track them that way rather than rounding everything up to "supported". Running dbt on our Integration Agent is not shipped yet.

The full compatibility matrix and the connection details are on the dbt integration page.

Try it on your own data.

14 days, no card. Point Etlworks at a real source and see what it does.