dbt integration

Run dbt on the connections you already have.

dbt is a first-class flow type in Etlworks. Point it at an existing warehouse connection and the engine builds the dbt profile at run time, no second credential store, no wrapper scripts. Or trigger a dbt Platform job and watch it in the same history.

Start free trial Talk to us

10: Warehouses and databases
0: Duplicate credentials
2: Execution modes
Git: Pinned projects

The problem

dbt only does the T.

Something has to land the data before dbt can model it, and something has to schedule dbt, hold its credentials, pin its project version, and tell you what happened. That is usually a second tool, a third credential store, and a wrapper script nobody wants to own.

Why a shell step is not an integration

Calling `dbt run` from a script is the easy part.

Plenty of tools can execute a command. What they leave you with is a warehouse password copied into a second place, a project on whatever revision the box happened to have, logs in a file nobody reads, and no answer to “which commit produced this table, and did the tests pass?” The work is in the connection handling, the project provenance, and the results, not in launching the process.

Capabilities

What’s built in.

Your warehouse connection, projected

Pick an existing Etlworks warehouse connection. At run time the engine projects it into a dbt profile on the execution node. You never keep a second copy of the credentials.

A real flow type, not a shell step

dbt is its own flow type, alongside JavaScript, CLI, and SSH flows. It nests inside Composer and workflows, so extract, load, dbt, and downstream delivery run as one governed execution.

The commands you actually use

build, run, test, seed, snapshot, compile, debug, list, plus source freshness and docs generate. Selectors, excludes, variables, target, threads, full refresh, and fail-fast.

Git-pinned projects

Point a flow at a Git repository, ref, commit, and subdirectory. Every run checks out an immutable revision into a private workspace and records the exact commit in execution history.

deps, state, and defer

dbt deps honours your lockfile with bounded, cached package acquisition. State and defer can select an authorised prior Etlworks run or an uploaded bundle, with version and project identity validated first.

Results in flow history

manifest, run_results, catalog, and freshness artifacts are captured and stored per tenant. Model and test outcomes, timings, warnings, and downloads sit in the same history as every other flow.

Specifications

Supported connections.

An approved compatibility matrix, not a generic adapter claim. Each row is a connection type and authentication mode validated against a pinned dbt Core and adapter pair. Anything not listed fails validation with a clear error rather than at run time.

Analytics warehouses

Snowflake

password, PKCS#8 key pair, Azure AD OAuth service principal · dbt-snowflake

Google BigQuery

OAuth2 token, JSON service-account key · dbt-bigquery

Databricks

personal access token, OAuth service principal · dbt-databricks

Amazon Redshift

username/password, SSL options · SSH tunnel · dbt-redshift

Azure Synapse Analytics

username/password, Entra, service principal, MSI · dbt-synapse

Microsoft Fabric Warehouse

Entra service principal, interactive OAuth · dbt-fabric

Databases

PostgreSQL

username/password, SSL files and mode · SSH tunnel · dbt-postgres

Amazon Aurora PostgreSQL

username/password, SSL files and mode · SSH tunnel · dbt-postgres

Microsoft SQL Server

username/password, Entra, service principal, OAuth, MSI · SSH tunnel · dbt-sqlserver

ClickHouse

username/password · SSH tunnel · dbt-clickhouse

Runtime

dbt Core

Pinned runtime, validated against the installed adapter before a run starts

dbt Platform

Dedicated connection · trigger an existing job, poll, cancel, collect logs and artifacts

Before you start

What has to be installed.

dbt Core runs as an operating-system process on the execution node, either the Etlworks host or an Integration Agent. That runtime is yours to install. Etlworks never installs dbt, an adapter, Git, an ODBC driver, or a native library on your behalf, so nothing appears on a machine without an administrator putting it there.

On the execution node

dbt Core

The validated baseline is 1.12.3

Your warehouse adapter

At the version pinned in the compatibility matrix above — dbt-snowflake, dbt-postgres, and the rest

Git

Needed for Git-backed projects and for Git flows

unixODBC and ODBC Driver 18

Microsoft’s driver, needed for SQL Server, Azure Synapse, and Microsoft Fabric Warehouse

Two settings decide where dbt runs

dbt Executable

dbt.executable · an executable name or an absolute path · defaults to dbt

dbt Project Root

dbt.project.root · the root directory for local projects · defaults to {app.data}/dbt

Precedence

Tenant settings override global ones, which override the defaults. On an Integration Agent the equivalent values come from the Agent parameters.

The work directory itself is not configurable. It is created below the Etlworks temporary folder and removed when the execution ends.

Hosted instances ship without a dbt runtime or any warehouse adapter. On a hosted instance there are three ways forward. Run local dbt Core on an Integration Agent, a machine you control and install on yourself. Use the Execute dbt Platform Job flow, which needs no dbt installation anywhere. Or ask Etlworks support to install dbt and the adapters on your instance, naming the adapters you need. On-premise installations put the runtime on the Etlworks host.

FAQ

Common questions.

Do I have to re-enter my warehouse credentials for dbt?

No, and that is the point of the integration. A local dbt flow selects an existing Etlworks warehouse connection. The engine resolves it and writes an execution-scoped dbt profile on the node that runs the job. Secrets are never written into the flow definition, never sent to the browser or the CLI, and the workspace is destroyed when the run finishes.

Does this replace dbt, or run it?

It runs it. dbt Core executes as a real process against your project, so your models, tests, macros, and packages behave exactly as they do locally. Etlworks supplies the connection, the project revision, the schedule, the permissions, and the history around it.

Can I use dbt Cloud instead of running dbt Core?

Yes. A separate flow type triggers an existing dbt Platform job through a dedicated connection, then polls it, surfaces the run URL, supports cancellation, and pulls logs and artifacts into Etlworks history. The Platform job keeps its own warehouse profile; Etlworks never copies those credentials.

Which warehouses are supported?

The ones in the table above, at the authentication modes listed there. This is an approved compatibility matrix rather than a generic JDBC claim: a connection or authentication mode that is not listed is unsupported, even where a community adapter exists. Unsupported combinations fail validation with an actionable error instead of failing mid-run.

How does Etlworks know which version of my project ran?

A dbt flow can take its project from an administrator-allowlisted local path or from Git. With Git, you give a repository, ref, commit, and optional subdirectory, and each run materialises that exact revision into a private per-run checkout. The resolved commit is recorded in execution history, so a run is reproducible after the fact.

Can dbt run as part of a bigger pipeline?

That is the normal case. A dbt flow nests inside Composer and standard workflows, so a single execution can land raw data, run dbt, test it, and push the result downstream. Scheduling, retries, cancellation, permissions, logging, and history are the same mechanisms every other Etlworks flow uses.

Start your trial

14 days. No card. Real workloads.

Connect your warehouse, point a flow at your dbt project, and run it on a schedule with the rest of your pipeline.