9.9.9: dbt flows, client-side bulk load, and built-in SSO
dbt Core and dbt Platform run as Etlworks flows, three databases get client-side bulk loading, and OpenID Connect single sign-on no longer needs a broker.
The biggest release of the year.
dbt is a flow type. dbt Core projects and dbt Platform jobs run as regular Etlworks flows. Projects come from Etlworks-managed storage or from a Git repository pinned to an exact commit, and your existing warehouse connection supplies the credentials through a profile built at run time, so there is no second profiles.yml anyone has to maintain. Package management with tenant-aware caches, state and defer, native dbt retry, and a dbt Results tab with visual lineage all ship with it, and dbt flows work with schedules, nested flows, the API, the CLI, and Integration Agents like any other flow. Why a shell step is not an integration, and the dbt integration page has the compatibility matrix.
Client-side bulk load for SQL Server, PostgreSQL, and MySQL. The Bulk Load SQL field accepts three provider directives: SQLSERVER_BULK_COPY, POSTGRES_COPY, and MYSQL_LOAD_DATA_LOCAL. Etlworks streams the staged CSV over the JDBC connection using each database's native bulk-ingestion API, which means the database server never needs access to the staged file. That is what makes staging in S3, Azure Storage, or Google Cloud Storage possible for these three. Split a large extract and load the partitions in parallel for the real gains.
A Bulk Load Builder sits in front of that field. It reads the destination, the staging connection, and the format, recommends the load methods that fit, disables the ones that do not and says why, then writes the directive or the server-side SQL template for you.
Built-in OpenID Connect single sign-on. Etlworks authenticates directly against Microsoft Entra ID, Google, Okta, or any OIDC-compliant provider, with no broker in the authentication path. Login behavior, unknown-user provisioning, and email-based account linking are all configurable, and the login and sign-up pages support signing in with Google, Microsoft, or Apple. SSO stays an enterprise feature.
Files and folders in Etlworks Explorer, our built-in view of connections and files. Browse nested folders, create, rename and delete folders, create and edit text files in place, upload into a chosen folder, and copy root-relative paths. Separately, every flow that selects files can now recurse into subfolders, and Copy, Move and Rename can preserve or flatten the source structure.
Parallel MongoDB. Large collections can be chunked by _id and snapshotted across threads with per-collection resumability, and ETL flows with a MongoDB source can partition collections the same way. Why the big table is the whole problem.
Also in this release: Get help on a failed execution, which asks Simba, our AI agent, to read the log and explain the failure or opens a ticket with the log attached; HTTP request and response logging with authorization headers redacted; tail-app-log and tail-flow-log in the CLI for reading logs without shell access on the host; z/OS path syntax on FTP and FTPS connections, with PDS and PDSE data sets appearing as folders; and license refresh across live nodes after a renewal, with no restart.