Flows as code

Flows as code

Since Etlworks 10.1.0, flows, connections, formats, macros and schedules can live in Git as readable files. You review changes in pull requests and deploy a project to any environment (dev, test, production, another tenant or another instance) with plan and apply. The visual editor keeps working as before; nothing changes for users who don't use Git.

Two ways to work

Both use the same project format and the same plan and apply engine, and you can mix them.

Git in EtlworksCI client
Who talks to GitEtlworks, through a Git Repository connectionYour pipeline checks out the repository
Git credentialsStored in the connectionNever given to Etlworks
Deploy withgit-sync-in, or deploy on push from a webhooketlworks project plan and apply
ExtrasManage in Git, code from Git, Git in Explorer, drift in the UIExit codes for gating pipelines, plan files
Start hereGit in EtlworksCI client

Git is optional. A project folder can also live on the server (project-export, then project-apply projects/analytics) or travel as a single readable bundle file.

Quickstart

From Etlworks, with a Git Repository connection called Analytics repo: put two flows under Git, then deploy the project to a test tenant.

CLI, development tenant
git-manage "Analytics repo" flows="Load orders,Load customers" project=analytics path=projects/analytics
CLI, test tenant
git-sync-in "Analytics repo" ref=main path=projects/analytics env=test dry-run=true
git-sync-in "Analytics repo" ref=main path=projects/analytics env=test plan-hash=<hash from the dry run>

From a laptop or CI, with the etlworks client:

Shell
etlworks project export ./analytics --project analytics --flows "Load orders,Load customers" --profile dev
etlworks project plan  ./analytics --env test --profile test --out plan.json
etlworks project apply ./analytics --env test --profile test --plan plan.json

Concepts

TermMeaning
ProjectA named set of objects deployed together. On disk, a project folder with a manifest and one file per object.
ObjectA flow, connection, format, macro or schedule that belongs to a project.
Manifestetlworks.json at the root of the folder: the project name and the authoritative list of its objects.
Readable bundleA single .etlworks.json file with the whole project: the readable twin of the encrypted .etld export.
Matching by nameObjects are identified by name and type (trimmed, case-insensitive), never by internal ID, so a project deploys to any tenant or instance. Two objects with the same name and type on the target are an error.
EnvironmentValues that differ per target (hosts, warehouses, variables, schedules), from environments/<env>.json and from values stored in Etlworks.
Collision policyWhat a deploy does with an object that already exists on the target: replace it, keep it, or fail.
Plan and applyThe plan shows what a deploy would do, object by object, and changes nothing. Apply deploys in one transaction.
Managed objectAn object recorded as belonging to a project after Manage in Git or a deploy. It gets a Git badge, a source panel and drift tracking.
DriftA change made directly in Etlworks to a managed object after its last deploy. The next deploy overwrites it, and the plan shows it first.
PruneDeleting objects that the project deployed earlier and no longer contains.
OwnershipEvery deployed object belongs to one project. Moving it to another project needs an explicit takeover.

Permissions

RoleCan
ViewerList branches and tags (git-refs), project-validate, project-drift, project-compare.
Editorproject-plan, project-apply, project-export, the workspace Git commands (git-status, git-commit, git-push and the rest), git-manage, project-revert, project-env-init, project-env-get, project-encrypt-value.
Tenant AdminStored environment values: project-env-set, project-env-unset.
Administratorprune=true, takeover=true, and creating or editing Run CLI Script flows and the schedules and workflows that run them.

Git commands also need access to the Git Repository connection: read access for git-refs, Edit for anything that changes the repository or the workspace. Commands that push, deploy, tag or discard work ask for confirmation.

What a project includes

Limits and defaults

WhatLimit or default
Project zip upload20,000 files and 500 MB unzipped
Client project upload200 MB, kept for one hour
Deploy lockOne deploy per tenant and project; a second waits up to 60 seconds
Workspace lockOne Git command per workspace; a second waits up to 60 seconds
Code from Git5 MB per file, 50 files per run
Explorer preview, edit and save5 MB; uploads up to 50 MB per file
Run CLI ScriptScript up to 64,000 characters; log capped at 1 MB; timeout 30 to 3,600 seconds (default 900)
Repository size1 GiB and 100,000 files (Settings → Git)
Git operation timeout300,000 ms (Settings → Git)
Workspace retention30 days unused (Settings → Git)
git-logUp to 500 commits

More detail

The support center guide, Data Integration as Code, walks through every dialog with screenshots and lists the exact error messages with fixes. A workflow that should work better? product@etlworks.com.