Flows as code
Since Etlworks 10.1.0, flows, connections, formats, macros and schedules can live in Git as readable files. You review changes in pull requests and deploy a project to any environment (dev, test, production, another tenant or another instance) with plan and apply. The visual editor keeps working as before; nothing changes for users who don't use Git.
Two ways to work
Both use the same project format and the same plan and apply engine, and you can mix them.
| Git in Etlworks | CI client | |
|---|---|---|
| Who talks to Git | Etlworks, through a Git Repository connection | Your pipeline checks out the repository |
| Git credentials | Stored in the connection | Never given to Etlworks |
| Deploy with | git-sync-in, or deploy on push from a webhook | etlworks project plan and apply |
| Extras | Manage in Git, code from Git, Git in Explorer, drift in the UI | Exit codes for gating pipelines, plan files |
| Start here | Git in Etlworks | CI client |
Git is optional. A project folder can also live on the server (project-export, then project-apply projects/analytics) or travel as a single readable bundle file.
Quickstart
From Etlworks, with a Git Repository connection called Analytics repo: put two flows under Git, then deploy the project to a test tenant.
git-manage "Analytics repo" flows="Load orders,Load customers" project=analytics path=projects/analytics
git-sync-in "Analytics repo" ref=main path=projects/analytics env=test dry-run=true
git-sync-in "Analytics repo" ref=main path=projects/analytics env=test plan-hash=<hash from the dry run>
From a laptop or CI, with the etlworks client:
etlworks project export ./analytics --project analytics --flows "Load orders,Load customers" --profile dev
etlworks project plan ./analytics --env test --profile test --out plan.json
etlworks project apply ./analytics --env test --profile test --plan plan.json
Concepts
| Term | Meaning |
|---|---|
| Project | A named set of objects deployed together. On disk, a project folder with a manifest and one file per object. |
| Object | A flow, connection, format, macro or schedule that belongs to a project. |
| Manifest | etlworks.json at the root of the folder: the project name and the authoritative list of its objects. |
| Readable bundle | A single .etlworks.json file with the whole project: the readable twin of the encrypted .etld export. |
| Matching by name | Objects are identified by name and type (trimmed, case-insensitive), never by internal ID, so a project deploys to any tenant or instance. Two objects with the same name and type on the target are an error. |
| Environment | Values that differ per target (hosts, warehouses, variables, schedules), from environments/<env>.json and from values stored in Etlworks. |
| Collision policy | What a deploy does with an object that already exists on the target: replace it, keep it, or fail. |
| Plan and apply | The plan shows what a deploy would do, object by object, and changes nothing. Apply deploys in one transaction. |
| Managed object | An object recorded as belonging to a project after Manage in Git or a deploy. It gets a Git badge, a source panel and drift tracking. |
| Drift | A change made directly in Etlworks to a managed object after its last deploy. The next deploy overwrites it, and the plan shows it first. |
| Prune | Deleting objects that the project deployed earlier and no longer contains. |
| Ownership | Every deployed object belongs to one project. Moving it to another project needs an explicit takeover. |
Permissions
| Role | Can |
|---|---|
| Viewer | List branches and tags (git-refs), project-validate, project-drift, project-compare. |
| Editor | project-plan, project-apply, project-export, the workspace Git commands (git-status, git-commit, git-push and the rest), git-manage, project-revert, project-env-init, project-env-get, project-encrypt-value. |
| Tenant Admin | Stored environment values: project-env-set, project-env-unset. |
| Administrator | prune=true, takeover=true, and creating or editing Run CLI Script flows and the schedules and workflows that run them. |
Git commands also need access to the Git Repository connection: read access for git-refs, Edit for anything that changes the repository or the workspace. Commands that push, deploy, tag or discard work ask for confirmation.
What a project includes
- Included: flows (nested flows and Composer children are followed), the connections, formats and macros they use (including a connection's Explorer format and the HTTP listener connections flows use), tenant macros, and schedules.
- Reported, not exported: admin-level (global) macros, and flows that a dynamic flow runs by name at run time.
- Not included: webhooks, Integration Agents, templates, users and permissions.
Limits and defaults
| What | Limit or default |
|---|---|
| Project zip upload | 20,000 files and 500 MB unzipped |
| Client project upload | 200 MB, kept for one hour |
| Deploy lock | One deploy per tenant and project; a second waits up to 60 seconds |
| Workspace lock | One Git command per workspace; a second waits up to 60 seconds |
| Code from Git | 5 MB per file, 50 files per run |
| Explorer preview, edit and save | 5 MB; uploads up to 50 MB per file |
| Run CLI Script | Script up to 64,000 characters; log capped at 1 MB; timeout 30 to 3,600 seconds (default 900) |
| Repository size | 1 GiB and 100,000 files (Settings → Git) |
| Git operation timeout | 300,000 ms (Settings → Git) |
| Workspace retention | 30 days unused (Settings → Git) |
git-log | Up to 500 commits |
The support center guide, Data Integration as Code, walks through every dialog with screenshots and lists the exact error messages with fixes. A workflow that should work better? product@etlworks.com.