Flows as code

Git in Etlworks

Keep flows, connections, formats, macros and schedules in a Git repository, review changes as pull requests, and promote them from development to production. Etlworks talks to Git directly: no client is needed on the server, and the etlworks client does the same from a laptop or CI. Available in Etlworks 10.1.0 and newer.

Git Repository connection

Everything starts with a Git Repository connection (Connections → Add → Developer Tools). It works with GitHub, GitLab, Bitbucket, Azure DevOps and any server that speaks Git over HTTPS or SSH.

FieldWhat to enter
Repository URLhttps://github.com/acme/etl.git or ssh://git@github.com/acme/etl.git (HTTPS and ssh:// URLs only). Never put a token or password in the URL.
AuthenticationNone (public repositories), HTTPS with a username and an access token or password, or SSH with a private key file and a known hosts file (both required, uploaded with the key picker) and an optional passphrase.
Default branchThe branch used when none is given. Empty: the repository's default.
Commit author name and emailWho commits from Etlworks. Empty: the user who runs the command.
If the repository is unavailable, use the last fetched commitFor code from Git: when the host can't be reached, runs use the newest commit this server already fetched, with a warning, instead of failing.

Test Connection lists the branches and tags. The server needs git (and ssh for SSH URLs) installed; Etlworks' Docker images include it, and Test Connection says so when it is missing.

Remote and workspace

Etlworks uses a repository in two ways, which explains most of how the features behave:

A Super Admin tunes the server-wide behavior under Settings → Git; tenant configuration can override most of it per tenant.

SettingDefault
Git Executablegit
Git Work Root{app.data}/git
Git Operation Timeout (ms)300000
Maximum Repository Size (bytes)1073741824 (1 GiB)
Maximum Repository Files100000
Git Workspace Retention (days)30
Git Snapshot Cache FolderEmpty: /var/cache/etlworks/git-code when it exists, otherwise the system temporary folder. On a virtual machine, set a local folder outside /tmp.
Git Snapshot Cache Size (bytes)2147483648 (2 GiB)

Repository layout

A repository can hold one project or several, each in its own folder such as projects/analytics, next to plain SQL and scripts that flows read at run time. The folder format (manifest, object files, references, credentials) is on Project format.

Start managing flows in Git

Manage in Git is the one-step start. In the flows list, select the flows, then Bulk actions → Manage in Git… and choose the Git connection, the branch (or New branch…, which starts from the default branch), the project name and the folder. Etlworks exports the flows with everything they use, pushes, and records them as managed in Git at that commit. Nothing changes in Etlworks itself.

CLI
git-manage "Analytics repo" flows="Load orders,Load customers" project=analytics path=projects/analytics

Promotion guide: development → test → production

One repository, one branch per stage of review, and an environment file per target. Development happens in a development tenant; test and production are only ever changed by deploys.

  1. Export from development to a branch. Build and test flows in the development tenant, then push them to a feature branch:
    CLI
    git-sync-out "Analytics repo" branch=feature/orders path=projects/analytics project=analytics flows="Load orders,Load customers"
  2. Review and merge the pull request on your Git host. Each object is its own file, so the diff shows exactly what changed.
  3. Plan, then deploy to test from main with the test environment. The plan lists every create, update, keep and delete before anything changes; plan-hash makes the deploy refuse a plan that changed since you reviewed it:
    CLI
    git-sync-in "Analytics repo" ref=main path=projects/analytics env=test dry-run=true
    git-sync-in "Analytics repo" ref=main path=projects/analytics env=test plan-hash=3f1a9c0d2e4b
  4. Tag a release and deploy it to production with the production environment. A tag never moves, so production runs exactly what was tested:
    CLI
    git-sync-in "Analytics repo" ref=release-2026.11 path=projects/analytics env=prod dry-run=true

Git commands

All of them take the Git Repository connection's name or ID first. Commands that change the repository or the workspace ask for confirmation.

CommandWhat it does
git-managePuts flows under Git in one step (Manage in Git): flows=, project=, path=, branch=, all-types=true, takeover=true.
git-sync-outExports flows, with what they use, as a project to a branch, commits and pushes. flows defaults to all; clean=true (the default with flows=all) removes files of objects no longer exported.
git-sync-inDeploys a project from a branch, tag or commit: plan and apply in one command. ref=, path=, env=, dry-run=true, plan-hash=, prune=true, collision options.
git-refsBranches, tags and the default branch.
git-logHistory: ref=, path=, count= (up to 500).
git-status, git-diffThe workspace: branch, changed files, commits to push or pull, and uncommitted changes (path=, stat=true).
git-pullBrings the remote's commits into the workspace. Fast-forward only, nothing is merged; branch= switches first.
git-commitCommits all changes, or paths=. Nothing is pushed.
git-pushPushes committed work. Never forces: if the remote has new commits, run git-pull first.
git-branch, git-tagCreate and push a branch (from=) or a tag (ref=). Never replace an existing one.
git-resetDiscards uncommitted changes and unpushed commits in the workspace; the remote is untouched. Requires --force.
git-sync-out makes the branch match Etlworks

A change made only in Git, and not deployed here, is reverted by the next git-sync-out from a tenant that doesn't have it (the commit keeps it in the Git history). Run git-sync-in first, or sync out to a feature branch and merge.

Run them in the CLI console, in a Run CLI Script flow, with the etlworks client (etlworks run "git-sync-in …"), or through the REST API. Full syntax and roles: command reference.

Code from Git

A flow can run SQL and scripts kept in the repository instead of pasted into the flow. Every code field of a flow has a Code from Git button: pick the connection, the branch or tag and the file, preview it, and Use this file. The field then shows From Git: analytics · sql/orders/merge.sql @ main.

Inside other code, use a reference (the Explorer's Copy reference copies one):

SQL
{{git:analytics:sql/orders/merge.sql@release-2026.11}}
-- {{git:<connection>:<path>[@<branch|tag|commit>]}}; without @, the connection's default branch

Run CLI Script flows

Run CLI Script (group Automation and Orchestration) is a flow type that runs CLI commands on the app, so deploys, exports and Git commands can be scheduled, chained in workflows, or started by webhooks. Only Administrators see it in the gallery and can create or edit these flows; for everyone else the editor is read-only.

FieldNotes
ScriptCLI commands separated by ; or new lines, up to 64,000 characters.
Timeout (seconds)30 to 3,600; default 900. A command still running at the deadline is cancelled.
Stop on first errorOn by default. Off: every command runs, and the flow fails if the last one failed.
Allow changesOff by default. Without it the script can only read.
Allow destructive and operational actionsRequires Allow changes. Allows deletes, imports, deploys (git-sync-in, project-apply), pushes, and running flows without a confirmation.
Git webhook secretThe shared secret for deploy on push.
HTTP listenerOptional: the listener a Git host's push webhook calls.

Deploy on push

Deploy a branch the moment it changes: the Git host's webhook calls an Etlworks HTTP listener, which runs a Run CLI Script flow.

  1. HTTP listener connection: a URL such as /git/push, method POST, Include request headers on, authentication JWT with the location Query Parameter (or Basic).
  2. Run CLI Script flow with the listener as its HTTP listener, Allow changes and Allow destructive on, a Git webhook secret, and the script:
    CLI
    git-sync-in "Analytics repo" ref={git_branch} path=projects/analytics env=prod --force
  3. Schedule the flow on the listener.
  4. Add the webhook on the Git host for push events, with content type JSON and the same secret:
    • URL: https://<your instance>/rest/v1/httplistener/git/push?Authorization=Bearer%20<API key>.
    • GitHub and Bitbucket sign the body with the secret (X-Hub-Signature-256).
    • GitLab sends the secret as its token (X-Gitlab-Token).
    • Azure DevOps has no signature: use Basic authentication on the listener instead.

A wrong or missing signature fails the run before any command runs. The script gets {git_branch}, {git_tag}, {git_ref}, {git_commit} and {git_repository} from GitHub, GitLab, Bitbucket and Azure DevOps push events. They are set only when the name is plain (letters, digits and . _ - / + @); the request body, its query parameters and headers never become script variables, so nothing a pusher controls can add commands. A listener without authentication requires the webhook secret.

Etlworks webhooks that run a flow

Etlworks' own webhooks (event notifications) have a Run Simba or CLI flow option: when the webhook's events fire, the selected flow runs on the app, as the Administrator who saved the webhook. The flow gets {webhook_event_id} and {webhook_event_type}; the Git signature check doesn't apply. Events about Simba or CLI flows themselves are ignored, so a flow can't trigger itself.

Managed objects and drift

Every object a project deploys is managed in Git. Etlworks remembers where it came from and notices when someone changes it in Etlworks afterwards. That's drift: the next deploy overwrites those changes.

Git in the Explorer

Git Repository connections appear in the Explorer with two views:

Step by step: Working with Git repositories in Explorer.


More detail

The support center guide, Data Integration as Code, walks through every dialog with screenshots and lists the exact error messages with fixes. A workflow that should work better? product@etlworks.com.