Git in Etlworks
Keep flows, connections, formats, macros and schedules in a Git repository, review changes as pull requests, and promote them from development to production. Etlworks talks to Git directly: no client is needed on the server, and the etlworks client does the same from a laptop or CI. Available in Etlworks 10.1.0 and newer.
Git Repository connection
Everything starts with a Git Repository connection (Connections → Add → Developer Tools). It works with GitHub, GitLab, Bitbucket, Azure DevOps and any server that speaks Git over HTTPS or SSH.
| Field | What to enter |
|---|---|
| Repository URL | https://github.com/acme/etl.git or ssh://git@github.com/acme/etl.git (HTTPS and ssh:// URLs only). Never put a token or password in the URL. |
| Authentication | None (public repositories), HTTPS with a username and an access token or password, or SSH with a private key file and a known hosts file (both required, uploaded with the key picker) and an optional passphrase. |
| Default branch | The branch used when none is given. Empty: the repository's default. |
| Commit author name and email | Who commits from Etlworks. Empty: the user who runs the command. |
| If the repository is unavailable, use the last fetched commit | For code from Git: when the host can't be reached, runs use the newest commit this server already fetched, with a warning, instead of failing. |
Test Connection lists the branches and tags. The server needs git (and ssh for SSH URLs) installed; Etlworks' Docker images include it, and Test Connection says so when it is missing.
Remote and workspace
Etlworks uses a repository in two ways, which explains most of how the features behave:
- The remote is the repository on your Git host. Deploys (
git-sync-in), code from Git and the Explorer's Remote view read any branch, tag or commit directly from it, through a per-commit snapshot cache on each server. - The workspace is Etlworks' own working copy, one per tenant and connection. The Explorer's Local view and
git-status,git-commit,git-pushand the related commands work in it. An unused workspace is removed after 30 days.
A Super Admin tunes the server-wide behavior under Settings → Git; tenant configuration can override most of it per tenant.
| Setting | Default |
|---|---|
| Git Executable | git |
| Git Work Root | {app.data}/git |
| Git Operation Timeout (ms) | 300000 |
| Maximum Repository Size (bytes) | 1073741824 (1 GiB) |
| Maximum Repository Files | 100000 |
| Git Workspace Retention (days) | 30 |
| Git Snapshot Cache Folder | Empty: /var/cache/etlworks/git-code when it exists, otherwise the system temporary folder. On a virtual machine, set a local folder outside /tmp. |
| Git Snapshot Cache Size (bytes) | 2147483648 (2 GiB) |
Repository layout
A repository can hold one project or several, each in its own folder such as projects/analytics, next to plain SQL and scripts that flows read at run time. The folder format (manifest, object files, references, credentials) is on Project format.
Start managing flows in Git
Manage in Git is the one-step start. In the flows list, select the flows, then Bulk actions → Manage in Git… and choose the Git connection, the branch (or New branch…, which starts from the default branch), the project name and the folder. Etlworks exports the flows with everything they use, pushes, and records them as managed in Git at that commit. Nothing changes in Etlworks itself.
- What gets managed: by default, flows and macros. Connections, formats and schedules are exported too, so the project is complete and deployable, but deploys leave them as they are on the target. Tick Also manage connections, formats and schedules (
all-types=true) to manage them as well. - Rights: Edit on the Git connection (to push) and on each selected object. If a selected flow is already managed by another project, nothing is pushed unless you take it over.
- Without Git: the same workflow runs from a plain folder. Export with
project-export(or Export → Readable JSON in the UI) and deploy withproject-apply; applying records the objects as managed by the project.
git-manage "Analytics repo" flows="Load orders,Load customers" project=analytics path=projects/analytics
Promotion guide: development → test → production
One repository, one branch per stage of review, and an environment file per target. Development happens in a development tenant; test and production are only ever changed by deploys.
- Export from development to a branch. Build and test flows in the development tenant, then push them to a feature branch:
CLI
git-sync-out "Analytics repo" branch=feature/orders path=projects/analytics project=analytics flows="Load orders,Load customers" - Review and merge the pull request on your Git host. Each object is its own file, so the diff shows exactly what changed.
- Plan, then deploy to test from
mainwith the test environment. The plan lists every create, update, keep and delete before anything changes;plan-hashmakes the deploy refuse a plan that changed since you reviewed it:CLIgit-sync-in "Analytics repo" ref=main path=projects/analytics env=test dry-run=true git-sync-in "Analytics repo" ref=main path=projects/analytics env=test plan-hash=3f1a9c0d2e4b - Tag a release and deploy it to production with the production environment. A tag never moves, so production runs exactly what was tested:
CLI
git-sync-in "Analytics repo" ref=release-2026.11 path=projects/analytics env=prod dry-run=true
- Environment values that differ between test and production (hosts, databases, schedules on or off) go in
environments/<env>.json. Secrets stay in Etlworks: set them once per tenant withproject-env-setand they are applied on every deploy. See environments. - Deleting:
prune=truedeletes objects that were deployed by the project and are no longer in it (they go to the recycle bin). Nothing else is ever deleted. - From CI: run the same steps with the etlworks client (GitHub Actions, GitLab CI, Azure DevOps), or deploy automatically when a branch changes with deploy on push.
Git commands
All of them take the Git Repository connection's name or ID first. Commands that change the repository or the workspace ask for confirmation.
| Command | What it does |
|---|---|
git-manage | Puts flows under Git in one step (Manage in Git): flows=, project=, path=, branch=, all-types=true, takeover=true. |
git-sync-out | Exports flows, with what they use, as a project to a branch, commits and pushes. flows defaults to all; clean=true (the default with flows=all) removes files of objects no longer exported. |
git-sync-in | Deploys a project from a branch, tag or commit: plan and apply in one command. ref=, path=, env=, dry-run=true, plan-hash=, prune=true, collision options. |
git-refs | Branches, tags and the default branch. |
git-log | History: ref=, path=, count= (up to 500). |
git-status, git-diff | The workspace: branch, changed files, commits to push or pull, and uncommitted changes (path=, stat=true). |
git-pull | Brings the remote's commits into the workspace. Fast-forward only, nothing is merged; branch= switches first. |
git-commit | Commits all changes, or paths=. Nothing is pushed. |
git-push | Pushes committed work. Never forces: if the remote has new commits, run git-pull first. |
git-branch, git-tag | Create and push a branch (from=) or a tag (ref=). Never replace an existing one. |
git-reset | Discards uncommitted changes and unpushed commits in the workspace; the remote is untouched. Requires --force. |
A change made only in Git, and not deployed here, is reverted by the next git-sync-out from a tenant that doesn't have it (the commit keeps it in the Git history). Run git-sync-in first, or sync out to a feature branch and merge.
Run them in the CLI console, in a Run CLI Script flow, with the etlworks client (etlworks run "git-sync-in …"), or through the REST API. Full syntax and roles: command reference.
Code from Git
A flow can run SQL and scripts kept in the repository instead of pasted into the flow. Every code field of a flow has a Code from Git button: pick the connection, the branch or tag and the file, preview it, and Use this file. The field then shows From Git: analytics · sql/orders/merge.sql @ main.
Inside other code, use a reference (the Explorer's Copy reference copies one):
{{git:analytics:sql/orders/merge.sql@release-2026.11}}
-- {{git:<connection>:<path>[@<branch|tag|commit>]}}; without @, the connection's default branch
- One commit per run. Each run asks the host for the newest commit of each branch or tag once, and every file comes from that commit, nested and Composer steps included. A push a moment before the run is used.
- The run shows the commit. The run's details list each file with its branch and commit.
- Integration Agents need no Git: Etlworks puts the code into the agent's configuration, and the agent picks up a new commit on its next check-in.
- When Git can't be reached the run fails with the reason, unless the connection allows the last fetched commit. A missing file, branch or tag always fails.
- Limits: 5 MB per file, 50 files per run. References and macros inside a Git file are not expanded;
{variables}work as usual.
Run CLI Script flows
Run CLI Script (group Automation and Orchestration) is a flow type that runs CLI commands on the app, so deploys, exports and Git commands can be scheduled, chained in workflows, or started by webhooks. Only Administrators see it in the gallery and can create or edit these flows; for everyone else the editor is read-only.
| Field | Notes |
|---|---|
| Script | CLI commands separated by ; or new lines, up to 64,000 characters. |
| Timeout (seconds) | 30 to 3,600; default 900. A command still running at the deadline is cancelled. |
| Stop on first error | On by default. Off: every command runs, and the flow fails if the last one failed. |
| Allow changes | Off by default. Without it the script can only read. |
| Allow destructive and operational actions | Requires Allow changes. Allows deletes, imports, deploys (git-sync-in, project-apply), pushes, and running flows without a confirmation. |
| Git webhook secret | The shared secret for deploy on push. |
| HTTP listener | Optional: the listener a Git host's push webhook calls. |
- Variables: the flow's own variables; for a Git push
{git_branch},{git_tag},{git_ref},{git_commit}and{git_repository}; for an Etlworks webhook{webhook_event_id}and{webhook_event_type}. - Results: the log shows each command with its output (capped at 1 MB) and exit code. A parent workflow gets
CLI_EXIT_CODE,CLI_LAST_OUTPUTand any variables the script set. - Who it runs as: the user who started it, the schedule owner, the listener's authenticated caller, or the Administrator who saved the webhook. Grants never raise that user's own rights.
- It runs on the app, never on Integration Agents, including inside a parent workflow. A scheduled run refuses to start if the schedule was last changed by a non-Administrator.
Deploy on push
Deploy a branch the moment it changes: the Git host's webhook calls an Etlworks HTTP listener, which runs a Run CLI Script flow.
- HTTP listener connection: a URL such as
/git/push, method POST, Include request headers on, authentication JWT with the location Query Parameter (or Basic). - Run CLI Script flow with the listener as its HTTP listener, Allow changes and Allow destructive on, a Git webhook secret, and the script:
CLI
git-sync-in "Analytics repo" ref={git_branch} path=projects/analytics env=prod --force - Schedule the flow on the listener.
- Add the webhook on the Git host for push events, with content type JSON and the same secret:
- URL:
https://<your instance>/rest/v1/httplistener/git/push?Authorization=Bearer%20<API key>. - GitHub and Bitbucket sign the body with the secret (
X-Hub-Signature-256). - GitLab sends the secret as its token (
X-Gitlab-Token). - Azure DevOps has no signature: use Basic authentication on the listener instead.
- URL:
A wrong or missing signature fails the run before any command runs. The script gets {git_branch}, {git_tag}, {git_ref}, {git_commit} and {git_repository} from GitHub, GitLab, Bitbucket and Azure DevOps push events. They are set only when the name is plain (letters, digits and . _ - / + @); the request body, its query parameters and headers never become script variables, so nothing a pusher controls can add commands. A listener without authentication requires the webhook secret.
Etlworks webhooks that run a flow
Etlworks' own webhooks (event notifications) have a Run Simba or CLI flow option: when the webhook's events fire, the selected flow runs on the app, as the Administrator who saved the webhook. The flow gets {webhook_event_id} and {webhook_event_type}; the Git signature check doesn't apply. Events about Simba or CLI flows themselves are ignored, so a flow can't trigger itself.
Managed objects and drift
Every object a project deploys is managed in Git. Etlworks remembers where it came from and notices when someone changes it in Etlworks afterwards. That's drift: the next deploy overwrites those changes.
- Lists of flows, connections, formats, schedules and macros show a Git badge, with a dot when the object changed since its deploy, and a Git filter (managed, changed since deploy, not managed).
- Editors show where the object came from: project, repository, branch or tag, commit (linked to GitHub, GitLab, Bitbucket and Azure DevOps), folder, and who deployed it when. Saving or deleting a managed object warns that the next deploy overwrites or recreates it.
- Compare with Git version shows what differs between the object here and the deployed version. Credentials and canvas layout aren't compared.
- Revert to Git version puts the deployed version back, for that object only.
- Export changes to a branch (flows) pushes the changed flow to a new branch, to review and merge on the Git host instead of losing the change.
- Stop managing in Git keeps the object as it is; the next deploy treats it as not managed yet and matches it by name.
- Monitor → Git drift (administrators; shown once something is deployed from Git) lists every managed object by project, with what changed here.
project-drift changed=truegives the same list from the CLI or a scheduled check.
Git in the Explorer
Git Repository connections appear in the Explorer with two views:
- Remote: browse any branch, tag or commit, open and download files, and copy a file's path or reference.
- Local: Etlworks' own working copy of the repository. Edit and upload files, see what's changed against the last commit or the remote, then commit, push, pull, switch or create branches, or discard changes.
Step by step: Working with Git repositories in Explorer.
The support center guide, Data Integration as Code, walks through every dialog with screenshots and lists the exact error messages with fixes. A workflow that should work better? product@etlworks.com.