Git in Etlworks

Keep flows, connections, formats, macros and schedules in a Git repository, review changes as pull requests, and promote them from development to production. Etlworks talks to Git directly: no client is needed on the server, and the etlworks client does the same from a laptop or CI. Available in Etlworks 10.1.0 and newer. The full guide, with screenshots, every dialog and troubleshooting, is Data Integration as Code in the support center.

Git Repository connection

Everything starts with a Git Repository connection (Connections → Add → Developer Tools). It works with GitHub, GitLab, Bitbucket, Azure DevOps and any server that speaks Git over HTTPS or SSH.

Test Connection lists the branches and tags. The server needs git (and ssh for SSH URLs) installed; Test Connection says so when it is missing.

Repository layout

A repository can hold one project or several, each in its own folder. A project folder has one readable JSON file per object; long SQL and scripts are separate files, so reviews read like code.

projects/
  analytics/
    etlworks.json                    manifest: project name, objects, collision policy
    flows/load-orders.json
    flows/load-orders/source-sql.sql
    connections/snowflake.json       without passwords by default
    schedules/nightly.json
    environments/test.json           per-environment overrides
    environments/prod.json
sql/
  orders/merge.sql                   code that flows read at run time (see Code from Git)

Objects are matched by name and type, never by ID, so the same folder deploys to any tenant or instance. The client page describes the folder, credentials and environment files in detail.

Start managing flows in Git

In the flows list, select the flows, then Bulk actions → Manage in Git…. Choose the Git connection, the branch, the project name and folder. Etlworks exports the flows with everything they use, pushes them, and records them as managed in Git; nothing changes in Etlworks. By default flows and macros are managed; tick Also manage connections, formats and schedules to include those (connections are exported without passwords). The same from the CLI:

git-manage "Analytics repo" flows="Load orders,Load customers" project=analytics path=projects/analytics

Promotion guide: development → test → production

One repository, one branch per stage of review, and an environment file per target. Development happens in a development tenant; test and production are only ever changed by deploys.

1. Export from development to a branch. Build and test flows in the development tenant, then push them to a feature branch:
   CLICopy

   git-sync-out "Analytics repo" branch=feature/orders path=projects/analytics project=analytics flows="Load orders,Load customers"

2. Review and merge the pull request on your Git host. Each object is its own file, so the diff shows exactly what changed.
3. Plan, then deploy to test from main with the test environment. The plan lists every create, update, keep and delete before anything changes; plan-hash makes the deploy refuse a plan that changed since you reviewed it:
   CLICopy

   git-sync-in "Analytics repo" ref=main path=projects/analytics env=test dry-run=true
   git-sync-in "Analytics repo" ref=main path=projects/analytics env=test plan-hash=3f1a9c0d2e4b

4. Tag a release and deploy it to production with the production environment. A tag never moves, so production runs exactly what was tested:
   CLICopy

   git-sync-in "Analytics repo" ref=release-2026.11 path=projects/analytics env=prod dry-run=true

• Environment values that differ between test and production (hosts, databases, schedules on or off) go in environments/<env>.json. Secrets stay in Etlworks: set them once per tenant with project-env-set and they are applied on every deploy.
• Deleting: prune=true deletes objects that were deployed by the project and are no longer in it (they go to the recycle bin). Nothing else is ever deleted.
• From CI: run the same steps with the etlworks client (GitHub Actions, GitLab CI, Azure DevOps), or deploy automatically when a branch changes with deploy on push.

Git commands

All of them take the Git Repository connection's name or ID first. Commands that change something ask for confirmation.

Run them in the CLI console, in a Run CLI Script flow, with the etlworks client (etlworks run "git-sync-in …"), or through the REST API.

Code from Git

A flow can run SQL and scripts kept in the repository instead of pasted into the flow. Every code field of a flow has a Code from Git button: pick the connection, the branch or tag and the file, preview it, and Use this file. The field then shows From Git: analytics · sql/orders/merge.sql @ main.

Inside other code, use a reference (the Explorer's Copy reference copies one):

{{git:analytics:sql/orders/merge.sql@release-2026.11}}
-- {{git:<connection>:<path>[@<branch|tag|commit>]}}; without @, the connection's default branch

• One commit per run. Each run asks the host for the newest commit of each branch or tag once, and every file comes from that commit, nested and Composer steps included. A push a moment before the run is used.
• The run shows the commit. The run's details list each file with its branch and commit.
• Integration Agents need no Git: Etlworks puts the code into the agent's configuration, and the agent picks up a new commit on its next check-in.
• When Git can't be reached the run fails with the reason, unless the connection allows the last fetched commit. A missing file, branch or tag always fails.
• Limits: 5 MB per file, 50 files per run. References and macros inside a Git file are not expanded; {variables} work as usual.

Deploy on push

Deploy a branch the moment it changes: the Git host's webhook calls an Etlworks HTTP listener, which runs a Run CLI Script flow.

1. HTTP listener connection: a URL such as /git/push, method POST, Include request headers on, authentication JWT with the location Query Parameter (or Basic).
2. Run CLI Script flow with the listener as its HTTP listener, Allow changes and Allow destructive on, a Git webhook secret, and the script:
   CLICopy

   git-sync-in "Analytics repo" ref={git_branch} path=projects/analytics env=prod --force

3. Schedule the flow on the listener.
4. Add the webhook on the Git host for push events, with content type JSON and the same secret:
   • URL: https://<your instance>/rest/v1/httplistener/git/push?Authorization=Bearer%20<API key>.
   • GitHub and Bitbucket sign the body with the secret (X-Hub-Signature-256).
   • GitLab sends the secret as its token (X-Gitlab-Token).
   • Azure DevOps has no signature: use Basic authentication on the listener instead.

A wrong or missing signature fails the run before any command runs. The script gets {git_branch}, {git_tag}, {git_ref}, {git_commit} and {git_repository} from GitHub, GitLab, Bitbucket and Azure DevOps push events. They are set only when the name is plain (letters, digits and . _ - / + @); the request body, its query parameters and headers never become script variables, so nothing a pusher controls can add commands. A listener without authentication requires the webhook secret. Only an administrator can create Run CLI Script flows, and they never run on Integration Agents.

Managed objects and drift

Every object a project deploys is managed in Git. Etlworks remembers where it came from and notices when someone changes it in Etlworks afterwards. That's drift: the next deploy overwrites those changes.

• Lists of flows, connections, formats, schedules and macros show a Git badge, with a dot when the object changed since its deploy, and a Git filter (managed, changed since deploy, not managed).
• Editors show where the object came from: project, repository, branch or tag, commit (linked to GitHub, GitLab, Bitbucket and Azure DevOps), folder, and who deployed it when. Saving or deleting a managed object warns that the next deploy overwrites or recreates it.
• Compare with Git version shows what differs between the object here and the deployed version. Credentials and canvas layout aren't compared.
• Revert to Git version puts the deployed version back, for that object only.
• Export changes to a branch (flows) pushes the changed flow to a new branch, to review and merge on the Git host instead of losing the change.
• Stop managing in Git keeps the object as it is; the next deploy treats it as not managed yet and matches it by name.
• Monitor → Git drift (administrators; shown once something is deployed from Git) lists every managed object by project, with what changed here. project-drift changed=true gives the same list from the CLI or a scheduled check.

Git in the Explorer

Git Repository connections appear in the Explorer with two views:

• Remote: browse any branch, tag or commit, open and download files, and copy a file's path or reference.
• Local: Etlworks' own working copy of the repository. Edit and upload files, see what's changed against the last commit or the remote, then commit, push, pull, switch or create branches, or discard changes.

Step by step: Working with Git repositories in Explorer.