Etlworks client

The etlworks command keeps flows, connections, formats, macros and schedules in Git and deploys them from your laptop or a CI pipeline. Export a project to a folder, review a plan of what will change, apply it. The client only moves commands and files; all the work runs on the Etlworks server, under the permissions of the API key you sign in with. To deploy straight from a Git repository, run code from Git, or deploy on push, see Git in Etlworks. The full guide is Data Integration as Code in the support center.

Install

Requirements: Java 17 or later (from JAVA_HOME or the PATH) and an Etlworks instance on release 10.1.0 or later, Cloud or on-prem.

1. Download the client (one zip for macOS, Linux and Windows). Check it against the SHA-256 checksum. A specific release: /installers/download/client?version=10.1.0.
2. Unzip it anywhere and add the folder to your PATH. On Windows, the command is etlworks.cmd.
3. Run etlworks version. The launcher tells you if Java 17 or later is missing.

curl -fsSL -o etlworks-client.zip https://etlworks.com/installers/download/client
unzip -q etlworks-client.zip -d ~/tools
export PATH="$(ls -d ~/tools/etlworks-client-* | tail -1):$PATH"
etlworks version

Sign in

The client signs in with an Etlworks API key. Create one in Etlworks under Preferences → Security (your user menu); an admin can create one for another user under Settings → Users. The client acts with that user's role and tenant.

etlworks login --url https://app.etlworks.com --api-key YOUR_API_KEY
etlworks login --profile prod --url https://etl.example.com --api-key PROD_KEY
etlworks profiles            # lists profiles; keys are shown masked
etlworks run "whoami" --profile prod

• login checks the URL and the key against the server before saving, and refuses a key that doesn't work.
• Profiles are saved in ~/.etlworks/config.json, readable only by you (0600 on macOS and Linux, an owner-only ACL on Windows). Set ETLWORKS_HOME to keep it somewhere else.
• In CI, skip login: set ETLWORKS_URL and ETLWORKS_API_KEY from the pipeline's secret store. ETLWORKS_PROFILE picks a saved profile. Environment variables win over the file.
• --insecure accepts a self-signed TLS certificate (on-prem test instances only). It is off by default.

The project folder

A project is a folder with one readable JSON file per object, ready for Git. Long SQL and scripts are stored as separate files, so diffs and reviews read like code.

etlworks.json                    manifest: project name, objects, collision policy
flows/load-orders.json
flows/load-orders/source-sql.sql code stored as files
connections/snowflake.json
formats/csv-default.json
macros/common-filters.json
schedules/nightly.json
environments/prod.json           per-environment overrides (you write these)

• Objects are matched by name and type, never by ID, so the same folder deploys to a different tenant or instance.
• Only the files listed in etlworks.json are used, plus environments/. Anything else in the folder is ignored and never uploaded.
• Credentials: --connections nosecrets (the default) exports connections without passwords; the target keeps its own. reference exports only the connection names, so the target must already have them. encrypted stores credentials encrypted for Etlworks instances that share your export key (Settings → Security → Export/Import Encryption); only those instances can read them. On a server shared by several tenants (such as Etlworks Cloud), encrypted credentials are available to Super Admins only, because the key is shared; tenant users export without secrets and keep credentials in each tenant.

Export, plan, apply

# 1. Export from the development tenant into a Git working copy
etlworks project export ./analytics --project analytics --flows "Load orders,Load customers" --profile dev
git add analytics && git commit -m "Load orders: new filter"

# 2. See what would change in production (exit code 2 = there are changes)
etlworks project plan ./analytics --env prod --profile prod --out plan.json

# 3. Deploy exactly what was reviewed
etlworks project apply ./analytics --env prod --profile prod --plan plan.json --message "Release 42"

• Export writes the files listed in the new manifest. --clean also removes files of objects that are no longer exported. environments/ is never touched. --project defaults to the folder name.
• Plan shows each object as + create, ~ update, = unchanged, keep or - delete, with field-level diffs and line diffs for SQL and scripts. Credentials appear only as "credentials changed". --verbose also lists unchanged objects.
• Apply plans again on the server, asks for confirmation in a terminal (without one, --yes is required), then deploys in one transaction. Unchanged objects aren't rewritten. With --plan plan.json, apply refuses if anything changed since the plan was reviewed. Ctrl-C cancels the deploy on the server.
• Prune: --prune deletes objects this project deployed earlier that are no longer in the folder. They go to the recycle bin. Prune needs an Administrator.
• Drift: etlworks project drift --project analytics lists deployed objects someone changed in the UI since the last deploy.

What happens to objects that already exist

The default policy, KeepAllReplaceFlowsAndMacros, replaces flows and macros from the folder and keeps the target's existing connections, formats and schedules. New objects are always created. Change it per run, or set it in etlworks.json or the environment file:

--collision Replace|Keep|KeepAllReplaceFlowsAndMacros
--replace formats,schedules      # also replace these types
--keep macros                    # keep these types as they are on the target
--fail connections               # stop if any of these already exist

Environments

One folder deploys to every environment. environments/<env>.json holds the values that differ, selected with --env. Overrides apply before planning, so the plan shows the final values.

{
  "connections": { "Snowflake": { "host": "acme.snowflakecomputing.com", "warehouse": "PROD_WH" } },
  "flows":       { "Load orders": { "variables": { "batch_size": "50000" } } },
  "schedules":   { "Nightly": { "enabled": true, "cron": "0 2 * * *" } }
}

• Secrets don't belong in Git. Store them on the server instead: etlworks project env-set analytics connection Snowflake password --value '…' --secret. Stored values win over the file and are kept encrypted. env-get lists them (secrets masked) and env-unset removes one.
• An override on an object the collision policy keeps is not applied, and the plan warns about it.

Commands

Global options: --profile NAME, --json (machine-readable output), --quiet, --insecure. Results go to stdout; progress and messages go to stderr, so stdout stays clean for pipes. etlworks --help prints the full syntax.

Exit codes

GitHub Actions

Plan on every pull request, deploy on merge to main. Store the URL and API key as repository secrets.

name: etlworks
on:
  pull_request:
  push:
    branches: [main]

env:
  ETLWORKS_URL: ${{ secrets.ETLWORKS_URL }}
  ETLWORKS_API_KEY: ${{ secrets.ETLWORKS_API_KEY }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: '17' }
      - name: Install the etlworks client
        run: |
          curl -fsSL -o client.zip https://etlworks.com/installers/download/client
          unzip -q client.zip -d "$RUNNER_TEMP"
          echo "$(ls -d "$RUNNER_TEMP"/etlworks-client-*)" >> "$GITHUB_PATH"
      - name: Validate and plan
        run: |
          etlworks project validate ./analytics --env prod
          etlworks project plan ./analytics --env prod --out plan.json || [ $? -eq 2 ]
      - name: Deploy
        if: github.event_name == 'push'
        run: etlworks project apply ./analytics --env prod --plan plan.json --yes --message "${{ github.sha }}"

|| [ $? -eq 2 ] lets the job continue when the plan has changes. To block a merge until someone reviews production changes, drop it, or post plan.json as a pull request comment.

GitLab CI

image: eclipse-temurin:17-jdk

# Set ETLWORKS_URL and ETLWORKS_API_KEY as masked CI/CD variables.

before_script:
  - curl -fsSL -o client.zip https://etlworks.com/installers/download/client
  - apt-get update -qq && apt-get install -y -qq unzip >/dev/null
  - unzip -q client.zip -d /opt
  - export PATH="$(ls -d /opt/etlworks-client-*):$PATH"

plan:
  stage: test
  script:
    - etlworks project plan ./analytics --env prod --out plan.json || [ $? -eq 2 ]
  artifacts: { paths: [plan.json] }

deploy:
  stage: deploy
  rules: [{ if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' }]
  script:
    - etlworks project apply ./analytics --env prod --plan plan.json --yes --message "$CI_COMMIT_SHA"

Azure DevOps

trigger: [main]
pool: { vmImage: ubuntu-latest }

steps:
  - task: JavaToolInstaller@0
    inputs: { versionSpec: '17', jdkArchitectureOption: x64, jdkSourceOption: PreInstalled }
  - script: |
      curl -fsSL -o client.zip https://etlworks.com/installers/download/client
      unzip -q client.zip -d "$(Agent.TempDirectory)"
      echo "##vso[task.prependpath]$(ls -d "$(Agent.TempDirectory)"/etlworks-client-*)"
    displayName: Install the etlworks client
  - script: |
      etlworks project plan ./analytics --env prod --out plan.json || [ $? -eq 2 ]
      etlworks project apply ./analytics --env prod --plan plan.json --yes --message "$(Build.SourceVersion)"
    displayName: Plan and deploy
    env:
      ETLWORKS_URL: $(ETLWORKS_URL)
      ETLWORKS_API_KEY: $(ETLWORKS_API_KEY)   # a secret pipeline variable