Flows as code

Environments and deploy

Deploying a project is always the same three steps, whatever the source: validate, plan, apply. Environments supply the values that differ between targets, collision policies decide what happens to objects that already exist, and drift tracking shows what changed on the target since the last deploy.

Sources

The project-* commands take a source as their first argument:

SourceExample
A folder under the server's app dataproject-plan projects/analytics
A bundle file or an uploaded zipproject-plan upload:<id> (what the client sends)
Git, read directly from the remoteproject-plan git:"Analytics repo"@main path=projects/analytics

git-sync-in is the Git shortcut: plan and apply in one command, with dry-run=true for the plan only.

Validate

project-validate checks the project itself (files, references, environment overrides, credentials) and its conflicts with this environment, without comparing or changing anything. Unknown keys in an environment file fail, and the message lists the valid ones.

Plan

project-plan compares the project with the target and prints one line per object. It changes nothing.

CLI
project-plan git:"Analytics repo"@main path=projects/analytics env=test
ActionMeaning
createThe object doesn't exist on the target and will be created.
updateThe object exists and will be updated in place: its ID, history and permissions stay.
adoptAn existing object that no project manages yet becomes managed by this one.
update (drift)The object changed here since its last deploy; the deploy overwrites that change.
keepThe collision policy leaves the existing object alone.
unchangedNothing differs; the object won't be written.
referenceA reference-only connection, which must exist on the target.
deletePrune: the project deployed this object earlier and no longer contains it.
errorSomething blocks the deploy, such as a duplicate name or an object owned by another project.

Apply

CLI
project-apply git:"Analytics repo"@main path=projects/analytics env=test plan-hash=<hash> message="Release 42"

Collision policies

A collision is an object in the project whose name and type already exist on the target. Missing objects are always created, under every policy.

PolicyFlowsMacrosFormatsSchedulesConnections
KeepAllReplaceFlowsAndMacros (Keep All, Replace Flows and Macros)replacereplacekeepkeepkeep
Replacereplacereplacereplacereplacereplace*
Keepkeepkeepkeepkeepkeep

* A replaced connection still follows its export mode: exported without secrets, it keeps the target's passwords.

CLI
git-sync-in "Analytics repo" ref=main path=projects/analytics collision=Keep replace=flows,macros

Environment files

An environment supplies the values that differ between targets, so one project deploys everywhere. environments/<env>.json in the project folder is chosen with env=<env> in the CLI or --env <env> in the client. Export never touches these files; you write them.

environments/prod.json
{
  "collision":   { "policy": "KeepAllReplaceFlowsAndMacros", "replace": ["connections"] },
  "connections": { "Snowflake": { "host": "acme.snowflakecomputing.com", "warehouse": "PROD_WH" } },
  "formats":     { "CSV default": { "properties": { "delimiter": ";" } } },
  "flows":       { "Load orders": { "variables": { "batch_size": "50000" }, "properties": { } } },
  "schedules":   { "Nightly": { "enabled": true, "cron": "0 2 * * *", "parameters": { "region": "us" } } }
}

Stored values and secrets

Each environment can also hold values stored in Etlworks itself, which is where secrets belong:

CLI
project-env-set analytics connection Snowflake password value=<password> secret=true
project-env-get analytics
project-env-unset analytics connection Snowflake password

Example. The project has a connection "Snowflake". environments/dev.json points it at the development account and environments/prod.json at production. The password is never in Git: type it into the connection in development, and run project-env-set … secret=true once in production. Every later deploy reuses it.

Prune

prune=true deletes objects that this project deployed earlier and no longer contains. It needs an Administrator. Pruned objects go to the Recycle Bin, so they can be restored. Prune never touches objects the project didn't deploy, and it is blocked when something outside the prune set still uses the object.

Ownership and takeover

Every deployed object is recorded as managed by its project, with the source, branch or tag, commit, folder, environment, who deployed it, and a hash of the deployed content. Deploying an object that belongs to another project is an error; takeover=true (Administrator) moves it to this project.

Drift

Drift is a change made directly in Etlworks to a managed object after its last deploy. Canvas layout and credentials don't count.

CLI
project-drift project=analytics changed=true
project-compare flow "Load orders"
project-revert flow "Load orders"

project-drift lists managed objects and whether each changed here; project-compare shows how one differs from its deployed version; project-revert re-applies the deployed version of that one object. The same information is in the UI: Git badges in the lists, the source panel in each editor, and Monitor → Git drift. See managed objects and drift.

Project commands

CommandWhat it does
project-exportExports a project to a folder, or out=download for a zip.
project-validateChecks files, references, environment overrides and credentials. Changes nothing.
project-planShows what a deploy would do, object by object. Changes nothing.
project-applyDeploys a project in one transaction.
project-driftManaged objects and whether each changed here since its deploy.
project-compare, project-revertCompare one managed object with its deployed version, or put that version back.
project-env-initWrites an environment file template.
project-env-set, project-env-get, project-env-unsetStored environment values.
project-encrypt-valuePrints an encrypted value for an environment file.

Full syntax, options and roles: command reference.


More detail

The support center guide, Data Integration as Code, walks through every dialog with screenshots and lists the exact error messages with fixes. A workflow that should work better? product@etlworks.com.