Environments and deploy
Deploying a project is always the same three steps, whatever the source: validate, plan, apply. Environments supply the values that differ between targets, collision policies decide what happens to objects that already exist, and drift tracking shows what changed on the target since the last deploy.
Sources
The project-* commands take a source as their first argument:
| Source | Example |
|---|---|
| A folder under the server's app data | project-plan projects/analytics |
| A bundle file or an uploaded zip | project-plan upload:<id> (what the client sends) |
| Git, read directly from the remote | project-plan git:"Analytics repo"@main path=projects/analytics |
git-sync-in is the Git shortcut: plan and apply in one command, with dry-run=true for the plan only.
Validate
project-validate checks the project itself (files, references, environment overrides, credentials) and its conflicts with this environment, without comparing or changing anything. Unknown keys in an environment file fail, and the message lists the valid ones.
Plan
project-plan compares the project with the target and prints one line per object. It changes nothing.
project-plan git:"Analytics repo"@main path=projects/analytics env=test
| Action | Meaning |
|---|---|
create | The object doesn't exist on the target and will be created. |
update | The object exists and will be updated in place: its ID, history and permissions stay. |
adopt | An existing object that no project manages yet becomes managed by this one. |
update (drift) | The object changed here since its last deploy; the deploy overwrites that change. |
keep | The collision policy leaves the existing object alone. |
unchanged | Nothing differs; the object won't be written. |
reference | A reference-only connection, which must exist on the target. |
delete | Prune: the project deployed this object earlier and no longer contains it. |
error | Something blocks the deploy, such as a duplicate name or an object owned by another project. |
- Diffs show as JSON paths with old and new values, and as line diffs for code files. Credentials show only as "credentials changed", never as values.
- The effective collision policy is printed at the top of every plan, for example Collision: Replace: flows, macros. Keep: formats, schedules, connections.
- The plan ends with a plan hash that identifies exactly this plan.
verbose=truealso lists unchanged objects;json=trueprints machine-readable output.
Apply
project-apply git:"Analytics repo"@main path=projects/analytics env=test plan-hash=<hash> message="Release 42"
- Plans again under a lock. One deploy per tenant and project at a time; a second waits up to 60 seconds, then fails.
- With
plan-hash, apply refuses if anything changed since the plan was reviewed, and nothing is deployed. - One transaction. Unchanged objects aren't written: no new revision, no webhook event. When nothing changed at all, apply reports Up to date: nothing deployed.
- Schedules created by a deploy start disabled unless the environment sets
"enabled": true. An existing schedule keeps its enabled state unless the environment sets it. - Revision message: every written object gets Deployed from <project> @ <branch or tag> (<commit>) "<commit subject>" by <user>: <message>, or Deployed from <project> by <user> when the source isn't Git.
Collision policies
A collision is an object in the project whose name and type already exist on the target. Missing objects are always created, under every policy.
| Policy | Flows | Macros | Formats | Schedules | Connections |
|---|---|---|---|---|---|
KeepAllReplaceFlowsAndMacros (Keep All, Replace Flows and Macros) | replace | replace | keep | keep | keep |
Replace | replace | replace | replace | replace | replace* |
Keep | keep | keep | keep | keep | keep |
* A replaced connection still follows its export mode: exported without secrets, it keeps the target's passwords.
- Modifiers adjust the policy per type:
replace=,keep=andfail=with a list offlows,macros,formats,schedules,connections. Each type may appear in one modifier only.failmakes the collision an error, and nothing is written. - Default for project deploys (
project-apply,git-sync-in, the client) and readable-bundle imports from the CLI:KeepAllReplaceFlowsAndMacros. The UI Import dialog has no default; you choose. - Precedence: command options, then the environment file's
collision, then the manifest's, then the default. - Add, which creates dated copies, is refused for project deploys, because applying again would create new copies every time.
- Kept objects don't get environment values, and the plan warns about it. Use
replace=connections, for example, when the environment's connection values should land.
git-sync-in "Analytics repo" ref=main path=projects/analytics collision=Keep replace=flows,macros
Environment files
An environment supplies the values that differ between targets, so one project deploys everywhere. environments/<env>.json in the project folder is chosen with env=<env> in the CLI or --env <env> in the client. Export never touches these files; you write them.
{
"collision": { "policy": "KeepAllReplaceFlowsAndMacros", "replace": ["connections"] },
"connections": { "Snowflake": { "host": "acme.snowflakecomputing.com", "warehouse": "PROD_WH" } },
"formats": { "CSV default": { "properties": { "delimiter": ";" } } },
"flows": { "Load orders": { "variables": { "batch_size": "50000" }, "properties": { } } },
"schedules": { "Nightly": { "enabled": true, "cron": "0 2 * * *", "parameters": { "region": "us" } } }
}
- Connection and format keys are the connector's property keys; the
field.connection.orfield.format.prefix is optional. - Flows take
variablesandproperties. Schedules takeenabled,cronandparameters. - Unknown keys fail validation, and the message lists the valid ones.
- Values apply after the project is read and before planning, so the plan always shows the final values.
project-env-initwrites a template listing the project's objects. Entries left empty change nothing.
Stored values and secrets
Each environment can also hold values stored in Etlworks itself, which is where secrets belong:
project-env-set analytics connection Snowflake password value=<password> secret=true
project-env-get analytics
project-env-unset analytics connection Snowflake password
- Stored values win over the environment file.
- Values set with
secret=trueare stored encrypted and always shown as******. - Setting and removing stored values needs Tenant Admin; listing them (secrets hidden) needs Editor.
project-encrypt-valueprints an encrypted value (enc:v1:…) to paste into an environment file instead of plain text. Like other encrypted-credential features, it isn't available to tenant users on shared servers.
Example. The project has a connection "Snowflake". environments/dev.json points it at the development account and environments/prod.json at production. The password is never in Git: type it into the connection in development, and run project-env-set … secret=true once in production. Every later deploy reuses it.
Prune
prune=true deletes objects that this project deployed earlier and no longer contains. It needs an Administrator. Pruned objects go to the Recycle Bin, so they can be restored. Prune never touches objects the project didn't deploy, and it is blocked when something outside the prune set still uses the object.
Ownership and takeover
Every deployed object is recorded as managed by its project, with the source, branch or tag, commit, folder, environment, who deployed it, and a hash of the deployed content. Deploying an object that belongs to another project is an error; takeover=true (Administrator) moves it to this project.
Drift
Drift is a change made directly in Etlworks to a managed object after its last deploy. Canvas layout and credentials don't count.
project-drift project=analytics changed=true
project-compare flow "Load orders"
project-revert flow "Load orders"
project-drift lists managed objects and whether each changed here; project-compare shows how one differs from its deployed version; project-revert re-applies the deployed version of that one object. The same information is in the UI: Git badges in the lists, the source panel in each editor, and Monitor → Git drift. See managed objects and drift.
Project commands
| Command | What it does |
|---|---|
project-export | Exports a project to a folder, or out=download for a zip. |
project-validate | Checks files, references, environment overrides and credentials. Changes nothing. |
project-plan | Shows what a deploy would do, object by object. Changes nothing. |
project-apply | Deploys a project in one transaction. |
project-drift | Managed objects and whether each changed here since its deploy. |
project-compare, project-revert | Compare one managed object with its deployed version, or put that version back. |
project-env-init | Writes an environment file template. |
project-env-set, project-env-get, project-env-unset | Stored environment values. |
project-encrypt-value | Prints an encrypted value for an environment file. |
Full syntax, options and roles: command reference.
The support center guide, Data Integration as Code, walks through every dialog with screenshots and lists the exact error messages with fixes. A workflow that should work better? product@etlworks.com.