Flows as code

Project format

A project folder is the Git layout of an Etlworks project: a manifest and one readable file per object, built so that a pull request reads like code. The same folder deploys from Git, from a folder on the server, or as an upload from the client.

Folder layout

Project folder
etlworks.json                      manifest: project name, objects, connections mode, collision policy
flows/load-orders.json             one file per object
flows/load-orders/source-query.sql long code stored as files (.sql, .js, .py, .sh, .xsl, .txt)
connections/snowflake.json
formats/csv-default.json
macros/common-filters.json
schedules/nightly.json
environments/prod.json             per-environment values (you write these; export never touches them)

A repository can hold one project or several, each in its own folder (for example projects/analytics), next to plain SQL and scripts that flows read with code from Git.

The manifest

etlworks.json describes the project. Export writes it; you rarely edit it by hand, except to set a default collision policy.

etlworks.json
{
  "formatVersion": 1,
  "project": "analytics",
  "connections": "nosecrets",
  "collision": { "policy": "KeepAllReplaceFlowsAndMacros", "replace": ["formats"] },
  "objects": [
    { "file": "flows/load-orders.json", "name": "Load orders", "type": "flow" },
    { "file": "connections/snowflake.json", "name": "Snowflake", "type": "connection" },
    { "file": "schedules/nightly.json", "name": "Nightly", "type": "schedule" }
  ]
}
FieldMeaning
formatVersionThe file format version. A file from a newer version is refused with a message to upgrade this instance or export again from the older one.
projectThe project name. Defaults to the folder name on export.
connectionsHow credentials were exported: nosecrets, reference or encrypted (see credentials).
secretsWith encrypted: the fingerprint and name of the key the credentials were encrypted with.
collisionThe project's default collision policy: a policy name, or an object with policy and replace, keep or fail lists of types.
notesFree text, including what the export reported but didn't include.
objectsThe authoritative list of the project's files. Files that aren't listed are ignored, never uploaded by the client, and reported by project-validate.

Object files

Each object is one JSON file with its type, its name and its data:

flows/load-orders.json
{
  "type": "flow",
  "name": "Load orders",
  "data": { … }
}

Inside data, anything that points to another object is a reference by name:

WhereWritten as
Connections of a transformation, named and loop connections{"$ref": {"type": "connection", "name": "Snowflake"}}
Formats{"$ref": {"type": "format", "name": "CSV default"}}
A nested flow step, a schedule's flow, Composer steps{"$ref": {"type": "flow", "name": "Load orders"}}
Macros in code{{macro:Open statuses}}
Code of 20 lines or more{"$file": "flows/load-orders/source-query.sql"}

The fastest way to learn the fields of a flow type is to export one: project-export in the CLI or Export → Readable JSON in the UI.

Credentials

How connection credentials are handled is the most important choice you make. Set it in the Export dialogs, in the manifest, with connections= in the CLI, or with --connections in the client.

ModeWritten to the filesOn deploy
nosecrets (default)Everything except password fields and secure schedule parametersNon-secret fields update and the target keeps its own secrets. A new connection is created without them: set them once in the target, or as stored environment values.
reference (flows only)Only connection names and types; flows reference themThe target must already have a connection with that name and type, or the deploy stops before writing anything.
encryptedPassword fields as encrypted envelopes (enc:v1:…)Decrypted with the matching key. If the target has no matching key, those credentials aren't imported, existing secrets are kept, and warnings name the key.

Our recommendation: keep secrets out of Git. Export with nosecrets and set credentials once per environment, in the target's connection or with project-env-set … secret=true.

Keys for encrypted credentials

Readable bundles

A readable bundle (.etlworks.json) is the whole project in one file: the same objects, with code files inlined. A folder and a bundle with the same objects are equivalent, and every deploy and import accepts both.

HTTP listener connections

HTTP listener connections that flows use are exported with the project. When a deploy creates a listener whose URL already exists on the server, the new one gets a dated URL (<url>_<MM-dd-yyyy_HH-mm-ss>) so the two never clash; an MLLP listener gets a different port instead.


More detail

The support center guide, Data Integration as Code, walks through every dialog with screenshots and lists the exact error messages with fixes. A workflow that should work better? product@etlworks.com.