Project format
A project folder is the Git layout of an Etlworks project: a manifest and one readable file per object, built so that a pull request reads like code. The same folder deploys from Git, from a folder on the server, or as an upload from the client.
Folder layout
etlworks.json manifest: project name, objects, connections mode, collision policy
flows/load-orders.json one file per object
flows/load-orders/source-query.sql long code stored as files (.sql, .js, .py, .sh, .xsl, .txt)
connections/snowflake.json
formats/csv-default.json
macros/common-filters.json
schedules/nightly.json
environments/prod.json per-environment values (you write these; export never touches them)
A repository can hold one project or several, each in its own folder (for example projects/analytics), next to plain SQL and scripts that flows read with code from Git.
The manifest
etlworks.json describes the project. Export writes it; you rarely edit it by hand, except to set a default collision policy.
{
"formatVersion": 1,
"project": "analytics",
"connections": "nosecrets",
"collision": { "policy": "KeepAllReplaceFlowsAndMacros", "replace": ["formats"] },
"objects": [
{ "file": "flows/load-orders.json", "name": "Load orders", "type": "flow" },
{ "file": "connections/snowflake.json", "name": "Snowflake", "type": "connection" },
{ "file": "schedules/nightly.json", "name": "Nightly", "type": "schedule" }
]
}
| Field | Meaning |
|---|---|
formatVersion | The file format version. A file from a newer version is refused with a message to upgrade this instance or export again from the older one. |
project | The project name. Defaults to the folder name on export. |
connections | How credentials were exported: nosecrets, reference or encrypted (see credentials). |
secrets | With encrypted: the fingerprint and name of the key the credentials were encrypted with. |
collision | The project's default collision policy: a policy name, or an object with policy and replace, keep or fail lists of types. |
notes | Free text, including what the export reported but didn't include. |
objects | The authoritative list of the project's files. Files that aren't listed are ignored, never uploaded by the client, and reported by project-validate. |
Object files
Each object is one JSON file with its type, its name and its data:
{
"type": "flow",
"name": "Load orders",
"data": { … }
}
Inside data, anything that points to another object is a reference by name:
| Where | Written as |
|---|---|
| Connections of a transformation, named and loop connections | {"$ref": {"type": "connection", "name": "Snowflake"}} |
| Formats | {"$ref": {"type": "format", "name": "CSV default"}} |
| A nested flow step, a schedule's flow, Composer steps | {"$ref": {"type": "flow", "name": "Load orders"}} |
| Macros in code | {{macro:Open statuses}} |
| Code of 20 lines or more | {"$file": "flows/load-orders/source-query.sql"} |
- Names are identity. File names are slugs of object names, but the name inside the file is what counts. Renaming an object in Git means changing that name: the next deploy creates the object under the new name, and the old one stays on the target unless you prune.
- References resolve on deploy to the IDs on the target. A reference to an object that is neither in the project nor on the target fails validation.
- Code becomes files. Values of 20 lines or more move into separate files next to the object and are referenced with
{"$file": "…"}, so a changed query shows as a line diff. - Canvas layout is kept in one
layoutobject, so moving boxes on the canvas is a small diff. - No noise. No IDs, timestamps, owners, run state or export metadata. An export of unchanged objects is byte-identical, so it produces an empty diff. Line endings are LF.
The fastest way to learn the fields of a flow type is to export one: project-export in the CLI or Export → Readable JSON in the UI.
Credentials
How connection credentials are handled is the most important choice you make. Set it in the Export dialogs, in the manifest, with connections= in the CLI, or with --connections in the client.
| Mode | Written to the files | On deploy |
|---|---|---|
nosecrets (default) | Everything except password fields and secure schedule parameters | Non-secret fields update and the target keeps its own secrets. A new connection is created without them: set them once in the target, or as stored environment values. |
reference (flows only) | Only connection names and types; flows reference them | The target must already have a connection with that name and type, or the deploy stops before writing anything. |
encrypted | Password fields as encrypted envelopes (enc:v1:…) | Decrypted with the matching key. If the target has no matching key, those credentials aren't imported, existing secrets are kept, and warnings name the key. |
Our recommendation: keep secrets out of Git. Export with nosecrets and set credentials once per environment, in the target's connection or with project-env-set … secret=true.
Keys for encrypted credentials
- Keys live in Settings → Security → Export/Import Encryption: the RSA Private Key with its passphrase (the same key encrypts
.etldexports), or a Readable JSON Key, a passphrase of at least 32 characters, used when there is no private key. - Import uses whichever configured key matches the file's fingerprint. Exporting with
encryptedand no key configured fails with a message naming the setting. - Each envelope is bound to its object and field, so a value copied into another field won't decrypt. Envelopes are deterministic, so an unchanged export stays byte-identical.
- Shared servers (Etlworks Cloud, and any server with tenants): the export key is shared by every tenant, so tenant users can't export or import encrypted credentials. Super Admins and users of single-tenant installs can. Tenant users export with
nosecretsand use stored values.
Readable bundles
A readable bundle (.etlworks.json) is the whole project in one file: the same objects, with code files inlined. A folder and a bundle with the same objects are equivalent, and every deploy and import accepts both.
- In the UI, the Export dialogs offer File format: Readable JSON next to the encrypted
.etlddefault, and the same three connection modes. The Import dialog accepts.etldand.jsonfiles. - In the CLI,
export-flows,export-connectionsandexport-formatstakeformat=json, andproject-export … out=downloadreturns the project folder as a zip.
HTTP listener connections
HTTP listener connections that flows use are exported with the project. When a deploy creates a listener whose URL already exists on the server, the new one gets a dated URL (<url>_<MM-dd-yyyy_HH-mm-ss>) so the two never clash; an MLLP listener gets a different port instead.
The support center guide, Data Integration as Code, walks through every dialog with screenshots and lists the exact error messages with fixes. A workflow that should work better? product@etlworks.com.