Automation examples

Real recipes used in production — monitoring & troubleshooting, flow ops, schedule lifecycle, multi-tenant tag cleanups, agent fleet upgrades, config-as-code workflows. All examples run identically in the in-UI Command Editor and via POST /v1/cli.

Monitoring & troubleshooting

Find long-running flows

Retrieve all running flows and filter for those running longer than one hour. Useful for detecting stuck flows, deadlocks, or slow external systems before they cause downstream failures.

list-running-flows
  select flowId, flowName, tenantId, date(started) as started
  where started < ts("now - 1h");

Get recent flow failures

Failed executions in the last 2 days. Quickly diagnose without opening UI logs in large multi-flow environments.

Across all tenants

flow-executions-aggregated tenants=all offset=1 includeMetrics=true
  select flowId, eventName, tenantId, status, user,
         date(started) started, date(ended) ended, exception
  from executions
  where status != "success"
  order by auditId desc;

Single / current tenant

flow-executions offset=1 includeMetrics=true
  select flowId, eventName, tenantId, status, user,
         date(started) started, date(ended) ended, exception
  from executions
  where status != "success"
  order by auditId desc;

Investigate flows with high data volume

Flows that read or processed the most data in the last 24 hours. Detects performance hotspots and heavy workflows that may need partitioning or rescheduling.

Across all tenants

flow-executions-aggregated tenants=all offset=1 includeMetrics=true
  select flowId, eventName, tenantId, status, user,
         date(started) started, date(ended) ended, records
  from executions
  order by records desc
  limit 10;

Single tenant (date required)

flow-executions date=2026-05-09 includeMetrics=true
  select flowId, eventName, tenantId, status, user,
         date(started) started, date(ended) ended, records
  from executions
  order by records desc
  limit 10;

Inspect a problematic flow for structural issues

Static inspection of a flow definition. Surfaces anti-patterns: missing formats, incomplete mappings, inefficient configurations.

inspect-flow 123;

Returns a JSON severity report with actionable suggestions:

{
  "severity": "CRITICAL",
  "flowIssues": [{
    "flowId": "345",
    "flowName": "Copy into S3 using REST",
    "severity": "CRITICAL",
    "issues": [{
      "severity": "CRITICAL",
      "issueType": "PERFORMANCE",
      "description": "Streaming is disabled for the transformation between flat datasets",
      "why": "When streaming is disabled, the entire source dataset is loaded into memory before being transformed and loaded into the destination, causing significant performance issues, especially with large datasets.",
      "suggestion": "Enable \"Stream Data\" to optimize performance"
    }]
  }]
}

Check system health

Node-level status, memory, thread pools, service availability. Health snapshot for cluster-wide automation and monitoring.

health;

Flow execution automation

Run a flow and wait for completion

Trigger by id synchronously and wait for the result. CI/CD pipelines and chained workflows requiring deterministic execution.

run-flow 123 sync=true;

Run a flow by name (multi-tenant)

Execute using a flow's name and an optional tenant override. Ideal for shared environments where flow IDs differ across tenants.

run-flow-by-name 'load data' tenant=test sync=false;

Stop a running flow

Attempt a graceful stop. Automated safeguard scripts that prevent uncontrolled resource consumption.

stop-flow 123 --force;

Flow management

Bulk-delete flows matching a name pattern

Retrieve all flows whose names match a substring, then delete them with their schedules. Use with caution — --force skips confirmation.

list-flows where name like '%test cdc%' | capture flows=stdout;
for-each (f in flows) {
  delete-flow {f.id} deleteSchedules=true --force;
};

Retrieve a flow configuration

Returns the flow object by id. Auditing and configuration-as-code workflows.

get-flow 123;

Backup matching flows to disk

Export every flow whose name matches a substring as a JSON file under the tenant's data folder. Auditing, backup, migration, and config-as-code workflows. Subfolders are auto-created.

list-flows --silent where name like "%script%" | capture flows=stdout;
for-each (f in flows) {
  get-flow {f.id} >> backup/{f.name}.json;
};

Identify where a flow is referenced

Schedules, agents, and other flows that depend on a given flow. Critical before deleting or modifying.

flow-usage 123;

Schedule management

List enabled schedules

Filter the schedule list by the enabled flag. Useful when auditing schedules after major deployments or tenant migration.

All tenants

list-schedules scope=all where enabled=true;

Current tenant

list-schedules where enabled=true;

Temporarily disable, then re-enable a schedule

// Pause for maintenance
disable-schedule 123 --force;

// Resume
enable-schedule 123 --force;

Bulk-disable schedules matching a name pattern

Find every enabled schedule whose name contains a substring, then disable each one. Bulk-disable related schedules during maintenance or migration.

list-schedules scope=all where name like "%your_substring%" and enabled = true
  | capture schedules=stdout;
for-each (s in schedules) {
  disable-schedule {s.id} --force;
};

Compare schedule versions

Diff between current and previous versions. Debug unexpected schedule behavior or verify config drift.

schedule-history-diff 123 current last;

Connections & data

Test a connection

Run a connection test using the Etlworks engine or a specific agent. Ensure connectivity before running flows that depend on it.

test-connection 123;

Find where a connection is used

List flows referencing a connection. Prevents accidental breaking changes when altering or replacing connections.

find-connection-usage 123;

Retrieve a connection configuration

Returns the connection object by id. Auditing and config-as-code workflows.

get-connection 123;

Build a default connection from a descriptor

Creates a connection template based on the descriptor type. Accelerates provisioning in infrastructure-as-code workflows.

build-default-connection 'connection.cloud.s3.sdk';

Retrieve raw data behind a connection

Returns the raw, untransformed data (file, object, API output). Validate, debug mappings, verify permissions.

data-get-raw connectionId=38 formatId=16 object=test.csv;

Run SQL directly against a connection

Execute SQL or extraction logic on a connection. Validate schemas, inspect anomalies, perform spot checks.

data-run-sql connectionId=123 sql="select * from orders";

Inspect inbound messages

Filter messages (HTTP requests to user-defined APIs / listener payloads) for size, format, status, or substring. Useful in systems that rely heavily on listeners.

// Find recent payloads matching a pattern
list-messages-filtered message=Notification;

// Pull the full body of one message
get-message 24a6e77b-c05f-45ed-b1e8-908ce361bf31;

Agents (on-prem workers)

List agents stale > 10 minutes

Show enabled agents whose last ping is older than 10 minutes. Important when diagnosing remote execution issues.

list-agents scope=all
  select parent.id, parent.name, parent.clientId, ping_date
  from ping
  where parent.enabled=true
    and ping_date < ts("now - 10m");

Update every enabled agent to the latest build

Fastest way to roll out a new agent version across infrastructure. Cluster-wide upgrades, fixing agent-level bugs, or ensuring consistent runtime environments.

// All tenants
list-agents scope=all select id, name where enabled=true | capture agents=stdout;
for-each (agent in agents) {
  update-agent {agent.id} --force;
};

Object history & versioning

Diff between two flow versions

Detailed diff between current and previous versions of a flow. Critical for debugging regressions and reviewing changes prior to deployment.

flow-history-diff 123 from=current to=last;

View change history for a connection

All past versions of a connection. Useful for auditing migration or verifying configuration drift.

connection-history 123;

Diff connection versions

Differences between two specific historical versions. Quickly identifies changes that lead to connection issues.

connection-history-diff f2472631-dedd-40ff-9238-abed8090fd16
                        75cc324e-bc6c-44e7-b60c-9a0eaeb95c18;

Tenants, users, and tags

List tenants & users

list-tenants;
get-tenant 1;
list-users;

Rename / delete tags

Standardize tagging in one shot. update-tags for the current account; update-all-tags for global super-admin operations.

Current account

update-tags '{
  "trim": true,
  "caseSensitive": true,
  "rename": [{"oldTag":"bigquery","newTag":"google bq"}],
  "delete": [{"tag":"bulk_merge"}, {"tag":"bulk_ merge"}]
}';

All tenants (super admin)

update-all-tags '{
  "trim": true,
  "caseSensitive": true,
  "rename": [{"oldTag":"bigquery","newTag":"google bq"}],
  "delete": [{"tag":"bulk_merge"}, {"tag":"bulk_ merge"}]
}';

Tag cleanup across production tenants

Apply consistent tag changes to a subset of tenants (e.g. all production environments) without impacting non-production accounts.

list-tenants --silent where name like "%prod%" | capture tenants=stdout;
for-each (t in tenants) {
  update-tenant-tags {t.id} '{
    "trim": true,
    "caseSensitive": true,
    "rename": [{"oldTag":"bigquery","newTag":"google bq"}],
    "delete": [{"tag":"bulk_merge"}, {"tag":"bulk_ merge"}]
  }';
};

Search across objects

Search flows, connections, formats, schedules, and other objects by keyword. Save time in environments with thousands of objects.

search 'etlConfig';

Runtime administrative controls

Suspend / resume all flow executions

Pause scheduler and manual executions during maintenance windows or outages, then restore.

flows-suspend --force;
flows-resume --force;

Emergency stop: force-suspend & stop active flows

Suspend execution AND attempt to stop currently running flows. Used for emergency stops.

flows-suspend-force --force;

Generate flow documentation

Output the full Markdown documentation for a flow. Integrates with automated documentation pipelines.

flow-documentation 123;