User-defined APIs

Expose any flow as a custom REST API endpoint, or have an external system trigger a flow by POSTing to one. Configure once, your endpoint lives at /rest/v1/httplistener/<your-pattern>, runs under the calling user's permissions, and shows up in the audit log alongside everything else.

Overview

The core building block is an HTTP Listener: a connection that listens on a configurable path and method. Depending on which side of the source-to-destination flow it sits on, the listener either pushes the request payload into the flow or returns the flow's output as the HTTP response:

What a request looks like

Receipt for POST and PUT

Unless the flow returns its own body, a POST or PUT gets a JSON receipt. Its shape depends on the listener's POST/PUT processing setting.

Async (the default): the receipt comes back as soon as the payload is stored, before the flow runs, so it has no auditId. status is received, or queued when throttling queued the request.

{
  "flowId": 3219,
  "messageUuid": "caaa8347-063c-449b-a29b-12e68799c986",
  "status": "received"
}

Sync: the caller waits for the flow. The receipt carries the execution’s auditId and its outcome, success or warning; a failed run returns an error status instead (see Response codes).

{
  "flowId": 3219,
  "messageUuid": "caaa8347-063c-449b-a29b-12e68799c986",
  "auditId": 106358,
  "status": "success"
}

Endpoint URL

Every user-defined API lives at:

https://<your-instance>.etlworks.com/rest/v1/httplistener/<your-url-pattern>

For Etlworks Cloud customers, the host is app.etlworks.com. For on-prem, it's your configured hostname.

URL pattern

Set the URL Pattern field when creating the HTTP Listener. The pattern becomes the path your endpoint responds at:

Path and query parameters become flow variables

Tokens like {patientid} in the URL pattern are extracted from the request path, URL-decoded, and exposed to the flow as variables: usable in source SQL, the FROM/TO fields, and JavaScript / Python. Query parameters are passed the same way (all of them except Authorization), so ?status=active becomes {status}. Do not put query parameters in the URL pattern itself.

patients/{patientid}

select * from patient where patientid = {patientid}

You can also surface request headers as variables: see Request & response headers.

Authentication: JWT (recommended)

Auth Type JWT is the default. Auth Location is either Header (the default, recommended) or Query Parameter. Despite the name, this auth type accepts both a JWT and an API key as the bearer token.

Three steps to call the endpoint

1. Create an API user with a role that can execute flows. (The API User role can call listener endpoints and the Messages and CLI endpoints of the platform API, but not flows, schedules or connections.)
2. Exchange username + password for an access token via the auth endpoint:
   ShellCopy

   curl -s -X POST https://app.etlworks.com/rest/v1/token-auth/issue \
     -H "Content-Type: application/json" \
     -d '{"login":"api-user","password":"…"}'
   # → { "token": "eyJ0eXAiOiJKV1Q…" }

   Or skip this step entirely: generate an API key, with or without an expiry date (Settings → Users → Edit user → API Key), and send it exactly where a JWT would go.
3. Send the token on every call to your endpoint:

   Auth location	Where the token goes
   Header	Authorization: Bearer eyJ0eXAi…
   Query Parameter	?Authorization=Bearer%20eyJ0eXAi…

Authentication: Basic

Less secure than JWT: use only when the calling system can't manage tokens (e.g. a third-party webhook that only sends a static Authorization header).

Set Auth Type to Basic on the listener. Auth Location is Header or Query Parameter; with Query Parameter, the credentials can also be embedded in the URL path (see next section). Users with two-factor authentication turned on cannot authenticate with Basic: use an API user without 2FA, or JWT.

1. Base64-encode user:password:
   Shell

   printf 'admin:admin' | base64
   # → YWRtaW46YWRtaW4=

2. Send on every call:

   Auth location	Where it goes
   Header	Authorization: Basic YWRtaW46YWRtaW4=
   Query Parameter	?Authorization=Basic%20YWRtaW46YWRtaW4=

Path-based auth (for restrictive webhooks)

Some third-party webhook systems (eBay, certain SaaS platforms) only let you configure a static URL, no header, no query string. The workaround: embed a doubly-base64-encoded Basic auth string as the last path segment.

1. Base64-encode user:password: admin:admin → YWRtaW46YWRtaW4=
2. Build the standard Basic auth string: Basic YWRtaW46YWRtaW4=
3. Base64-encode that: QmFzaWMgWVdSdGFXNDZZV1J0YVc0PQ==
4. On the listener, set Auth Type to Basic, Auth Location to Query Parameter, and end the URL Pattern with a token: some/path/{auth}. (A URL pattern can only contain letters, digits, . - _ /, so the encoded string itself, with its = padding, cannot be part of the pattern.)
5. Point the webhook at the URL with the encoded string as the last segment: …/rest/v1/httplistener/some/path/QmFzaWMgWVdSdGFXNDZZV1J0YVc0PQ==

When a request has no Authorization query parameter, the listener base64-decodes the last path segment and authenticates with it if it starts with Basic. The encoded string must not contain /; if it does, change the password. Changes to a listener can take up to a minute to take effect.

PULL: expose a data source as an API endpoint

Tableau wants the latest customer list, in JSON, behind an authenticated URL. Wire up a flow whose source is your warehouse and whose destination is the listener: the listener serves the flow's output as the response.

Setup

1. Create an HTTP Listener with Method = GET and a URL pattern.
2. Create a Format for the response: JSON, JSON Dataset, XML, XML Dataset, CSV, or Fixed-Length Text.
3. Create an "any-to-web-service" flow where the source is your data (database, file, API) and the destination is the listener + format from steps 1 & 2.
4. Configure the source query, mapping, and any flow parameters needed. Use {patientid}-style tokens to bind path parameters into the SQL.
5. Schedule the flow as event-driven. Etlworks executes it on every inbound GET.

Calling it

curl -s "https://app.etlworks.com/rest/v1/httplistener/patients/123" \
  -H "Authorization: Bearer eyJ0eXAi…"
# → { "id": 123, "name": "…", … }

Test it from inside Etlworks Explorer too: create an Etlworks API Connection pointing at the endpoint and browse the response there.

PUSH: trigger a flow when a payload arrives

A SaaS application updates a record. Its webhook fires and POSTs the updated record's JSON to your endpoint. Your flow runs: transforms the payload, writes it to the warehouse, sends an email.

Setup

1. Create an API user with a role that can run flows.
2. Create an HTTP Listener with Method = POST (or PUT).
3. Create a Format matching the inbound payload: JSON, XML, CSV, fixed-length, HL7 v2.x, FHIR.
4. Build a flow whose source is a web service. The source connection (FROM) is the listener.
5. Add mapping + parameters as you would normally.
6. Schedule the flow as event-driven. It runs every time a payload arrives.

Calling it

curl -s -X POST "https://app.etlworks.com/rest/v1/httplistener/hl7/patient" \
  -H "Authorization: Bearer eyJ0eXAi…" \
  -H "Content-Type: application/json" \
  -d '{"id":123,"name":"…"}'
# → { "auditId": 106358, "messageUuid": "…", "flowId": 3219, "status": "success" }

UPLOAD: ingest files via HTTP

An HTTP Listener can be the source of a Copy Files flow: turning a POST endpoint into a file dropzone that lands bytes in any supported file storage (server, FTP/FTPS/SFTP, S3, etc.).

Setup

1. Create an HTTP Listener with Method = POST and an Auth Type. URL Pattern uses a path token for the filename:
   URL Pattern

   file/{filename}

2. Create a Copy Files flow.
3. Set the FROM connection to the listener.
4. Set the TO connection to any file connection.
5. Set the destination filename to either a literal value (test.json) or the path token ({filename}) so the caller controls it.
6. Schedule the flow as event-driven.

Calling it

curl -s -X POST "https://app.etlworks.com/rest/v1/httplistener/file/test2.json" \
  -u "api-user:…" \
  --data-binary @local-file.json
# The body of local-file.json lands at the configured destination as test2.json.

CRUD: multiple methods on one resource

Build a full create / read / update / delete API by configuring listeners with the appropriate HTTP method and wiring each to a flow. Path tokens parameterize destination queries.

Example: tokenized destination query

update patient
   set name = {name}
 where patientid = {patientid}

With URL Pattern patients/{patientid} and a JSON payload {"name":"Jane"}, calling PUT /rest/v1/httplistener/patients/123 with body {"name":"Jane"} resolves to update patient set name='Jane' where patientid=123.

Custom responses

By default, POST and PUT endpoints return the receipt. Setting a POST/PUT Response File Name makes the listener synchronous even if POST/PUT processing says Async, since it has to wait for the file. Two ways to use it:

Option A: return the destination's response

Useful when the flow forwards the payload to another HTTP API and you want to relay that API's reply back to the caller.

1. On the listener, set POST/PUT Processing = Sync and POST/PUT Response File Name = {app.data}/response.json.
2. On the destination HTTP connection, set Response File Name to the same path.
3. Build the flow with the listener as source and that HTTP connection as destination.
4. Schedule event-driven.

The destination's response gets written to response.json and returned to the caller. The file is auto-deleted after the response goes out.

Option B: user-defined response file

Generate a response from inside the flow itself, without delegating to an upstream API.

1. Listener: POST/PUT Processing = Sync, POST/PUT Response File Name = {app.data}/response.json.
2. Build a primary flow that processes the inbound payload.
3. Build a second flow that writes the response file (same path as in step 1).
4. Combine them into a nested flow so the response file always exists by the time the listener replies.
5. Schedule the nested flow event-driven.

The response file is auto-deleted after the listener returns it.

Choose the status code from the flow

Set HTTP Code Variable on the listener to the name of a global variable, and set that variable in the flow; the listener returns its value as the status. Codes of 400 and above are returned as errors. If the field is empty or the variable is not set, the default code applies.

com.toolsverse.config.SystemConfig.instance().getProperties().put('patient_status', '404');

For GET, Empty GET response code decides what an empty result (nothing, {} or []) returns: 200 (default, the body as is), or 204, 400 or 404 with an empty body.

Request & response headers

Read inbound headers

Enable Pass request headers to the flow as parameters on the listener (off by default). Each header except Authorization becomes a flow variable. Variable lookups are case-insensitive to the uppercase form, so Validation-Token can be read as VALIDATION-TOKEN.

var token = scenario.getVariable('VALIDATION-TOKEN').getValue();

Set outbound headers

From any place a flow runs JavaScript or Python:

etlConfig.addResponseHeader('header-name', headerValue);

Setting Content-Type this way overrides the content type the listener would otherwise derive from its Format.

Real-world example: webhook subscription handshake

RingCentral's webhooks require the endpoint to echo a Validation-Token request header back in the response to complete the subscription. The flow:

var token = scenario.getVariable('VALIDATION-TOKEN');
etlConfig.addResponseHeader('Validation-Token', token != null ? token.getValue() : '');
value = true;

Empty payload

Some upstream systems POST with no body, especially webhook handshakes. Without a default, the parser raises an exception. Fix: set Default Payload on the listener to a valid document in the chosen format. The flow runs against the default whenever a POST or PUT body is empty.

Request preprocessor

For POST and PUT, Request Preprocessor script runs JavaScript against the raw request body before anything else happens: before the payload is stored and before the flow starts. The script sees the request as request:

• request.getInput() is the raw body; request.setInput(newBody) replaces what the flow receives.
• request.setSuccess(false) answers the caller without running the flow: the value of the script’s last expression is sent back with 200. If that value is empty, the caller gets 500 and a receipt with "status": "error".

var body = JSON.parse(request.getInput());
if (body.type == 'ping') {
    request.setSuccess(false);                 // reply now, skip the flow
    JSON.stringify({ pong: true });            // last expression = the reply
} else {
    request.setInput(JSON.stringify(body.data)); // pass only the data element on
}

Binary payloads

The body is read as UTF-8 text by default, and characters that are not valid UTF-8 fail the request with 400. Two settings change that:

• Non strict UTF-8 payload encoding accepts payloads with invalid characters instead of rejecting them.
• Preserve raw request body reads POST/PUT bodies as bytes and passes them to the flow Base64-encoded in http_request_body_base64, with http_request_body_is_base64, http_request_content_length, http_request_content_type and http_request_character_encoding alongside. Use it for files, images and other binary protocols.

Response codes

Test from inside Etlworks

You don't need an external system to exercise an event-driven flow. Build a "sender" flow that calls your listener.

1. Prepare test data. A source connection (file, cloud storage, database) plus a Format matching what real callers will send.
2. Configure the destination. Create an Etlworks API Connection with:
   • Endpoint path: /rest/v1/httplistener/<your-pattern>
   • Method: matches your listener (POST, PUT, GET, DELETE)
   • User / password: the API user configured for the listener
3. Build the sender flow. Source = test data + format. Destination = the API connection from step 2. Format on both sides matches.
4. Run the sender. It POSTs (or GETs / PUTs / DELETEs) to your listener, triggering the event-driven flow exactly as a real caller would.

The sender flow is reusable: keep it around for regression testing or wire it into a CI flow.

View inbound payloads

POST and PUT payloads are kept in the platform’s messages store, queryable by id, by listener, by status and by substring. Async requests are always stored; synchronous ones only when Keep message is on in the listener’s schedule. GET, DELETE and OPTIONS requests have no payload to store. Use the Messages app in the UI, or the CLI:

// Filter recent messages
list-messages-filtered message=Notification;

// Pull one message body
get-message 24a6e77b-c05f-45ed-b1e8-908ce361bf31;

Or via REST: GET /v1/messages: see Messages in the REST API reference.

Queueing and throttling

On busy instances, async POST and PUT requests to listeners can be rate-limited. Throttling is a global setting, off by default; when on, it allows 100 requests per minute unless configured otherwise (on-premise: listeners.throttle.enabled and listeners.throttle.throughput). Requests over the limit are either queued, and answered with a receipt whose status is queued, or rejected with 429. A listener’s Queue priority (1 high to 10 low) orders its queued requests against other listeners’. GET, DELETE, OPTIONS and synchronous POST/PUT requests are never throttled.

AS2 listeners

An AS2 listener is an HTTP listener for inbound AS2 messages, at the same /rest/v1/httplistener/<url-pattern> address. It accepts POST only, keeps the raw body, decrypts and verifies the message with the certificates you configure (local decryption certificate, partner signing certificate, AS2-To and optional AS2-From), and returns a synchronous MDN, signed or unsigned. Authentication is optional (None or Basic), because AS2 exchanges usually rely on TLS and certificates. The decrypted payload goes to the flow like any other listener payload.