MQTT 3 and MQTT 5
Pick the protocol version your broker runs and the quality of service the data deserves — at most once, at least once, or exactly once. Connections speak mqtt, mqtts, ssl, or tcp.
IoT and MQTT
Your devices already publish. Something has to listen.
Etlworks subscribes to your MQTT broker the way any other client does — wildcard topics, QoS you choose, certificates if you use them — and lands the telemetry in a warehouse, a database, or a file, continuously.
From device to destination
- Devices Publish to the broker Sensors, gateways, and controllers, on the topics they already use
- Etlworks Subscribes and transforms One client, many topics, decoded and reshaped in flight
- Destination Warehouse, database, file Or straight back out to a second broker, byte for byte
MQTT is deliberately small — it assumes a constrained device and an unreliable network. The work is everything after the subscribe: decoding the payload, naming the destination, and not falling over when the link drops.
Capabilities
What’s built in.
Wildcards, not one flow per device
Subscribe to test/device/# and every device underneath it arrives on the same flow. Several topics, wildcard or not, can be listed together, and a single flow can publish to a different topic per transformation.
Certificates, not just a password
Authenticate with a username and password, or with a root certificate, client certificate, and private key. TLS encryption is a switch, and hostname verification can be relaxed for the brokers that need it.
Payloads that are not JSON
Read text payloads as strings or binary ones as an array of bytes, in JSON, CSV, or raw form. Messages can be Base64 or Sparkplug B encoded and gzip compressed, and the decode happens before your transformation sees the record.
Reshape before it lands
A preprocessor written in JavaScript sees each message on the way through. Read a field, rewrite it, add one the device never sent, or return false and the reading is dropped — which is how you filter a chatty sensor without touching the device.
Stream it or batch it
Run the flow continuously so readings land as they arrive, or schedule it on a short interval for micro-batching when the destination is happier with fewer, larger writes. The same flow definition does both.
Broker to broker, untouched
The Byte Array format forwards a message from one broker to another without altering it — for a migration that has to keep continuity, for regional brokers that each serve their own network, or for two applications that will not share one.
Built for a link that drops
Keep-alive interval, connection and publish timeouts, a ceiling on failed probes before the connection counts as lost, and a last will topic the broker publishes if the client disappears without saying goodbye.
The same engine as everything else
MQTT is one more queue connector, so telemetry joins reference data from a database, gets validated, and lands in the warehouse in one governed flow. CDC events can be routed through an MQTT topic as well, if that is where your consumers already listen.
Specifications
What the connection asks for.
Everything below is set on the connection, once, and reused by every flow that points at that broker. Credentials live in the connection, never in the flow definition.
Connection
Broker URL
Host and port, optionally prefixed
mqtt://, mqtts://, ssl://, or tcp://Topics
One or more, wildcards allowed, comma-separated for a list
Client ID
Required, and unique per client — most brokers disconnect the previous session if a second client claims the same ID
Quality of service
0 at most once, 1 at least once, 2 exactly once
Protocol version
MQTT 3 or MQTT 5; the publish timeout applies to MQTT 5 only
Will topic
Published by the broker when the client disconnects unexpectedly
Security
Authentication
Username and password, or root certificate, client certificate, and private key
Encryption
TLS, with optional relaxed hostname verification
Reading and writing
Serializer
String for text payloads, array of bytes for binary
Formats
Reading: JSON, CSV, Byte Array. Writing: JSON, Avro, Byte Array
Encoding
None, Base64, or Sparkplug B, with optional gzip compression
Polling
Records per poll, wait between polls, retry count and window, and whether to drain remaining messages before stopping
FAQ
Common questions.
Do I need one flow per device?
No, and you should not build it that way. Subscribe to a wildcard topic such as
test/device/# and every device beneath it arrives on one flow. The destination table or filename is built from the topic name, so a thousand devices can land in one table or a thousand, depending on the template you write.Our payloads are binary, not JSON.
Set the value serializer to array of bytes rather than string. From there the payload can be Base64 or Sparkplug B encoded and gzip compressed, and it is decoded before the transformation runs. If the payload should not be interpreted at all, the Byte Array format passes it through untouched.
Why does my connection keep dropping?
The usual cause is a client ID collision. Most brokers allow one client per ID, so a second client connecting with the same one ends the first session — which looks like a flapping connection rather than a configuration mistake. Give every client, including every Etlworks connection, an ID of its own.
Can I forward messages to a second broker without changing them?
Yes. Use the Byte Array format on both the source and the destination and the message crosses byte for byte. This is the pattern for migrating between brokers without a gap, for distributing to regional brokers, and for bridging two applications that each own their own.
Is this real-time, or batched?
Either, and it is the same flow. Schedule it as a continuous run and readings are written as they arrive. Schedule it on a short interval instead and it micro-batches, which some destinations prefer. Clearing the record limit and the stop-after window is what keeps a streaming flow from ending on its own.
Can telemetry be joined with data from elsewhere?
That is the reason to land it here rather than in a purpose-built IoT tool. MQTT is one more connector in the same engine, so a reading can be enriched from a database lookup, validated, reshaped, and loaded alongside the rest of your data in one flow with one execution history.
Start your trial
14 days. No card. Real workloads.
Point a connection at your broker, subscribe to a wildcard topic, and see what your devices are actually sending.