Streaming and message queues

The queue is real time. The pipeline usually isn’t.

Kafka, Azure Event Hubs, Google Pub/Sub, Amazon Kinesis, SQS, RabbitMQ, and ActiveMQ. Read from them, publish to them, or stream continuously into a warehouse with lag you can watch — and send database changes straight to a topic.

Start free trial Talk to us

Three directions

  1. Publish Any source → topic Extract, transform, and write messages in JSON or Avro
  2. Stream Topic → destination Loaded as messages arrive, not on the next scheduled run
  3. Replicate Transaction log → topic Inserts, updates, and deletes published as they commit

The middle one is the difference. Polling a topic on a schedule is still batch — it just runs more often, and the lag is whatever your interval is.

Capabilities

What’s built in.

Seven brokers, one flow model

Kafka — including Confluent, Aiven, and Amazon MSK — plus Azure Event Hubs, Google Pub/Sub, Amazon Kinesis, Amazon SQS, RabbitMQ, and ActiveMQ. The flow types are the same across all of them, so moving from one broker to another is a connection change, not a rebuild.

Streaming, not micro-batching

A streaming flow runs continuously until you stop it or it errors, then restarts itself after a configured number of seconds. There is no interval to tune and no window where messages sit waiting for the next run.

Database changes into a topic

CDC flows read the transaction log of SQL Server, MySQL, Oracle, PostgreSQL, Db2, MongoDB, or IBM i and publish each insert, update, and delete to the queue. The flow takes an initial snapshot of the included tables first, then switches to real time.

And back out again

Stream CDC events off the queue into any supported destination. It reads the events Etlworks enqueued, and it reads the ones Debezium enqueued — so an existing Kafka and Debezium setup keeps its producer and gets a consumer that lands the data.

JSON, Avro, CSV, or untouched

Read and write JSON and CSV, and Avro in both directions — including Avro written by a third-party producer, where you supply the schema. The Byte Array format moves a message from one broker to another without altering a single byte.

Reshape a message in flight

A consumer or producer preprocessor written in JavaScript sees each message as it passes. Read a column, change it, add one that was never in the payload, or return false to drop the message entirely before it reaches the destination.

Many topics, one flow

Subscribe with a wildcard such as inbound.* or a comma-separated list. The destination name is built from [table], [db], and [schema] tokens or the topic name, and a script can compute it per message when the naming is not mechanical.

Lag you can actually see

Every running stream reports messages per second, current lag, the highest lag since it started, total records processed, and how long ago the last message landed. Checkpoints update every 60 seconds, in the UI and through the API.

The auth your broker expects

Kafka over SASL with SCRAM or PLAIN, and AWS IAM for MSK with no username or password at all. Kinesis and SQS take an access key and secret or an attached IAM role. Pub/Sub takes a project and a service account.

Specifications

Where a stream can land.

Streaming needs two things: a destination that can accept a continuous write, and messages in JSON. Everything else is still reachable — through a scheduled flow that reads the queue in micro-batches, which also lets you add columns in the mapping.

Streams continuously
Warehouses
Snowflake, Amazon Redshift, Azure Synapse Analytics, Google BigQuery, Databricks, ClickHouse, Vertica, Greenplum
Databases
Relational databases, row by row or through the destination’s bulk loader
Cloud storage
Amazon S3, Azure Blob, Google Cloud Storage, Box, Dropbox, Google Drive, OneDrive for Business, SharePoint
File transfer
Server storage, FTP, FTPS, SFTP, WebDAV, SMB share — written as CSV or JSON in real time
Use a scheduled queue flow instead
HTTP endpoints
Including well-known APIs such as Google Sheets
NoSQL and caches
NoSQL databases and Redis
Another queue
Queue-to-queue moves run as a flow; the Byte Array format keeps the payload byte-identical
Outbound email
Sent per batch rather than per message

FAQ

Common questions.

What is the difference between streaming and reading the queue on a schedule?
A scheduled queue flow wakes up, drains a micro-batch, and stops. A streaming flow holds the subscription open and writes each message as it arrives. Streaming is faster and uses fewer resources, and it is the only one of the two that gives you a real-time pipeline. The scheduled flow is still the right answer when the destination cannot take a continuous write, or when you need to add columns in the mapping.
Do I need Kafka to do change data capture?
No. The CDC engine reads the transaction log directly and can load a destination with no broker anywhere in the picture. A queue is worth adding when you want to decouple extraction from loading, fan the same change stream out to several consumers, or feed services that are already listening to topics.
Our messages are Avro, written by another application.
That case is handled separately from Avro that Etlworks itself wrote. Set the value deserializer to Avro Record and supply the Avro schema, and the payload is read as it stands. Avro written by an Etlworks flow uses the plain Avro deserializer and needs no schema.
Can I move messages between two brokers without changing them?
Yes, and this is worth doing deliberately rather than by accident. Use the Byte Array format on both sides and the message crosses untouched — the usual reasons being a broker migration that has to preserve continuity, distribution across regional brokers, or two applications that each own a different broker.
How do I know the stream is keeping up?
Open the running task and read the metrics: messages per second, the last recorded lag, the maximum lag since the stream started, and the timestamp of the last message processed. They refresh on a 60-second checkpoint, with manual and automatic refresh in the UI, and the same numbers come back from the API for a single flow or for every running flow at once.
What happens when a streaming flow stops?
A continuous run schedule restarts it after a set number of seconds, whether it stopped because of an error or because the queue went quiet. On Kafka, leaving auto-commit off means the offset is committed after the messages have been processed rather than on a timer, so a restart picks up where the work actually finished.

Start your trial

14 days. No card. Real workloads.

Point a flow at a topic you already have and watch the lag number while it drains.

Start free trial Talk to us