Validation rules
Row-level and field-level rules written in JavaScript or Python, evaluated per record. Each rule chooses its own failure action, so a null postcode and a corrupt batch are not the same event.
Data quality
Clean it on the way through.
Validate, filter, dedupe, enrich, and reshape while the data is moving — then verify what landed with SQL or dbt tests. Not a separate quality tool bolted on after the fact.
Where each check runs
- Source side Filter before it moves SQL source query · predicate filter · ORDER BY
- In flight Validate, dedupe, enrich, reshape Per record, as the dataset streams
- Destination side Verify what landed Pre-load and post-load SQL · dbt tests
Three places to catch a problem, and the cheapest one is first. A row filtered in the source query costs nothing to move, transform, or roll back.
Capabilities
What’s built in.
Predicate filtering
Drop rows that fail an expression before they reach the destination. Write it in SQL or JavaScript. Against a relational source, push it into the WHERE clause and the rejected rows never leave the database.
Deduplication
Drop records matching an earlier row on a field set you define. Each incoming row is compared against what the transformation has already processed.
SQL over sources that have no SQL
Query CSV, JSON, XML, Excel, Parquet, Avro, APIs, message queues, and NoSQL with real SQL — filtering, aggregation, and joins, including over nested structures.
Enrichment
Look up reference data and derive calculated fields in script. Cache a reference dataset in memory once and reuse it across transformations instead of querying per row.
Nested reshaping
Flatten hierarchies into tables or nest flat rows back into documents, by dragging. Any shape on either side, across JSON, XML, Avro, and Parquet.
Pivot and denormalize
Turn rows into columns grouped by a key, or convert key-value pairs into real columns. Control the leading column, the column cap, and what carries through.
Set operations
JOIN, UNION, MINUS, INTERSECT, and EXCEPT — on one database through native SQL, or across heterogeneous sources that have no business being joined but need to be.
Verification, before and after
Run SQL checks ahead of a load and again once it lands, or hand the assertion to dbt tests in the same workflow. A load that passed is not the same as a load that is correct.
Specifications
You decide what a bad row costs.
A validation rule carries its own failure action, per field and per record. The same source can drop a malformed phone number and refuse the entire file for a missing primary key.
Failure actions
Reject record
The offending row is dropped, the rest of the dataset loads. Use for dirty rows in an otherwise good batch.
Reject dataset
The whole batch is refused. Use when a bad row means the file or extract cannot be trusted.
Halt flow
Execution stops and raises an exception. Use when downstream steps must not run on this data.
Continue
The row passes with the failure recorded. Use while profiling a new source, before you know which rules are real.
Where checks run
Source side
SQL source query, predicate filter, ORDER BY · Rejected rows never leave the source system.
In flight
Validation, dedup, enrichment, reshape, pivot, set operations · Per record, as the dataset streams through.
Destination side
Pre-load SQL, post-load SQL, dbt tests · Verify the landed result, not just the transfer.
FAQ
Common questions.
What happens to the rows that fail?
Whatever you decide. A validation rule picks one of four actions: reject the record, reject the dataset, halt the flow, or continue with the failure recorded. Rejected records can be written to a quarantine table from script, so nothing disappears silently.
Do I have to write code for all of this?
No. Dedup, filtering, pivots, set operations, and nested reshaping are configuration. Validation and calculated fields are script because that is where the rules get specific, and they take JavaScript or Python. You can also ask Simba, our AI agent, to build the mapping and refine it by conversation.
Can I really run SQL against a CSV file or an API?
Yes. SQL works over CSV, JSON, XML, Excel, Parquet, Avro, API responses, and other non-relational sources, including nested structures, with filtering, aggregation, and joins. It is the same engine, not a pushdown to a database.
Where should filtering happen?
As early as possible. When the source is a relational database, put the predicate in the source query so the database filters and Etlworks never sees the rejected rows. For files and APIs, the filter transformation runs per row as the dataset streams.
How does this relate to dbt tests?
They cover different moments. These checks run inside the pipeline, on the data in flight, before it lands. dbt tests run in the warehouse against what landed. A dbt flow nests in the same workflow, so one execution can load, verify, and test.
Does cleaning slow the load down?
The transformations run as the dataset streams rather than as extra passes, and source-side filtering reduces what moves at all. The expensive pattern is per-row lookups against a database, which is exactly what caching a reference dataset in memory is for.
Start your trial
14 days. No card. Real workloads.
Point a flow at a source you do not trust and see what the rules catch.