Etlworks on Kubernetes: pods that drain before they go

In May we put official Kubernetes support on the roadmap: charts we maintain, a documented multi-node topology, behavior we test rather than infer. It shipped, and it turned into more than charts. Etlworks now runs on your Kubernetes as a managed cluster that adds application pods when demand grows and removes them safely when it falls.

Scaling in is the hard part

Adding a pod is easy. Removing one is where data platforms get hurt, because a pod that is halfway through loading a table is not idle just because the autoscaler wants fewer replicas.

So the scaling is split in two. KEDA and the Kubernetes HPA decide how many application pods you should have. A separate lifecycle adapter, which stays in the cluster after installation, does the adding and the removing. Before it removes a pod it waits for that pod's work to finish and its traffic to drain: ordinary flows run to completion, and only continuous executions that are eligible to move get handed over. The HPA never deletes a busy pod directly.

The consequence is visible and intended: while a long flow runs, the actual pod count can stay above the desired count. That is the drain doing its job. A new pod takes new executions; it does not split a flow that is already running.

Each pod runs the full application with its ETL engine. Your Cluster license sets the starting pod count, the pod size and the maximum, and the operational policy decides demand-based scaling inside those bounds.

Installing it

You bring the Kubernetes cluster: version 1.34, at least two Linux worker nodes, AMD64 or ARM64. The installer runs from a Linux or macOS workstation with Java 25, Python 3.12, kubectl and Helm 3, and deploys everything else, including PostgreSQL, Redis, shared storage, HAProxy, KEDA and cert-manager where they are missing. You do not build images or install Tomcat on the workers.

There are two downloads: the installer ZIP for your node architecture, and a separate ZIP with the signed Cluster license and its verification keys. You describe the target in an install.json (kubeconfig context, hostname, two storage worker nodes, a file with the initial administrator password), then:

sh deployment/installer/etlworks-kubernetes.sh preflight \
  --config ../etlworks-private/install.json

sh deployment/installer/etlworks-kubernetes.sh install \
  --config ../etlworks-private/install.json --wait 20m

If the terminal drops or the wait runs out, status and resume continue the same installation. Its state lives in the cluster, so a resume reconciles instead of starting a second fleet.

The installer is on the tools page. To try it, request a 14-day Kubernetes trial and pick a pod size from 8 GiB and 2 CPUs up to 128 GiB and 32 CPUs. The trial starts one pod and can scale to two, in a separate evaluation environment, so an existing customer's license is not touched.

The limits

The profile in this release is called local-nfs-v1. It provisions PostgreSQL, Redis and the NFS server as single instances on worker storage. Application pods scale, but losing the storage worker can still take the application down. That makes it an evaluation profile, not a highly available production layout. For production, talk to us and we agree the profile for your cluster.

Amazon EKS, Azure AKS and Google GKE need a profile agreed with us before you install. Their networking, storage and node maintenance differ, and the installer will not replace a provider's network or storage to make a check pass.

Node scaling is your platform's job. More desired pods do not create machines unless you run a node autoscaler, and each pod has to fit on a single worker, so two half-empty nodes cannot host one large pod.

The installer has a PowerShell launcher, but installing from a Windows workstation is not qualified yet.

The full procedure, from cluster preparation to troubleshooting, is in Install Etlworks on Kubernetes. The roadmap now lists it as shipped, and the scorecard compares it with what we promised in May.