All posts

A green run tells you almost nothing

Run history says what happened. It does not say whether your schedules collide, which flow is eating the box, or what breaks next. That is what Insights is for.

Most data platforms hand you a run history and call it monitoring. Green means it ran, red means it did not, and there are logs if you go looking.

That answers one question, and it is the least interesting one. The questions that cost money are different. Are my schedules stacked on top of each other? Which flow is consuming the box? Is anything degrading slowly enough that nobody has noticed yet? What breaks next week?

Insights is where we put those answers. Admin and SuperAdmin only, since most of it is account-wide.

Statistics

Total records processed, executions split across scheduled, on-demand, and listener, daily and monthly trends, success against error distribution, and counts of flows, connections, schedules, and formats. SuperAdmins can switch to multi-tenant mode and see the same metrics aggregated, with a per-tenant breakdown behind each number.

When you actually run

The Executions Heatmap plots activity by hour, day, week, or month with intensity by volume, and reports average executions, records processed, average duration, and your peak window. Click a cell to drill into the executions behind it.

Nearly every environment we look at has the same shape: everything on the hour, nothing in between. The heatmap is what makes that visible.

Scheduled Activity goes further. Expected Timeline simulates upcoming runs from your schedules. Gaps finds windows with nothing in them, which is where the next workload belongs. Scheduled vs Actual compares planned against real and shows the drift. Ask takes the same questions in plain language.

Resource utilization

CPU load and processor count, total and used RAM, JVM heap, thread count and peak, OS and data disk space, open file descriptors against the maximum, and running flow count. Broken out per cluster node.

The part that earns its keep is Top flows by estimated resource consumption, with CPU time, memory average and max, and a confidence level on the estimate. It answers "which flow is doing this to us" in about ten seconds.

Incidents open automatically when CPU, RAM, or disk breaches a threshold and stays there for a sustain duration, then close when it settles back for a configured period. Each records the resource, a severity from Info to Critical, opened and resolved timestamps, and the trigger value. The sustain and settle windows exist so a brief spike does not wake anyone.

The one that finds problems for you

Flow Findings, our dashboard of automatic inspection reports, is the piece I would look at first. Every time a flow runs the engine writes a report: schema drift, slow steps, recurrent errors, data quality concerns. The dashboard aggregates them across the instance.

Each report carries a severity from ALL_GOOD through CRITICAL, computed from the worst issue in it. Each issue names what was detected, why it is flagged, and what to do about it. Filter by severity, flow, tenant, or report age. Reports outlive the flows that produced them, so a deleted flow still shows its history.

That last part matters more than it sounds. It is the difference between a dashboard that reports numbers and one that tells you which three flows to fix this week.

AI usage

Tokens split by prompt and completion, agentic against non-agentic so tool calls are separated from questions, conversations and requests, and a daily chart for the month. On billing: total, billable, and non-billable cost, wallet balance, monthly drain, auto-recharge status, and the monthly cap with current usage. Tenants running their own OpenAI key show zero billable cost, since their provider bills them directly.

Findings retention is configurable from one to thirty-six months. Start at the heatmap and Flow Findings. Between them they usually explain whatever you came to investigate.

Full reference: Account dashboard (Insights), and everything else operational lives under Operations, Monitoring and Observability.

Try it on your own data.

14 days, no card. Point Etlworks at a real source and see what it does.