The agent loop is the easy part

Simba, our AI agent, is the part of Etlworks people ask to see in a demo. You describe an integration and it builds one. That part is easy to show and easy to copy.

The part nobody sees is the constraint that shaped it. A large share of our customers run Etlworks on their own hardware, inside networks that do not permit outbound connections to somebody else's service. Every decision below comes from that one fact.

The agent is about a thousand lines of Java. No AI framework, no vendor SDK. The loop was the fast part to build. Everything around the loop is where the work actually went, and that is the part worth reading about.

The version we threw away

Version one was a separate service written in Python on an off-the-shelf agent framework, reached from the main application over a proxied connection. It was a reasonable choice. The framework did what it claimed, we shipped fast, and in our own cloud it ran well.

It failed where it mattered most. Customers who run the platform themselves, behind corporate firewalls and in locked-down networks, do not keep a tunnel open to a third-party service so a chat box can work. The agent had become the one component of the product that assumed a network posture our most security-conscious customers refuse to have, and those accounts have the largest data volumes and the longest contracts.

Version two moved the agent inside the application. One deployable, no companion service, no proxy hop. If the product runs, the agent runs.

That decision settled the framework question. The Java runtime we support across our installed base predates what the Java agent frameworks require, so there was nothing to adopt. We wrote directly against the model provider's HTTP API.

We expected that to be painful. It wasn't. There is no dependency in the agent we can't read, nothing breaks when a framework renames its abstractions, and it is small enough that any engineer here can hold it in their head.

How it works now

The loop. Send the conversation to the model. If the model asks for tools, run them, append the results, and send the whole conversation again. Repeat until it returns an answer instead of another tool request. That is the entire algorithm, and every agent you have used runs it.

The loop is bounded by construction. It runs a fixed maximum number of rounds, and on the final round the tools are simply not offered to the model. It cannot request another one because there is nothing to request, so it answers. No prompt asking it politely to wrap up, and no runaway. It is the cheapest reliability win in the design.

Everything streams, but not everything has to. The model streams tokens and we forward them to the browser as they arrive. Callers that want a single response, our API and external agent clients, run the same loop with a handler that buffers instead. One code path, several surfaces; two implementations would drift, and the one used less often would break.

Tools describe themselves. Each tool is a small class declaring its name, a description, and its input schema, and the application finds all of them at startup. Adding a capability means adding a class: no central list to edit, nothing to forget to register. The model is shown exactly the tools the current user is permitted to use, and nothing else.

A failing tool is a message, not an outage. The error text goes back to the model as the tool's result. The model reads it and adapts. One broken tool does not end the conversation.

Retrieval with no vector database. Documentation, support history, templates, and the command reference are chunked, embedded, and stored in compact files beside the application. The vectors are compressed to a quarter of their original size and searched in compressed form, so a large corpus stays in memory on an ordinary instance. Search combines semantic similarity with keyword matching and merges the two rankings, which beats either alone. Exact product names and error strings are where pure semantic search is weakest, and they are most of what people search for. With no model provider configured at all, retrieval degrades to keyword-only and still returns something useful. There is no vector database to deploy, license, or operate.

The prompts are hand-written, and they were the slowest part. There is no prompt framework here. Every system prompt in the agent was written by hand, run against real work, and rewritten. Flow creation was the hardest of them by a wide margin: describing what a good integration looks like, which defaults to pick, when to ask a question instead of assuming an answer. Getting a model to build an integration the way an experienced engineer would build it is still where most of our agent work goes.

What we didn't have to teach it

Three things work that we never wrote instructions for.

The command layer returns structured JSON, and what comes back is the real response from the underlying service, thousands of attributes wide. We assumed we would have to explain what those attributes mean. We never did.

The platform ships more than four thousand flow templates, so building a flow from one is a single click. Flows are also stored as JSON, and the agent turned out to read that representation directly and change the right attributes in it, with no hint from us about which attribute does what.

It also chains tool calls on its own, switching tools mid-question when the first does not return what it needed. That last one we specifically did not expect.

What it can actually do

This is not a chat box bolted onto a product. The agent has the same reach as an experienced administrator, through the same interfaces:

• Answer questions about the product, grounded in real documentation and support history and returned with sources, rather than in the model's general knowledge.
• Build and change integrations. Create a flow, edit its mappings and transformations, wire up sources and destinations, from a plain-language description.
• Work with connections and formats. Find them, describe them, create them, test them.
• Research an API on the web. Connecting to something the platform has not seen before, the agent reads the vendor's public documentation and builds the connection from it: endpoints, authentication, pagination, and the shape of what comes back.
• Schedule and operate. Put a flow on a schedule, run it, stop it, read its log, explain why the last run failed.
• Investigate. Search across artifacts, trace where a connection is used, inspect execution history and errors, check resource use.
• Do administrative work through the platform's own command layer, the same operations an administrator would run by hand.
• Assist inside the editors. While someone is building a flow or editing field mappings, the agent sees the same context and can act on it.

It runs wherever the product runs

Because the agent is part of the application rather than a service it calls, every deployment shape gets the same agent with no extra moving parts: our multi-tenant cloud, dedicated single-tenant instances, any major cloud, on-premise, containerized, and restricted networks.

Restricted networks deserve a precise statement. The agent needs a model endpoint. What it does not need is one on the public internet, because the endpoint is a configuration value: an instance can point at a private or self-hosted API-compatible endpoint inside the customer's network. Everything else already lives in the deployment: the retrieval corpus, the tools, session storage, and metering.

Tuning an agent you can't redeploy

Nearly every control below exists because something happened to us: a provider price change, an outage, a customer with an unusual network, a cost we did not see coming. You will hit the same things in roughly the same order.

Configuration arrives in three layers, in priority order:

1. Local configuration on the instance, owned by whoever operates it. It always wins: an on-premise operator can pin any behavior and nothing we do overrides them.
2. Configuration from our license server. The application reaches out on a schedule and pulls its configuration down. The license server never connects in and nothing is ever pushed. That is the same outbound-only posture that forced the in-process design, and it means managing an instance never requires an inbound firewall rule.
3. Built-in defaults, so a fresh install works with nothing configured.

The payoff is that we can change how the agent behaves for one customer, which model it uses, what it is allowed to do, what it costs, with no build and no restart. For a product that ships on-premise, that is the difference between a same-day fix and a release cycle.

A separate set of settings belongs to the customer and is edited in the product's own admin interface, most importantly their own model-provider key. The split is deliberate: what we control, what the operator controls, and what the customer controls are three layers with three owners.

What is configurable about the model, which is more than most people plan for:

• Which model. Not compiled in. Swapping it is a configuration change, which is what makes it survivable when a provider deprecates a model or ships a better one mid-quarter.
• The API endpoint. Easy to skip, expensive to retrofit. This is what makes restricted-network deployment a setting rather than a project. Make the endpoint configurable on day one even if you only ever intend to call one provider.
• Pricing. Per-model rates are configuration, not code: input, output, the discounted rate for cached input, and the threshold at which very large requests move to a higher rate. Providers change prices with little notice, and an agent with hardcoded rates starts quietly reporting the wrong cost the day they do. This is the most under-appreciated item on the list.
• The embedding model used for retrieval, configured separately from the chat model. They change on different schedules and are rarely from the same generation.

Four routing patterns worth stealing. Most of this is graceful degradation, which is what routing turns out to mean in practice.

1. A different model per surface. The in-app agent and the assistant on our public website are the same code with different models. The public one runs a smaller, cheaper model because its job is narrower. One agent, several cost profiles.
2. A cheap model gates, an expensive model answers. Before the full agent runs on an inbound support ticket, one cheap call with no tools decides whether the ticket deserves an automated answer at all. Most volume never reaches the expensive path. This generalizes to nearly any high-volume trigger.
3. Tier fallback when the primary model is unavailable. Retries with exponential back-off come first. If the primary is still unavailable, the agent drops to a configured lower tier and retries the failed step rather than failing the conversation. Two details matter. The switch lasts only for the current exchange, so a brief provider blip does not leave the system quietly degraded. And it deliberately does not trigger on billing or quota failures, where a cheaper model would not help and hiding the problem is worse than showing it.
4. Degrade rather than deny at the spending limit. When a customer exhausts their allowance the agent does not go dark. It loses the ability to act, keeps answering questions, and can drop to the cheaper model so what remains costs less. People get a working assistant that explains its own limits instead of an error page.

The through-line: decide in advance what your agent does when something is unavailable or unaffordable. Most implementations define the happy path and then discover their failure behavior in production.

Other things that turned out to need a switch. Each was added after we needed it, which is itself the lesson:

• Individual tools can be turned off per instance with no build. When a customer is not comfortable with a class of action, that becomes a configuration change instead of a negotiation.
• Agentic behavior can be turned off entirely, per instance or per user, leaving questions and answers. AI can also be disabled outright. Some customers need to be able to say "off" and mean it.
• Spending limits, a prepaid balance, and optional auto-recharge, so reaching a limit is a soft landing rather than a wall. The public assistant carries its own daily limit, because anything anonymous users can reach needs a budget independent of customer accounts.
• Secret redaction in stored transcripts, and a retention period. People paste credentials into chat windows. Assume they will, and scrub what you store rather than what you display.
• Tracing as two independent switches, one for the operator's log and one for what is attached to a conversation for support to read. Different audiences and different privacy weight. We shipped it as a single flag first and corrected it.
• A ceiling on how much one tool call can return. With no ceiling, one broad query floods the model's context and turns a cheap exchange into an expensive one. Design this in early; it is an unpleasant thing to learn from an invoice.
• An origin allowlist on the endpoint that external agent clients connect to.

If there is one takeaway for someone building this: the agent loop is the easy part. What takes the milestones is everything around it. What happens when the model is down, when the money runs out, when the network is locked down, when a provider changes prices, and when somebody pastes a password into the chat box.

Constraints that live in code, not in the prompt

The agent acts as the person talking to it. It cannot see or do anything that user could not do directly. Tenant isolation is enforced by the same mechanism as the rest of the product, and an administrator working across tenants has to name the tenant explicitly.

Tool exposure is enforced in code, not in the prompt. On restricted surfaces, the assistant on our public website being the clearest example, the disallowed tools are never presented to the model at all. There is no instruction to disregard, because there is no instruction. Prompt injection has nothing to escape into. It is the line between a real implementation and a demo: if your safety story is a paragraph in a system prompt asking the model to behave, you do not have one.

Per-user controls. Administrators can restrict individual users from taking actions, from using their own provider key, and from reading stored conversations.

Provider keys are never stored on a conversation. They are resolved for each request, and the conversation records only which kind of key was used, which is all billing needs.

Secrets are redacted from stored transcripts. The live conversation is unaffected; what lands in durable storage, and in any exported transcript, is scrubbed. Conversations are deleted automatically on a schedule.

Operator kill switches, as described above. Turning the whole thing off is part of the security model.

Who pays, and for what

Usage is recorded for every model call rather than once per conversation: tokens in, tokens out, cached tokens, which model, what it cost, and who it was for. An agent that meters per conversation is guessing.

Cached input is priced separately, and accounting that ignores it overstates cost badly. An agent re-sends the whole conversation on every round, so a five-round exchange sends the early turns five times. Providers bill that repeated context at a steep discount, and totalling it at the full input rate makes long conversations look far more expensive than the invoice says they were.

Acting and answering are billed differently. Asking a question and getting an answer out of documentation is cheap, and it does not count against the allowance. Having the agent do something, build a flow, run a command, change a schedule, is what is metered. One is a search engine and the other is work. Charging the same for both would push people away from asking questions, which is the behavior we most want.

Bring your own key. A customer, or an individual user where permitted, can supply their own model-provider key. The provider bills them directly, we do not meter them, and usage limits do not apply. This matters for enterprises with an existing provider agreement.

Internal and staff usage is recorded but never billed, so the audit trail is complete regardless of who pays.

What we'd do again

Writing the loop ourselves, and making the model endpoint configurable before we had any reason to. Those two decisions are why the agent runs in a bank's data center and in our cloud from the same source.

What we would do differently is smaller and duller. We would split the two tracing switches at the start, and put the ceiling on tool-call output in before the first surprising bill rather than after it. We would also write the pricing table as configuration from the first commit, because we did it later, under time pressure, right as a provider changed its rates.

If you are starting now, budget accordingly. The loop is not the project. The project is everything that happens when the loop meets a real network, a real budget, and a real customer.