Built Jitterbit's agentic integration layer

Architecture and reasoning are shared here; customer workloads and confidential details are omitted.

Someone in warehouse operations needs a new order in the storefront to reach the ERP, and finance to hear about it. The integration itself is a solved problem. Building it is not: a canvas of connectors, mappings and operations, and a vocabulary nobody hired them to learn. So they file a ticket and wait for IT.

Jitterbit sells the platform that closes the second half of that gap. Its iPaaS builds and runs integrations across a few hundred connectors and a marketplace of prebuilt recipes. Its API Manager turns those integrations into APIs, then secures, publishes, monitors and governs them. Both are low-code, which is not the same thing as no-vocabulary.

I joined the early AI team in October 2024 and built the agentic layer over both, through to production rollout. An APIM bot that walked a user through designing, exposing, monitoring and managing an API across its lifecycle, and an iPaaS bot that turned a request written in natural language into an integration workflow. The foundations were already there and already good. What I owned was the layer that let someone operate them by saying what they wanted.

That layer had four jobs, in order. Understand the request. Choose the right tools, and the right prior art from the platform's own corpus. Execute against a live system other people's businesses were running on. And stop to ask when the workflow needed a judgment that was not the agent's to make. The last one is what shaped the architecture: an agent that drafts an integration is a demo, an agent that can change an order flow or expose an API is an operator, and operators need boundaries.

Which is why almost none of the interesting work was the model. It was context, memory, retrieval, caching, evaluation, tracing, access and failure handling, treated as execution architecture rather than as the things you add once the demo works.

APIM: when tools stopped scaling

API lifecycle management is a fixed set of workflows. Design an API, configure it, expose it, secure it, monitor it, govern it. The steps are known, they are countable, and they do not branch much. That is a different problem from planning an integration, and it did not need an orchestrator deciding a route through it. The APIM bot ran on LangChain, and every action against the platform was a tool call.

Two things then grew together. Every API action the bot learned added a tool, and every tool added instructions about when and how to use it. The system prompt was quietly becoming the place all procedural knowledge accumulated, which is the worst place for it: it is paid for on every request whether the turn needs it or not, and it competes with the user's actual question for the model's attention.

Because the lifecycle workflows were fixed and countable, they did not have to live there. I moved them out into skills: bounded, reusable procedures that call the underlying tools in a defined sequence. The model selected the skill, and the skill carried the workflow. That kept the system prompt focused on decisions instead of turning it into a second implementation of the product, and adding an action stopped meaning taxing every future conversation.

The second decision was credentials. Securing and publishing an API means handling them, and the obvious path is to accept them in the conversation and then defend them: mask them in logs, encrypt them at rest, keep them out of traces, and be right about all three on every future code path. That is a control you have to get right every time and only have to get wrong once.

I built a browser popup that captures credentials from the user directly and passes them to the runtime that needs them. The model never sees a credential. There is nothing to mask, nothing to scrub from a trace, and no prompt change that can leak one. The failure mode was removed rather than guarded.

Three things carried into the iPaaS bot: procedural knowledge belongs in loadable skills rather than in the prompt, context is a budget you spend on purpose, and the safest way to handle a secret is to keep it outside the model's reach. The second one is where the iPaaS side got expensive.

iPaaS: when the conversation was the unit of context

The iPaaS bot had the harder problem. A single request could cross warehouse operations, CRM, ERP, finance, payroll and whatever else the customer had connected, and it could start as one sentence and turn into a long run of lookups, mappings, decisions and confirmations.

Three things went wrong at once. The bill climbed turn over turn, answer quality drifted down as sessions got longer, and some sessions ran out of context and fell over.

Three symptoms invite three budgets. They had one cause: the conversation was the unit of context. Every turn re-sent the whole session, so a long session cost more, buried the relevant part in a larger haystack, and eventually did not fit.

The architecture I designed

A LangGraph orchestrator over two agents. A Planner that works out what the user is asking for and plans against the recipe corpus, and an Executor that carries it out against the platform.

What needs a boundary is the handoff. The Planner emits a Pydantic-validated model, serialised as JSON, and the Executor consumes that. The Executor never re-reads the user's original wording, so there is exactly one place where natural language becomes a plan, and it is typed and validated at the boundary rather than parsed hopefully on the other side. Every agent confirms with the user before it implements anything.

Diagram: a natural-language turn enters a LangGraph orchestrator, which routes to a Planner and an Executor. The Planner reads intent and slots itself and does RAG over the recipe corpus on Pinecone; it passes a Pydantic-validated JSON contract to the Executor, which acts on that rather than re-reading the user's wording. Each agent holds its own context. Every agent confirms with the user before implementing. Underneath sit a rolling summary regenerated on a token threshold and run off the hot path, checkpoints that resume a session across restarts, and a cache for repeated lookups. — The Planner is the only component that reads what the user wrote. Everything downstream acts on the validated contract, which is what makes the boundary enforceable rather than aspirational.

Three fixes I turned down

Each of these was credible, and two of them I would recommend to someone else in a different situation. That is what makes this a decision rather than a default.

A sliding window. Cheapest by far, and wrong for this workload. A user refining a workflow states the requirement early and spends the next nine turns adjusting details. A window drops the requirement and faithfully keeps the adjustments to it.

A longer-context model. Buys headroom with no new component, and raises cost per call in exactly the dimension that was already the complaint. The ceiling does not go away, it moves, and the workflow that reaches it later is in production by then.

Retrieval over the conversation. The most tempting, because retrieval was already running well over the recipe corpus. I turned it down on the failure mode. When retrieval misses over documents, an answer is worse and the user can push back. When it misses over session state, the system forgets something the user explicitly told it and proceeds confidently. I was not willing to put session state behind a recall metric.

The fourth option, leaving one shared context and attacking cost elsewhere, was never in the race. It treats the symptom that is easiest to measure and leaves the ceiling exactly where it was.

What I built instead

Isolated context per agent. Each agent gets the slice it acts on. The Executor sees the validated plan, not the transcript. The Planner sees the task and the retrieved recipes, not the Executor's tool output.

That has a second effect worth as much as the first. Once agents no longer share a context, they no longer have to share a model. The Planner does the hard reasoning and gets the strongest one, and nothing else pays for it. Per-agent model routing was not a separate initiative, it fell out of the context decision.

A token-aware summarization and memory layer, so what carries forward stays bounded as a session grows. And a cache, so the lookups every session repeats are paid for once.

What it cost, and how I paid it down

Summarization added around nine seconds when a session crossed its token threshold and the work ran inside the response path. The trigger was the amount of context, not a turn schedule: sessions grow at different rates, and a fixed cadence summarizes some too early and misses others that fill faster.

The tradeoff was lower cost, more stable answer quality, and fewer context-limit failures, against an additional wait whenever summarization ran in the response path.

Then I moved it off the hot path. The summary regenerates between turns rather than inside one, so the nine seconds stops sitting between the user and their answer. The repeated reference lookups went to cache, which takes work out of the turn entirely rather than making it faster.

How I knew it worked

Langfuse for request-level tracing. This is what made the latency visible in the first place, and it is how the token thresholds were set to something observed rather than guessed.

DeepEval for response checks, aimed at what summarization most threatens: whether an answer late in a session still reflects what was established early in it. For the retrieval path, semantic chunking and hybrid retrieval were evaluated on context precision, context recall, and answer faithfulness and groundedness.

Human review, because the failure that mattered most was a summary that read perfectly and had dropped the one constraint the user cared about. An automated check that could reliably catch that would be the harder system.

The recorded figures: context precision improved by over 60% and retrieval latency dropped 30% on the RAG-augmented planning agent, and the summarization and memory layer held answer quality as sessions grew while cutting LLM spend by around 40%. Those were measured at the time, on the platform's own tracing. I no longer have access to the systems that produced them, so I quote them as what was recorded rather than as something I can re-run for you now.

The decision underneath

Refusing to let the conversation be the unit of context. Everything else follows: bounded per-agent context, a typed contract at the handoff so the boundary is enforced rather than hoped for, model routing as a consequence rather than a project, and summarization reduced to a small addition instead of a rescue.