Agents That Survive Production: Rebuilding 21 Design Patterns on Flyte

Four months porting an agent-design book to a workflow engine, and what it taught me about why agents break once they leave the demo.

Why I did this

Late last year, Antonio Gulli shared a 400-plus-page draft of Agentic Design Patterns for community review. I read it with a sense of relief. The agentic AI space moves so fast that a lot of what you pick up is framework-specific detail that's outdated within a quarter. I wanted something more durable: a clear mental model of how agents are actually architected. How a complex goal gets decomposed into steps. How specialized agents hand work to one another. How memory, tools, and planning fit together, and how all of it maps onto real use cases.

The book gave me that model. So I set myself an exercise: port every code example to Flyte V2 and make each one as Flyte-native as I could. Not Flyte plus LangGraph but Flyte itself, carrying the agent. Flyte works well alongside LangChain, LangGraph, and the rest, but I specifically wanted to find the edges. How far can a workflow engine take an agentic workload before you reach for a dedicated agent framework?

The answer took about four months of early mornings (my 5-to-6am study slot) and two complete rewrites. The first came when the Flyte Devbox shipped, so that anyone could clone the repo and run all 21 patterns on a free backend. The second came when the `flyte.ai` module landed, which let me delete a lot of hand-rolled scaffolding (tool loops wired together with `@flyte.trace`, dataclasses shuttled between functions) and replace it with first-class agent primitives. This post is what I learned in the process, with a particular focus on the thing the book kept circling back to and the thing Flyte turned out to be unusually good at: reliability.

Agents are a systems problem, not a prompt problem

In Why Do Multi-Agent LLM Systems Fail?, the Multi-Agent System Failure Taxonomy (MAST) study accepted to NeurIPS 2025, researchers hand-annotated 200+ execution traces across seven popular multi-agent frameworks and built the first empirical taxonomy of how these systems break. They found 14 distinct failure modes, grouped into three categories: specification and system-design issues, inter-agent misalignment, and task verification.

The single largest category, roughly 44% of failures, wasn't the model falling short. It was specification and system design: agents disobeying the task contract, repeating steps, failing to recognize when they were done. A related study, Where LLM Agents Fail and How They Can Learn From Failures, looks at the same problem from another angle: in multi-step tasks, a single early error tends to propagate, quietly skewing every downstream decision until the whole trajectory fails.

Read together, they point at the same thing. The LLM-and-tools loop, the part everyone calls "the agent," is the easy part. The hard part is everything around it. What happens when step four of five hits a rate limit? Where does the conversation live when the process restarts? How do you stop one bad tool call from cascading? Who approves the irreversible action? How do you even see a failure that didn't throw an exception?

Those aren't prompt-engineering questions. They're orchestration questions: retries, timeouts, durable state, isolation, observability, human gates. In other words, the failure modes that dominate real agent systems are exactly the problems a mature workflow engine has spent years solving for data and ML pipelines. That's the bet this repo tests.

What changes when you define an agent in Flyte

Start with the most familiar pattern: an agent that calls tools. In LangChain you reach for `@langchain_tool` and an `AgentExecutor`, and you hand control to `.ainvoke(...)`. It works, but the loop is a black box, your state lives in an in-process scratchpad, and when it fails you're left with logs.

In Flyte, a tool is a plain function with a decorator, and the schema is inferred from your type hints and docstring:

@tool
def get_stock_price(ticker: str) -> dict:
    """Get the current price, daily change %, and market cap for a stock ticker.

    Args:
        ticker: The stock ticker symbol, e.g. 'AAPL' for Apple.
    """
    ...

The agent itself is a single declarative object that bundles the model, the instructions, the tools, and the loop guardrails:

router_agent = Agent(
    name="request-router",
    model="claude-haiku-4-5",
    instructions="You are a customer-service router...",
    tools=[handle_booking, handle_info, ask_clarification],
    max_turns=2,
    parallel_tool_calls=False,
)

The earlier, pre-`flyte.ai` version of the tool-use notebook was about 120 lines: a system prompt, a traced dispatcher, an LLM-call helper, and an explicit `for turn in range(max_turns)` loop that parsed stop reasons and threaded the message list by hand. As the notebook now puts it, the `Agent` object "collapses all of it into a single declarative object": you get the LLM-tool loop, the `max_turns` guard, optional parallel tool calls, and a typed `AgentResult` (`.summary`, `.error`, `.attempts`) for free. Crucially, the loop is managed but inspectable, not a black box.

Here's where an orchestration engine changes things: a tool doesn't have to run in-process. Define it as a Flyte task instead of a `@tool`, and every call the agent makes becomes a nested, retriable, resource-scoped child action with its own pod, its own image, its own CPU and memory, its own secrets, and its own retries:

@env.task
async def handle_booking(request: str) -> str:
    """Handle requests about booking or changing flights and hotels."""
    ...

This reframes whole patterns. Routing, for example, stops being a hand-written classifier feeding an `if/else` dispatcher. Routing becomes tool selection, where each route is a task-tool with a docstring describing when to use it, and the harness does the classify-and-dispatch step for you. You choose, tool by tool, between a cheap in-process call and a fully isolated, observable, retriable unit of work. That dial doesn't exist in an in-process framework.

Durable by construction

The MAST taxonomy's second-largest failure category is inter-agent misalignment: agents losing the thread, dropping context, mangling hand-offs. This is where it helps that in Flyte, everything is a durable, typed artifact.

Take memory. LangChain's `ConversationBufferMemory` lives in the process; restart the process and it's gone. Flyte's `MemoryStore` is keyed to a session id and persisted to object storage, so any task, on any pod, can reload the same store from just the key. You wire it straight into the agent, which then manages the transcript for you:

store = await MemoryStore.get_or_create.aio(key=session_key)

agent = Agent(
    name="travel-agent",
    model="claude-haiku-4-5",
    instructions=system,
    memory=store,
)
result: AgentResult = await agent.run.aio(user_message)

await store.write_json.aio("profile.json", {...}, actor="chat_turn",
                           reason="profile update")
await store.save.aio()

And because it's real storage rather than a dict, it comes with the things real storage needs: optimistic concurrency (a stale write raises `ConcurrencyError` via an `expected_sha` check) and an audit log recording the actor, a sha256, and a timestamp for every write. Memory becomes something you can inspect and trust, not just append to.

The same principle, typed and persisted hand-offs, shows up in planning and multi-agent collaboration. In the planning pattern, the decomposed plan is a `ResearchPlan` dataclass written to object storage between the planning task and the execution task. If the execution task fails (e.g., API rate limit on step 4 of 5), you can retry only the execution task. The expensive plan generation step is not re-run. That handles cascading failure in the architecture itself, instead of bolting on retries.

Multi-agent collaboration follows suit. Each agent emits a typed dataclass rather than a raw string, so the next agent references fields by name and Flyte stores every hand-off durably. Parallelism, meanwhile, needs no special primitive. It's just asyncio.gather:

fact_check, seo = await asyncio.gather(
    fact_checker_agent(article=article),
    seo_optimizer_agent(article=article),
)
return await synthesizer_agent(article=article, fact_check=fact_check, seo=seo)

Reliability as a first-class API

Map the MAST failure categories onto Flyte features and they line up almost one-to-one.

Recovery. Transient failures (the rate limits and network hiccups that the research reports flag as task-killers) are handled at the infrastructure layer, not by asking the LLM to cope:

@recovery_env.task(retries=2, timeout=timedelta(minutes=2))
async def location_agent(address: str) -> RecoveryResult:
    ...

`retries=` gives you automatic exponential backoff; `timeout=` bounds the work; and typed errors (an OOMError, say) can be caught and retried with `.override(resources=...)`. Recovery is an infrastructure attribute, not an LLM decision. When retries are exhausted, the pattern degrades gracefully into a typed, inspectable result instead of an exception that vanishes into a log. `escalate=True` signals a human is needed, and the field shows up directly in the Flyte UI.

Isolation. One of the more interesting reliability hazards turns up in the learning-and-adaptation pattern, where the agent writes its own code and then runs it. An evolutionary loop generates candidate programs, scores them, and mutates the survivors, which means you're executing model-generated code that you really don't want running in your main process. Flyte's answer is a sandbox: each candidate runs in its own ephemeral, isolated container with a typed input/output contract.

eval_sandbox = flyte.sandbox.create(
    name="code-evaluator",
    code=_SANDBOX_EVAL_CODE,
    inputs={"code": str, "fn_name": str, "test_cases_json": str},
    outputs={"score": float, "error_msg": str, "passed": int, "total": int},
    cache="disable",
)

The agent driving the loop is a `CodeModeAgent.` Rather than calling tools one at a time, it emits a single Python script and runs it, and here it goes a step further by designing its own test cases for each candidate:

eval_agent_adaptive = CodeModeAgent(
    tools=[run_tests_tool],
    model="claude-sonnet-4-6",
    max_retries=2,
    system_prompt_prefix="You are an adaptive code evaluator...",
)

The isolation level here is a dial, not a rewrite. On the Devbox the loop stays dependency-free by scoring candidates in-process through a lightweight Monty sandbox; on a Union cluster, the very same code runs container-per-candidate via `flyte.sandbox.create`, isolating the execution environment for each candidate. You choose the trade-off based on how much you trust the code you're about to execute.

Verification. MAST's third category is inadequate verification: nobody checking the agent's work. The guardrails pattern answers with cheap layers first. A pre-flight check, implemented by subclassing Agent, rejects obvious violations in microseconds with no LLM call and no tokens spent:

@dataclass
class GuardedAgent(Agent):
    banned_terms: tuple[str, ...] = ()

    async def run(self, message: str, history: list | None = None) -> AgentResult:
        hit = next((t for t in self.banned_terms if t.lower() in message.lower()), None)
        if hit is not None:
            return AgentResult(summary="I can't help with that request.",
                               error=f"Blocked by input guardrail: '{hit}'.", attempts=0)
        return await super(GuardedAgent, self).run.aio(message, history)

That composes with a mid-tier LLM policy screen and, at the top, a human gate. It's worth being clear about what this layer is and isn't, the way the notebooks are: this pre-flight guard is a literal substring match, trivially bypassed by paraphrase; a cheap first filter, not a jailbreak defense. Being upfront about that matters, because a guard that promises more than it delivers is precisely how the silent failures in the research slip through.

Human-in-the-loop. For irreversible actions, escalation is enforced by the runtime, rather than by an outer if branch you have to remember to write:

@tool(requires_approval=True)
def issue_refund(order_id: str, amount_usd: float) -> str:
    """Issue a refund for an order. High-risk and irreversible; requires human approval."""
    ...

When the agent calls this tool, the run pauses until a person approves or denies it in the UI.

Observability and evaluation. One result surprised me. The evaluation pattern is the one place I didn't use the agent harness at all. Evaluation is a deterministic pipeline (run the system over a dataset, score each output, aggregate), so it's a plain task-plus-`flyte.map` job with exact token counts pulled from `response.usage`, `time.monotonic()` latency, and an LLM-as-judge producing structured scores. The Agent is the subject being evaluated, not the right tool to implement the evaluation. Wrapping an agent around your scoring would only make the results less reproducible, and reproducible scoring is exactly what you need to catch the silent failures, the ones that never throw.

What didn't need the harness, and why that's the point

About half the patterns never touch `flyte.ai.agents` at all. Prompt chaining, parallelization, reflection, plain exception handling, resource-aware model routing, evaluation, exploration: these stay on ordinary task composition, and each notebook ends with a short note on why the agent abstraction wouldn't have been a win.

I think that restraint is the most important thing the exercise surfaced. Flyte never pushed me to wrap an "agent" around a problem that was really a deterministic pipeline. When the work was a fixed pipeline, I wrote a pipeline. When it genuinely needed an autonomous LLM-tool loop, I reached for `Agent`. The `flyte.ai` rewrite mattered precisely because it gave me native abstractions for the agentic cases (memory, tools, approval gates), so I could stop hand-rolling them, while leaving the deterministic cases as plain, reliable tasks. Choosing the right altitude for each pattern, rather than forcing one abstraction over all of them, is most of what I mean by a system-design mindset.

Closing

The gap between an agent that demos and an agent that ships is mostly the gap MAST measured: specification, coordination, and verification. System design, not raw model capability. Building all 21 patterns on Flyte helped me have a better grasp of the fundamental ideas behind AI Agents, and demonstrated that an orchestration engine is a genuinely good place to build agents, because the primitives agents need in production (retries, timeouts, durable typed state, isolation, human gates, reproducible evaluation) are the primitives a workflow engine already has.

If you want to see for yourself, the repo ships with the Devbox config so you can run every pattern on a free Flyte backend. Read it next to Gulli's Agentic Design Patterns: the book gives you the architecture, and the notebooks show you what it looks like when reliability is a first-class API instead of an afterthought.

Sources

• Cemri et al., Why Do Multi-Agent LLM Systems Fail? (MAST), NeurIPS 2025 Datasets & Benchmarks Track. arXiv:2503.13657
• Where LLM Agents Fail and How They Can Learn From Failures. arXiv:2509.25370
• Antonio Gulli, Agentic Design Patterns. Springer
• Flyte V2 documentation and the Flyte Devbox. union.ai/docs/v2