Human-in-the-Loop Approvals for Any Pipeline or Agent

AI engineering tip of the week: Human-in-the-Loop Approvals for Any Pipeline or Agent

Some steps in AI workflows should not run without a person saying yes. Like promoting a model to production, issuing a refund, or letting an agent send the email it just drafted.

The common way to get a human in the loop is a polling loop, a sidecar app, or a Slack bot that writes to a database the pipeline checks every thirty seconds. `flyte.new_condition()` replaces all of that. The task suspends, the run shows a prompt in the UI, and when someone answers, the task resumes with their answer. You can also have it send a Flyte notification to slack, email, etc.

Gate a deploy on approval

from datetime import timedelta
import flyte
from flyte.errors import ConditionTimedoutError

env = flyte.TaskEnvironment(name="approval-demo")

@env.task
async def release(model: str = "fraud-v7") -> str:
    metrics = await evaluate(model)

    approval = await flyte.new_condition.aio(
        "prod_deploy",
        prompt=(
            f"## Deploy {model} to production?\n\n"
            f"- accuracy: **{metrics['accuracy']}**\n"
            f"- p50 latency: **{metrics['p50_latency_ms']} ms**\n"
        ),
        prompt_type="markdown",
        data_type=bool,
        timeout=timedelta(hours=4),
    )

    try:
        approved = await approval.wait.aio()
    except ConditionTimedoutError:
        return f"no decision on {model} within 4 hours, holding the release"

    if not approved:
        return f"{model} rejected by reviewer"
    return await deploy(model)

`new_condition` registers the gate and `wait` suspends the task. The prompt renders as markdown, so you can put the evaluation numbers right in front of the reviewer instead of making them hunt through logs.

The run is genuinely paused, not busy-waiting. It survives restarts while it waits, and it costs nothing while nobody has answered.

Answer from wherever you are

In the Flyte UI the condition shows up as a form on the run. From the terminal, list the run's conditions to get the action ID, then signal it:

flyte get condition 
flyte signal condition   true

Leave the value off and the CLI shows the prompt and asks you for a typed answer.

Or from Python, which is how you wire it to a Slack button or a webhook handler:

import flyte.remote

condition = flyte.remote.Condition.get("prod_deploy", run_name="")
condition.signal(True)

Running with `--local`, the prompt appears in your terminal and you type the answer. Same code, no cluster needed to develop it.

Collect a value, not just a yes

`data_type` can be `bool`, `int`, `float`, or `str`. A condition can ask for a threshold, a reason, or a free-form note:

reason = await flyte.new_condition.aio(
    "deploy_reason",
    prompt=f"Why are we shipping {model} today?",
    data_type=str,
)
note = await reason.wait.aio()

Whatever the human types comes back as a typed Python value and flows downstream like any other task output.

Timeouts with a default

Reviewers go home. Give the condition a `timeout` and catch `ConditionTimedoutError` to fall back to a safe default, or to hold the release as above:

cond = await flyte.new_condition.aio("threshold", data_type=float, timeout=timedelta(minutes=30))
try:
    threshold = await cond.wait.aio()
except ConditionTimedoutError:
    threshold = 0.5

Notify an external system

Pass a `webhook` and the backend POSTs to your URL when the condition is created, with a callback URI the external system can hit to answer:

condition = await flyte.new_condition.aio(
    "external_approval",
    prompt="Waiting for approval from the ticketing system",
    data_type=bool,
    webhook=flyte.ConditionWebhook(
        url="https://hooks.example.com/approvals",
        payload={"callback": "{callback_uri}", "message": "Pipeline is waiting for approval"},
    ),
)

That is the whole integration surface for Slack, PagerDuty, Jira, or an internal tool.

Put a human between an agent and a sensitive tool

This is where conditions earn their keep as an agent guardrail. A tool is a Flyte task, and a Flyte task can suspend on a condition. So an agent can decide to call `issue_refund`, and a human decides whether it actually happens:

@tool
@env.task(retries=3)
async def issue_refund(account_id: str, amount_usd: float) -> str:
    """Issue a refund. Requires human approval before it runs."""
    condition = await flyte.new_condition.aio(
        f"approve_refund_{account_id}",
        prompt=f"Approve a ${amount_usd:.2f} refund to account {account_id}?",
        data_type=bool,
    )
    if not await condition.wait.aio():
        return f"Refund to {account_id} was declined by a human reviewer."
    return f"refunded ${amount_usd:.2f} to account {account_id}"

The agent sees a decline as an ordinary tool result and carries on. No agent SDK has an equivalent, because none of them can suspend a process for a day and pick it back up.

Where this fits

Human-in-the-loop comes up anywhere a pipeline or an agent is about to do something a person should see first:

• Model factories. An agent retrains, evaluates, and proposes a promotion. The condition puts the eval numbers in front of a reviewer, and only a yes moves the model to production. The approval is recorded on the run, so you have an audit trail for free.
• Agents that act in the world. Sending the email, merging the PR, issuing the refund, placing the order. The agent plans and drafts freely, and a human gates the one tool call that can’t be undone.
• Expensive jobs. A condition before a large training run or a full backfill, with the cost estimate in the prompt. Nobody kicks off a ten thousand dollar job by accident.
• Destructive data steps. Overwriting a gold table, deleting old artifacts, or changing a schema. Pause, show the diff, and wait.
• Labeling and active learning. A loop that trains, finds the samples it’s least sure about, and asks a person to label them with a typed answer that flows back into the next round.
• Regulated workflows. A two-person rule for clinical, financial, or safety-critical releases, where the condition’s answer and the reviewer’s note become part of the run’s record.
• Research agents. A long-running agent that checks in at a decision point and asks which direction to take next, then resumes with the answer.

Why this matters

• One call: `new_condition` plus `wait`, no sidecar app or polling loop
• Truly paused: The run suspends, survives restarts, and burns nothing while waiting
• Typed answers: bool, int, float, or str flow downstream as normal outputs
• Answer anywhere: UI form, CLI, Python, or a webhook callback
• Agent-safe: Gate any tool behind a person without changing the agent loop

If you used the `flyteplugins-hitl` package, conditions replace it. The migration is a rename from `hitl.new_event` to `flyte.new_condition`.

Full docs: https://www.union.ai/docs/v2/flyte/user-guide/tasks/task-programming/conditions/

See what's happening in the Flyte Community:

Latest from the blog

• Union Artifacts part 1: naming the things your tasks produce - Read on Union
• Flyte Devbox on the Apple Silicon GPU: model serving and durable training on your Mac - Read on Union
• Rerun, Recover, Fork: Three Verbs for a Run That Already Failed - Read on Union
• Inside Union: Resource-Aware Scheduling for Flyte 2.0 - Read on Union
• It’s now super easy to start using Union.ai - Read on Union
• Union.ai Launches Self-Service Access on AWS Marketplace to Enable Sovereign AI - Read on Union

Recent talks & recordings

• Flyte + Apple Silicon for durable local AI workflows: Your Mac Is Now an AI Devbox - Watch on YouTube
• Union: AI Infrastructure Made Easier - Watch on YouTube
• Flyte 2: The Durable Runtime Built for AI - Watch on YouTube
• When the Pipeline Breaks: Building ML Infrastructure for Biotech R&D | Session 1 - Watch on YouTube
• LLM fine-tuning with GRPO - Watch on YouTube

Upcoming events

• [SF] Own Your AI: Build Your First Model Factory - Hack Night SF | Nov 3 - RSVP on Luma

Releases & updates

• Flyte 2 Is Generally Available: The Durable, Open-Source AI Runtime - Read on Union.ai

<div class="button-group is-center"><a class="button" href="https://www.union.ai/docs/v2/flyte/user-guide/run-modes/running-devbox/">Download Devbox</a></div>

From the community

• AI Book Club: Reinforcement Learning from Human Feedback (RLHF) - RSVP on Luma

That's all for this week! - Sage Elliott