Review and release gates

A gate is a check that must pass before a change can proceed. CI handles cheap checks well. Use Flyte for the checks that need real data, real compute, or a person, and for changes that never appear as a diff.

Match the gate to the change

For the third kind of change there is no pull request to attach a check to. The gate has to start when the new model or dataset appears, and it has to record its verdict somewhere other than a PR status.

Gate a pull request

The GitHub integration receives the pull request event and launches a task. The receiver is the one shown in Event-driven automation. The launched task does the work and writes the result back to the pull request. This one labels the pull request by size:

env = flyte.TaskEnvironment(
    name="github-triage",
    image=flyte.Image.from_debian_base().with_pip_packages("PyGithub"),
    # Token for reading the pull request. The webhook secret is mounted on
    # the receiver app, not here.
    secrets=[flyte.Secret(key="github-token", as_env_var="GITHUB_TOKEN")],
    resources=flyte.Resources(cpu=1, memory="512Mi"),
)

@env.task
async def triage_pr(repo: str, number: int) -> str:
    """Label a pull request by size."""
    import os

    from github import Auth, Github

    client = Github(auth=Auth.Token(os.environ["GITHUB_TOKEN"]))
    pull = client.get_repo(repo).get_pull(number)
    changed = pull.additions + pull.deletions
    label = "size/s" if changed < 50 else "size/m" if changed < 500 else "size/l"
    pull.add_to_labels(label)
    return label

To make the result a required status, have the task publish a check run through the GitHub API (for example, Repository.create_check_run in PyGithub), and require that check in branch protection. GitHub accepts check runs only from a GitHub App, so authenticate with an installation token rather than a personal access token.

Running the gate as a Flyte task instead of a CI job lets it:

• Hold a GPU for only the step that needs it, with per-task resources.
• Skip the expensive work on a re-run, with caching.
• Run a thousand-case suite in parallel, with fan-out.
• Absorb a provider’s transient 503 with retries, instead of failing the build.

Gate on a person’s review

review_pr pauses a run on a condition that carries the pull request’s metadata, waits for a person to answer in the Union.ai UI, and returns a typed verdict:

review_env = flyte.TaskEnvironment(
    name="github-review",
    image=flyte.Image.from_debian_base().with_pip_packages("flyteplugins-github[review]"),
    secrets=[flyte.Secret(key="github-token", as_env_var="GITHUB_TOKEN")],
    resources=flyte.Resources(cpu=1, memory="512Mi"),
)

@review_env.task
async def gated_merge(repo: str, number: int) -> str:
    """Wait for a review decision in the Flyte UI, then act on it.

    `review_pr` creates a condition carrying the pull request's metadata and
    waits for a reviewer to answer it. The condition is stored on the backend,
    so the run survives restarts while it waits.
    """
    from flyteplugins.github import review_pr

    decision = await review_pr(repo, number, instructions="Block on missing tests.")
    if decision.is_approved:
        return f"approved by {decision.reviewer}: {decision.summary}"
    blocking = ", ".join(c.path for c in decision.blocking_comments)
    return f"{decision.verdict}: {decision.summary} ({blocking})"

See Human gates and approvals for timeouts, who may answer, and how long a run can wait.

Promote the same version

Treat a release as a promotion: the same versioned code moves between domains as it passes each gate, instead of being rebuilt for each environment.

1. On every merge, deploy to development pinned to the commit, with flyte deploy --version ${{ github.sha }}. See CI/CD deployments.
2. Run the gate there, against real data.
3. If it passes, promote the same version to staging, then to production.

Deploying the same commit twice produces the same version, and tasks already registered at that version are not registered again. Every run traces back to its commit. If you rebuild for each environment, you can no longer show that what you tested is what you shipped.

Gate agent pull requests

When an agent opens pull requests, change two things.

Credentials. Authenticate the agent as a GitHub App. It mints a short-lived installation token for each operation instead of holding a personal access token:

agent_env = flyte.TaskEnvironment(
    name="github-agent",
    image=flyte.Image.from_debian_base().with_pip_packages("flyteplugins-github[auth]"),
    # Each key maps to the environment variable `mint_installation_token`
    # reads (github-app-id -> GITHUB_APP_ID), so `as_env_var=` isn't needed.
    secrets=[
        flyte.Secret(key="github-app-id"),
        flyte.Secret(key="github-app-installation-id"),
        flyte.Secret(key="github-app-private-key"),
    ],
    resources=flyte.Resources(cpu=1, memory="512Mi"),
)

@agent_env.task
async def clone_at_head(repo: str) -> str:
    """Build an authenticated clone URL with a GitHub App installation token.

    The token expires after one hour.
    """
    import asyncio

    from flyteplugins.github import clone_url, mint_installation_token

    # mint_installation_token is synchronous; run it off the event loop.
    # It returns None if the App credentials are missing.
    token = await asyncio.to_thread(mint_installation_token)
    url = clone_url(repo, token)
    # Never return or log the token itself.
    return url.replace(token, "***") if token else url

Installation tokens expire after one hour. That is long enough for a clone or a gh pr create, and a leaked token stops working soon after.

Review load. An agent can open more pull requests than reviewers can read carefully. To keep review meaningful:

• Make an automated gate the only gate for mechanical changes, and reserve human review for changes that alter behavior.
• Test generated code before review. The code generation integration runs it in a sandbox first.

Agent frameworks covers running agent SDKs as durable tasks, and Agents covers Flyte’s own agent harness.

Gate a dataset change

Gate a dataset change like a model change, and add a schema check in front of it. Pandera validates dataframes at task boundaries, so a changed column type or an unexpected null fails at the step that introduced it. See Evaluation gates for input checks on model output.

Gates to avoid

A gate that people work around gives false assurance. Avoid these:

• A non-deterministic gate with no tolerance. If it fails one run in five, people re-run it until it passes.
• An absolute threshold. “Accuracy above 0.9” turns permanently red when the data shifts, or permanently passes. Compare against what is live instead. See Evaluation gates.
• A slow gate in front of a fast loop. A 40-minute gate belongs after merge, with the ability to roll back.

See also

• Evaluation gates for measuring quality well enough to gate on it.
• Human gates and approvals for putting a person in the loop.
• CI/CD deployments for deployment from CI.