Pandera 0.36: columns that say where they come from

A pandera schema has always been good at saying what a column is: its dtype, whether it can be null, the range its values have to fall in. It's never had much to say about where a column comes from. If `n_words` is computed from `body`, that fact lives in a closure somewhere upstream, or in a parser that happens to run before validation, and the schema just sees a column of integers that shows up on time.

That gap matters more once the derivation gets expensive. When the thing producing a column is a model reading a support ticket and deciding which team should handle it, you want the schema to know about it. You want it to tell you what the model will be asked, how many requests that costs, and whether the answers are any good, before you find out from a bill or a misrouted queue.

Pandera 0.36.0 does both halves. `ParsedColumn` and `ParsedField` put the derivation on the column, and `pandera.decisions` uses that to fill columns by asking a decision model typed questions.

pip install 'pandera[pandas,decisions]==0.36.0'

Derived columns, declared on the column

A `ParsedField` is an ordinary typed field that also states how its values are produced. `Config.parser_source` names the column it reads from by default:

import pandas as pd
import pandera.pandas as pa

class Tickets(pa.DataFrameModel):
    body: str
    n_words: int = pa.ParsedField(
        parser=lambda s: s.str.split().str.len(),
        source="body",
        ge=1,
    )

Tickets.validate(pd.DataFrame({"body": ["a b c", "d e"]}))
#     body  n_words
# 0  a b c        3
# 1    d e        2

You don't pass `n_words` in. Validation fills it, coerces it to `int`, runs the `ge=1` check against it, and allows it under `strict=True`. From that point on it's a column like any other. The object-based API is the same idea, with `pa.ParsedColumn(int, source="body", parser=...)` in a `DataFrameSchema`.

Declaring the source and target buys you more than documentation. Parsers now run in dependency order rather than list order, so a column derived from another derived column just works, and a cycle raises `SchemaInitError` naming the columns involved. A parser reading a column that isn't there raises `ParserSourceError`, which names the parser, the missing column, and the columns that are present, instead of a bare `KeyError`.

One thing to watch when you upgrade: `source` and `target` are now real parameters on `pa.Parser`. They used to be forwarded to your parser function as keyword arguments, so a parser function that took a kwarg by either name will break.

Filling columns with a decision model

The `parser=` slot also accepts parser objects, and that's what `pandera.decisions` plugs into. A decision model, like TypeSafe's Jev, doesn't generate text. It takes a piece of state plus a set of typed questions and returns typed answers, so it can't give you a value outside the domain you asked about. That makes the column's declared type the answer domain, which is exactly what a schema already describes.

Here's the support-ticket triage example from the release notes:

import enum
import pandera.decisions as decisions

class Department(str, enum.Enum):
    billing = "billing"
    """Payment, invoices or subscription issues."""
    technical = "technical"
    """Bugs, outages or integration problems."""
    sales = "sales"
    """Pricing, plans or account expansion."""

class Frustration(enum.IntEnum):
    calm = 0
    """Calm, simply stating facts."""
    annoyed = 1
    """Frustrated but civil."""
    angry = 2
    """Very angry, strong language."""

class Triage(pa.DataFrameModel):
    ticket_body: str
    department: Department = pa.ParsedField(
        description="Which team should handle this ticket",
        parser=decisions.Choice(),
    )
    frustration: Frustration = pa.ParsedField(
        description="How frustrated the customer appears",
        parser=decisions.Score(),
    )
    is_urgent: bool = pa.ParsedField(
        description="The message conveys time-sensitivity",
        parser=decisions.Noul(),
    )

    class Config:
        parser_source = "ticket_body"

There are three question types. `Choice` picks one option from an `Enum`, `Literal` or categorical. `Score` places the row on an ordered scale, typically an `IntEnum` with two to ten levels. `Noul` is a yes/no that returns a `bool`, or the probability itself if you type the column as `float`.

Nothing in that model is written twice. The `Choice` options come from `Department`'s members, the `Score` rubric comes from `Frustration`'s member docstrings, and each question's instructions come from the field's `description`. You can check exactly what will be asked without credentials or network access:

decisions.questions(Triage)["department"]
# Choice(instructions='Which team should handle this ticket',
#        criteria=(('billing', 'Payment, invoices or subscription issues.'),
#                  ('technical', 'Bugs, outages or integration problems.'),
#                  ('sales', 'Pricing, plans or account expansion.')),
#        values=('billing', 'technical', 'sales'), ...)

That makes the prompt reviewable in a pull request and testable in CI. It also means the docstrings are where answer quality comes from, so they're worth writing carefully, and they're versioned with the type.

The question type is named rather than inferred, which catches a class of mistake early. Put `Score()` on `Department` and you get this when the schema is built, before any request goes out:

SchemaInitError: column 'department' is filled by Score() but its type is
unordered, so there is no scale to rate against. Use an IntEnum, pass
dtype_kwargs={'ordered': True}, or use Choice() if the options are not ranked.

Who answers is configuration

The schema says what to ask. Which model answers is a runtime setting, so the same `Triage` class runs against a deterministic mock in CI and a real model in production without being edited:

decisions.set_provider("typesafe:jev-1.13.0")    # hosted Jev
decisions.set_provider("ollaya:decider:2b")      # open-weight model on your own hardware

with decisions.provider(decisions.MockProvider(seed=7)):   # scoped, offline, free
    triaged = Triage.validate(tickets_df)

There's deliberately no default. Asking a model costs money and time, so validating a decision schema with no provider configured raises `DecisionsConfigError` naming the columns rather than quietly calling out to someone's API. The shipped providers are `MockProvider`, `RecordingProvider` and `ReplayProvider` for tests (record real answers once, replay them offline), `TypeSafeProvider` for hosted Jev, and `OllayaProvider` for open-weight decision models you serve yourself. A provider is any object with `id`, `model_version`, `compile`, `decide` and `limits`, and `register_provider` makes yours reachable by name.

Providers declare what they can be asked, too. A small local model might take fewer options per question than a hosted one, and the schema is checked against that as soon as the provider is known, so a 180-option `Choice` sent to a model that accepts 125 fails with `ProviderCapabilityError` before the first row is sent.

Seeing the cost before you pay it

Decision requests bundle one state with many questions, and adding a question costs tokens but very little latency. So columns that share a provider, a source and an error policy are filled with a single request per row, not one per column. `decisions.plan` reports that before anything is sent:

with decisions.provider(decisions.MockProvider(seed=7)):
    print(decisions.plan(Triage, tickets_df))
# Plan(rows=3, batches=1, requests=3, questions=3, est_input_tokens=14, est_cost=unknown)

Three rows and three derived columns come to three requests. The cost is `unknown` here because the mock doesn't declare a price, and pandera reports unknown as `None` rather than zero. After validation, `decisions.stats(triaged)` tells you what actually happened: rows, requests made, requests served from cache, seconds spent, and the exact model version.

Caching matters because dataframe pipelines re-run constantly. Pass `cache="sqlite:///.pandera_decisions.db"` to a parser and answers are keyed on the provider, the resolved model version, the row's state and the question itself, so rewording a `description` correctly invalidates the old answers. Pin a model version when you cache, since `jev-latest` moves.

Checking the answers

The answers are ordinary column values, so ordinary pandera checks apply to them. A decision model can't return an out-of-domain value, so the useful thing to check is the shape of the distribution rather than any single answer:

class Triage(pa.DataFrameModel):
    ...

    @pa.dataframe_check
    def routing_is_not_degenerate(cls, df):
        return df["department"].value_counts(normalize=True).max() < 0.9

Confidence is a column too. `Choice(abstain_below=0.55)` on a nullable field turns answers the model was unsure about into nulls instead of guesses, and a `Confidence("department")` field exposes the score so you can put `ge=0.7` on it and get pandera's normal failure-case report for every row that falls short. `Confidence` reads an answer the named column already paid for, so it adds no requests.

A noul is a bare probability in every implementation so far, so it has no separate confidence, and pandera refuses `abstain_below` on a noul rather than letting a floor exist that never fires. Type the column as `float` and check the probability directly if you need that.

Finally, a decision model can judge a column that already exists. `decisions.Holds("The description is a coherent description of a product in the stated category", context=[...], min_probability=0.85)` is a regular `Check`, so `lazy=True`, failure cases and `raise_warning` all work as usual, and `PANDERA_DECISIONS_ENABLED=0` lets a schema carrying semantic checks run offline with a warning.

Limitations

`ParsedColumn` and `ParsedField` ship for the pandas backend in this release. The decisions examples in this post ran on `MockProvider`, whose answers are valid for each question's domain but meaningless, which is why the output above shows plans, questions and errors rather than triaged tickets. Quality differs between decision models even when the interface is the same, so before you switch production traffic from one provider to another, run the same schema under both and compare the distributions.

Also in 0.36

`Enum` and `Literal` annotations now map to categorical dtypes (#2515), which is what lets an enum-typed decision column come back as a proper categorical. The release also adds pluggable data-synthesis backend primitives, fixes a remote-code-execution issue in TensorDict loading (GHSA-vpwq-239f-3cf7), speeds up Polars schema reads, and includes a batch of IO and backend fixes from nine first-time contributors. Thanks to everyone who sent a PR.

Get started

pip install 'pandera[pandas,decisions]==0.36.0'

The derived columns guide covers `ParsedColumn` and `ParsedField`, the decision parsing guide covers question types, providers, caching and confidence, and the tutorial walks through a ticket-triage schema end to end with `MockProvider`, so you can run all of it without an API key.