One validation engine, many dataframes: Pandera's new Narwhals backend

Once upon a time, Pandera started as a schema validation library for pandas. Then, Polars users wanted data validation, so we added a Polars backend. Then, PySpark users wanted it, so we added a PySpark backend. Then, Ibis. Each backend is a separate implementation of the same idea: check a column's dtype, check it's non-null values, run a `greater_than`, collect the rows that failed. Each implementation was written against a different dataframe API, and each schema API required four backends, implemented four different times.

That is exactly the kind of duplication that makes projects hard to maintain, for example:

• A built-in check like `str_matches` has to be written once per backend.
• A bug fix in Polars null-handling doesn't reach the PySpark path.
• Feature completeness drifts: a capability lands for pandas and quietly never arrives for Ibis. The tracking issue that kicked this off put it plainly: feature completeness differs between these engines, and every new engine multiplies the maintenance surface.

Pandera `0.32.*` introduces a way out: an optional validation backend powered by Narwhals that expresses validation logic once and runs it natively across Polars, Ibis, PySpark SQL, and Pandas. Same public API, no data copying, opt-in. This post explains what it is, how to turn it on, and where its edges are today.

One expression API, many engines

Narwhals is a lightweight, dependency-free compatibility layer that exposes a subset of the Polars expression API on top of many underlying dataframe libraries: Polars, pandas, PyArrow, Modin, cuDF, Dask, Ibis, DuckDB, and PySpark among them. You write dataframe-agnostic logic against Narwhals expressions, hand it a native frame, and it executes on that frame's engine without converting or copying the data. Altair adopted it to drop its hard pandas dependency; a growing list of libraries use it the same way.

The insight for Pandera is that data validation is mostly expression evaluation. "Is this column an `Int64`?" "Are all values `> 0`?" "Which rows fail `isin([...])`?" These are the kinds of things the Polars expression API describes well, and Narwhals makes portable. If you can write those expressions once against Narwhals, you get validation on every engine Narwhals supports, for free, from a single code path.

That's the motivation behind PR #2223, which introduced a unified Narwhals-backed engine that replaces the separate Polars, Pyspark SQL, and Ibis backends with one shared implementation. The 14 built-in checks (`equal_to`, `greater_than`, `in_range`, `isin`, `str_matches`, `str_contains`, `str_length`, and others) are now defined a single time as Narwhals expressions instead of once per backend. Fix a check once, every engine gets the implementa.

The design principle that makes this safe to adopt is the one the PR holds to strictly: Narwhals is internal plumbing, not a user-facing API. You keep passing `pl.DataFrame`, `pl.LazyFrame`, `ps.DataFrame`, or `ibis.Table`. Pandera routes them through Narwhals internally. Your call sites don't change.

What Pandera 0.32.* ships

The unified backend covers three public integrations today:

The distinction that matters for behavior is eager vs. lazy. An eager `pl.DataFrame` runs the full schema-and-data validation. A `pl.LazyFrame` stays lazy: Pandera does not force a `.collect()`, and only materializes a bounded set of failure cases if something actually fails. The SQL-lazy engines (Ibis, DuckDB, PySpark, PySpark Connect, SQLFrame) push validation down to the query engine and likewise avoid premature materialization.

Crucially, none of your validation code changes. The same DataFrameSchema and DataFrameModel you already write is what runs on the new backend.

Using it

Install Pandera with the `narwhals` extra plus whichever engine you use:

pip install 'pandera[narwhals,polars]'   # Polars
pip install 'pandera[narwhals,ibis]'     # Ibis
pip install 'pandera[narwhals,pyspark]'  # PySpark SQL

The backend is off by default (more on why below). Turn it on with an environment variable at process start:

export PANDERA_USE_NARWHALS_BACKEND=True
python your_script.py

Or programmatically, before you build a schema:

import pandera
import pandera.polars as pa

pandera.set_config(use_narwhals_backend=True)

One subtlety: backends register lazily, when a schema is first constructed or validated, and they read `use_narwhals_backend` at that point, not at import time. So set the config before you build your schema, and the right backend gets wired up.

From there it's ordinary Pandera. The object-based `DataFrameSchema` API looks exactly the same:

import polars as pl

schema = pa.DataFrameSchema(
    {"x": pa.Column(int, pa.Check.greater_than(0))}
)

schema.validate(pl.DataFrame({"x": [1, 2, 3]}))

And the class-based `DataFrameModel` API works the same way:

class Transactions(pa.DataFrameModel):
    transaction_id: int = pa.Field(unique=True)
    amount: float = pa.Field(gt=0)
    currency: str = pa.Field(isin=["USD", "EUR", "GBP"])

Transactions.validate(df)

To use the Narwhals backend on the other supported frameworks, simply change the import:

import pandera.ibis as pa  # for ibis table validation
import pandera.pyspark as pa  # pyspark dataframe validation

Then point the same schema at an Ibis table or a Spark DataFrame and the validation logic is identical. That's the whole point: write it once, validate everywhere.

Custom checks that travel

Built-in checks are portable for free. For custom checks, the `Check` constructor takes a `native` flag that controls what frame your function receives:

# native=True (default): your function gets the engine-native frame.
# Backend-specific, but you can use the full Polars/Ibis/Spark API.
pa.Check(lambda df: df["age"] > 0, native=True)

# native=False: your function gets a Narwhals-wrapped frame, so the
# same check runs unchanged across every supported backend.
pa.Check(lambda data: data.frame["age"] > 0, native=False)

If you want a check that works on Polars and PySpark without rewriting it, write it against Narwhals with `native=False`. If you need an engine-specific escape hatch, `native=True` hands you the raw frame.

Why it's opt-in, and off by default

It would be tempting to just flip everyone over. We deliberately didn't, and PR #2314 is where that decision lives. Narwhals is a transitive dependency of many popular packages (if you have Altair installed, you already have Narwhals), so simply detecting that Narwhals is importable and activating the new backend would silently change validation behavior for people who never asked for it. Silent behavior changes on a validation library are the last thing you want; a validation library's whole job is to be predictable.

So the native backends remain the default, the public API (`import pandera.polars as pa`, and the rest) is unchanged, and you opt into the Narwhals engine explicitly. Changing the config at runtime re-registers the backends and emits a warning so the swap is never invisible. You get the new engine when you decide you want it, not because a dependency dragged it in.

What you get, and what you don't yet

The upside is structural: one implementation to maintain instead of four, built-in checks defined once, lazy frames that stay lazy, and custom checks that can be made portable across engines. But this is a new backend, and it isn't yet at full parity with the mature native backends. The current known gaps, straight from the docs:

None of these affect the native backends, which keep working exactly as before. They're the boundary of where the unified engine is today. Each one is a well-scoped follow-up rather than a design dead end. If your schemas lean on coercion or hypothesis-based data generation, stay on the native backend for now; if you're validating across multiple engines and want one consistent code path, the Narwhals backend is ready to try.

Conclusion: one engine, many dataframes

The Narwhals backend is Pandera paying down its own maintenance debt in a way that lands directly on users:

One implementation, not four: the 14 built-in checks are defined once as Narwhals expressions, so a fix or a new check reaches Polars, Ibis, and PySpark SQL at the same time instead of drifting apart across per-engine backends.

Same API, native execution: you keep writing `DataFrameSchema` and `DataFrameModel`, keep passing `pl.DataFrame`, `ibis.Table`, or a Spark DataFrame, and validation runs on that engine without copying data. Narwhals is internal plumbing, not something you import.

Opt-in: it's off by default in `0.32.*`, so a transitive Narwhals dependency can't silently change how your schemas behave, and the parity gaps (coercion, schema IO, hypothesis strategies) are documented rather than papered over.

Write once, validate on every supported engine. That's the same pitch that got Pandera onto Polars, PySpark, and Ibis in the first place, only this time without maintaining a fourth copy of the logic.

Try it

# Polars
pip install 'pandera[narwhals,polars]'
export PANDERA_USE_NARWHALS_BACKEND=True

# or Ibis / PySpark SQL
pip install 'pandera[narwhals,ibis]'
pip install 'pandera[narwhals,pyspark]'

Then point an existing `DataFrameSchema` at a Polars, Ibis, or Spark frame and watch the same validation run on all three. The Narwhals backend guide has the full reference, the implementation lives in `pandera/backends/narwhals`, and if you hit an edge, open an issue.