Selected is not filtered

In Feast's local compute engine there is a node whose job is to enforce the single property a feature store exists to provide. Until two days ago, the part of it that does so read, in full:

if ENTITY_TS_ALIAS in self.backend.columns(df):
    # filter where feature.ts <= entity.event_timestamp
    df = df[df[timestamp_column] <= df[ENTITY_TS_ALIAS]]

The comment states the guarantee precisely: a feature row may contribute to a training example only if it was already knowable at that example's timestamp. The line beneath it does not enforce the guarantee. It hands a boolean mask to a dataframe library and trusts the library to understand what was meant — and the two libraries this engine supports do not understand the same thing.

Feast 0.67.0, released on 13 October, fixes it. The commit is titled "Apply point-in-time filter correctly with the Polars backend," and the project's own description of the defect is admirably exact: "polars treats a boolean Series passed to DataFrame.__getitem__ as a column selection, so with the Polars backend every row was kept (including feature rows newer than the entity timestamp) and columns were dropped instead; with a TTL the node then failed with ColumnNotFoundError."

Two paragraphs into Feast's own introduction sit the two promises this collects on. The first: "Avoid data leakage by generating point-in-time correct feature sets so data scientists can focus on feature engineering rather than debugging error-prone dataset joining logic. This ensures that future feature values do not leak to models during training." The second, immediately after: decouple machine learning from data infrastructure, "ensuring models remain portable as you move from training models to serving models, from batch models to real-time models, and from one data infra system to another." Portability is sold as the mechanism by which the first promise survives a change of engine. Here it was the mechanism by which the first promise was voided.

The dispatch is worth following, because it is not a careless library. In polars, a Boolean-dtype Series handed to DataFrame.__getitem__ routes to _select_columns_by_mask, which raises unless the mask's length equals the frame's width and otherwise returns the columns whose mask position is true. Polars is emphatic that boolean row selection is not a getitem operation: it ships a dedicated helper, _raise_on_boolean_mask, whose message is "selecting rows by passing a boolean mask to __getitem__ is not supported", followed by "Hint: Use the filter method instead." That helper fires when a mask reaches a Series, when it arrives as a row key, when a boolean Series is converted to indices. It does not fire for a bare df[mask] on a DataFrame, because there the key has a legal reading as columns. The one call shape Feast used is the one shape in which polars withholds the warning it wrote for exactly this mistake.

What that produced depends on the shape of the data, which is the detail that makes this more than a typo. Run the pre-fix expression on four rows and four columns — the shape of the regression test added alongside the fix — and it returns all four rows, including the one a minute newer than the entity timestamp, having quietly dropped the column at the mask's first false position. Run it on five rows and four columns and it raises: ValueError: expected 4 values when selecting columns by boolean mask, got 5. The behaviour is identical on polars 1.33.1, from September 2025, and on polars 2.0.0, released on 6 October — thirteen months and a major version apart. With a TTL configured the node stops either way; the project's note records a ColumnNotFoundError, and on both tested versions the second comparison raises a TypeError first.

So whether a training-set join silently kept the future or stopped with an error turned on an arithmetic coincidence between the number of rows a join produced and the number of columns it carried. And when there was an error, it spoke about columns while the code was filtering time. How many pipelines ran this configuration is not something the public record shows.

Which configuration, then? LocalComputeEngineConfig carries one optional field, backend, documented in its own docstring as "Backend to use for DataFrame operations (e.g., 'pandas', 'polars')". The string backend: appears nowhere in the compute engine reference or the feature_store.yaml reference; the local engine's documentation page, which the engine reference links as local.md, is not in the repository at all, while Spark, Ray, Trino, Flink, Snowflake and Lambda each have one. The same reference lists the local engine as running on "Arrow + Specified backend (e.g., Pandas, Polars)" and, in its roadmap further down, lists "Scalable distributed backend via Ray or Polars" as unchecked. The word polars does not appear in Feast's pyproject.toml — not as a dependency, not as an extra — so the library whose indexing semantics decided the answer is one the user installs at a version of their own choosing. When the field is unset, BackendFactory.infer_from_entity_df selects by the runtime type of the entity frame. The Polars backend has been in the tree since 23 October 2025.

None of that is the real failure, though. The real failure is that nothing, anywhere, asserted the property. The node's only unit test before 13 October exercised a user filter expression, value > 15, on the pandas backend; the point-in-time branch had no test under either backend. Meanwhile every node in every engine opens by checking the kind of value it was handed — assert_format(DAGFormat.ARROW) in the local nodes, DAGFormat.SPARK six times in the Spark nodes, DAGFormat.RAY five times in the Ray ones. The DAG is scrupulous about whether an intermediate value is the right sort of object and says nothing about whether any surviving row is older than the timestamp it is attached to. It validates types on every hop and time nowhere.

The fix's shape is the admission. It adds filter_by_mask to the DataFrameBackend abstract base class and implements it twice — df[mask] for pandas, df.filter(mask) for polars. That interface already had join, groupby_agg, drop_duplicates, rename_columns, to_timedelta_value and a string-expression filter. The single operation carrying the product's headline guarantee was the one operation not in the contract, which is why a caller could express it in one engine's dialect and have another engine answer in its own.

The strongest defence is that this is the local engine, which the documentation describes as "designed for local dev, testing, or lightweight feature generation". Real training runs go to Spark or a warehouse, and the Spark node writes the same condition natively and correctly, filter(F.col(timestamp_column) <= F.col(ENTITY_TS_ALIAS)). A defect in an opt-in backend of a development engine, found by a contributor and closed within a release with a test parameterised over both backends, is open source working. And runtime assertions are not free: a feature store that re-checked its own joins would be a slower feature store on every run of every pipeline, forever, and most of those runs are fine.

That defence is fair about the blast radius and wrong about the lifecycle. get_historical_features on the local engine is what somebody runs to find out whether a feature is worth having at all. Development is not where correctness matters least; it is where the evidence is manufactured. A leaky join in a notebook does not surface as an incident — it surfaces as an offline metric good enough to justify a quarter of work, after which the production pipeline that computes the feature correctly is the component that looks broken. Leakage is the only class of data defect whose symptom is encouragement.

It is also not an isolated slip. The same release carries two more fixes of the same shape. ApplyMaterialization and ApplyValidationReference built their domain object inside the argument list of their own permission check, and because Python evaluates arguments first, from_proto() — which dill-loads a caller-supplied user-defined function — always ran before assert_permissions could reject the request: an unauthorised caller achieved code execution and then received a permission error. Separately, a lifecycle gate introduced in 0.64.0 to stop serving feature views with no online data meant that routine incremental materialization, which flips a view to MATERIALIZING in the shared registry, interrupted every concurrent feature server; 0.67.0 adds an opt-in flag, default off. Three guarantees, three places they were not: a dataframe idiom, an argument list, a shared status field.

The portability claim deserves one more look in that light. "Run the same features anywhere" is sold as operational convenience, and what it actually asserts is that several independently hand-written implementations of one temporal semantic agree. Feast has at least three for this filter alone, plus whatever a warehouse pushes down. Nothing compares them. The abstraction was drawn around the engine, and the meaning was left to travel on trust.

A training set does not record the dataframe library that produced it. It carries its features, its entity keys, its timestamps, perhaps a dataset version — and nowhere a note of which implementation evaluated the comparison that decided which rows it was allowed to contain. The window closed on 13 October. The datasets made inside it will not mention that they were in it, and the only trace they left is a model that performed slightly better than it should have, which is not a signal anyone has been trained to investigate.