Skip to content

Add FFI query planner support - #1677

Merged
timsaucer merged 44 commits into
mainfrom
feat/ffi-query-planner-core
Aug 28, 2026
Merged

timsaucer merged 44 commits into
mainfrom
feat/ffi-query-planner-core

Conversation

@timsaucer

@timsaucer timsaucer commented Aug 7, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Related to #1612. This PR does not close it, but provides the FFI query planner plumbing that a datafusion-distributed integration can build on.

This is part 1 of 3 in the split of #1672. These are enabled as a github stack so you should be able to swab between the 3 PRs in github interface (above, next to the "Open" oval).

Rationale for this change

Extension libraries (for example distributed execution engines) need to supply their own QueryPlanner to a SessionContext without compiling against the datafusion-python crate. This PR exposes the query planner over the FFI boundary, following the same PyCapsule pattern used for table providers and catalogs.

What changes are included in this PR?

  • SessionContext.set_query_planner(planner) installs a planner exported via a __datafusion_query_planner__ PyCapsule. It mutates the session and returns nothing, the same way add_physical_optimizer_rule does — the query planner lives in SessionState, so it belongs to the session rather than to a particular handle on it.
  • SessionContext.__datafusion_query_planner__() exports the current planner so another planner can wrap it as an explicit fallback (a session holds exactly one planner; layering is explicit delegation).
  • The three capsule getters that need something only a session has — __datafusion_query_planner__, __datafusion_logical_extension_codec__, and __datafusion_physical_extension_codec__ — now receive the SessionContext they are being installed on, matching what __datafusion_table_provider__ and friends have done since 52.0.0. A codec takes the TaskContextProvider from it; a planner takes both codecs. Neither example constructs a SessionContext any more, and decode callbacks now resolve names against the session running the query.
  • A PySessionContext keeps the Arc<SessionContext> it was created with for its whole life. This is load-bearing rather than incidental: FFI_TaskContextProvider holds its provider weakly, and a registered catalog provider upgrades that handle on every supports_filters_pushdown and every scan, so replacing the allocation orphans every handle bound to it. Deriving a new context to install a planner made ctx = ctx.with_query_planner(planner) break the next query against a previously registered foreign catalog with TaskContextProvider went out of scope over FFI boundary. Installing in place avoids the problem instead of repairing it — a codec already embedded in a registered FFI_CatalogProvider is not reachable to rebind, and a codec cannot retain its own session without closing a cycle through the catalog it gets registered into.
  • Every __datafusion_*__ capsule getter now goes through call_capsule_getter, including the ones taking no argument and the four taking a codec capsule rather than a session. An out-of-date catalog provider, schema provider, table factory, or catalog provider list is now diagnosed by name instead of raising a bare TypeError, and the message names the argument that was actually refused.
  • New example crate datafusion-ffi-query-planner-example demonstrating a real three-library plan exchange (host, provider library, planner library as separate cdylibs), including session config transfer via SessionConfig.with_extension.
  • Both example codecs accept require_udf_on_decode, and tests assert which session a decode callback resolves against — including a function registered on the host and one registered after the codec was installed.
  • Installing an extension codec no longer resets python_udf_inlining. Python{Logical,Physical}Codec::new defaults it to on, so a context that had opted out silently started embedding cloudpickled callables again after a codec install.
  • New docs/source/contributor-guide/ffi.md sections covering the capsule protocol, why a session keeps one Arc<SessionContext>, and what a derived context shares. New .ai/skills/ffi-capsule-protocol/ recording the convention, with a pointer from AGENTS.md. Corrected user-guide/io/table_provider.md, which still showed the pre-52.0.0 signature.

Are there any user-facing changes?

Yes, including breaking changes.

New public API: SessionContext.set_query_planner and SessionContext.__datafusion_query_planner__.

Breaking: __datafusion_logical_extension_codec__ and __datafusion_physical_extension_codec__ now take a session: Bound<PyAny> parameter, so any extension library implementing them must be updated. docs/source/user-guide/upgrade-guides.md has a 55.0.0 section with before and after. Calling the old signature raises an import error naming the method rather than a bare TypeError. SessionContext's own getters accept the argument optionally, so ctx.__datafusion_logical_extension_codec__() is unaffected.

Breaking: physical_codec_from_pycapsule is removed from the datafusion-python-util crate, and ffi_logical_codec_from_pycapsule takes a second argument. Both are covered in the upgrade guide.

A new example crate ships under examples/.

Notes for reviewers

Cargo.toml pins DataFusion to a pre-release git rev, and the pin is required. PySessionContext::set_session_query_planner rebuilds an installed foreign query planner against the session's current codecs, which depends on FFI_QueryPlanner::new_with_ffi_codecs unwrapping a ForeignQueryPlanner and replacing its codecs (apache/datafusion#24722). That replacement is a silent no-op in released 55.0.0, so building against it would leave a codec installed after set_query_planner with no effect — the case covered by test_provider_codecs_can_be_installed_after_planner. The pin can be dropped once 55.1.0 is released.

SessionContext.enable_url_table is the one remaining method that mints a second Arc<SessionContext> for a session, so its result must not outlive the receiver. That predates this PR and is documented rather than changed.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants