Repository navigation
Add FFI query planner support - #1677
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to #1612. This PR does not close it, but provides the FFI query planner plumbing that a
datafusion-distributedintegration can build on.This is part 1 of 3 in the split of #1672. These are enabled as a github stack so you should be able to swab between the 3 PRs in github interface (above, next to the "Open" oval).
Rationale for this change
Extension libraries (for example distributed execution engines) need to supply their own
QueryPlannerto aSessionContextwithout compiling against thedatafusion-pythoncrate. This PR exposes the query planner over the FFI boundary, following the same PyCapsule pattern used for table providers and catalogs.What changes are included in this PR?
SessionContext.set_query_planner(planner)installs a planner exported via a__datafusion_query_planner__PyCapsule. It mutates the session and returns nothing, the same wayadd_physical_optimizer_ruledoes — the query planner lives inSessionState, so it belongs to the session rather than to a particular handle on it.SessionContext.__datafusion_query_planner__()exports the current planner so another planner can wrap it as an explicit fallback (a session holds exactly one planner; layering is explicit delegation).__datafusion_query_planner__,__datafusion_logical_extension_codec__, and__datafusion_physical_extension_codec__— now receive theSessionContextthey are being installed on, matching what__datafusion_table_provider__and friends have done since 52.0.0. A codec takes theTaskContextProviderfrom it; a planner takes both codecs. Neither example constructs aSessionContextany more, and decode callbacks now resolve names against the session running the query.PySessionContextkeeps theArc<SessionContext>it was created with for its whole life. This is load-bearing rather than incidental:FFI_TaskContextProviderholds its provider weakly, and a registered catalog provider upgrades that handle on everysupports_filters_pushdownand everyscan, so replacing the allocation orphans every handle bound to it. Deriving a new context to install a planner madectx = ctx.with_query_planner(planner)break the next query against a previously registered foreign catalog withTaskContextProvider went out of scope over FFI boundary. Installing in place avoids the problem instead of repairing it — a codec already embedded in a registeredFFI_CatalogProvideris not reachable to rebind, and a codec cannot retain its own session without closing a cycle through the catalog it gets registered into.__datafusion_*__capsule getter now goes throughcall_capsule_getter, including the ones taking no argument and the four taking a codec capsule rather than a session. An out-of-date catalog provider, schema provider, table factory, or catalog provider list is now diagnosed by name instead of raising a bareTypeError, and the message names the argument that was actually refused.datafusion-ffi-query-planner-exampledemonstrating a real three-library plan exchange (host, provider library, planner library as separate cdylibs), including session config transfer viaSessionConfig.with_extension.require_udf_on_decode, and tests assert which session a decode callback resolves against — including a function registered on the host and one registered after the codec was installed.python_udf_inlining.Python{Logical,Physical}Codec::newdefaults it to on, so a context that had opted out silently started embedding cloudpickled callables again after a codec install.docs/source/contributor-guide/ffi.mdsections covering the capsule protocol, why a session keeps oneArc<SessionContext>, and what a derived context shares. New.ai/skills/ffi-capsule-protocol/recording the convention, with a pointer fromAGENTS.md. Correcteduser-guide/io/table_provider.md, which still showed the pre-52.0.0 signature.Are there any user-facing changes?
Yes, including breaking changes.
New public API:
SessionContext.set_query_plannerandSessionContext.__datafusion_query_planner__.Breaking:
__datafusion_logical_extension_codec__and__datafusion_physical_extension_codec__now take asession: Bound<PyAny>parameter, so any extension library implementing them must be updated.docs/source/user-guide/upgrade-guides.mdhas a 55.0.0 section with before and after. Calling the old signature raises an import error naming the method rather than a bareTypeError.SessionContext's own getters accept the argument optionally, soctx.__datafusion_logical_extension_codec__()is unaffected.Breaking:
physical_codec_from_pycapsuleis removed from thedatafusion-python-utilcrate, andffi_logical_codec_from_pycapsuletakes a second argument. Both are covered in the upgrade guide.A new example crate ships under
examples/.Notes for reviewers
Cargo.tomlpins DataFusion to a pre-release git rev, and the pin is required.PySessionContext::set_session_query_plannerrebuilds an installed foreign query planner against the session's current codecs, which depends onFFI_QueryPlanner::new_with_ffi_codecsunwrapping aForeignQueryPlannerand replacing its codecs (apache/datafusion#24722). That replacement is a silent no-op in released 55.0.0, so building against it would leave a codec installed afterset_query_plannerwith no effect — the case covered bytest_provider_codecs_can_be_installed_after_planner. The pin can be dropped once 55.1.0 is released.SessionContext.enable_url_tableis the one remaining method that mints a secondArc<SessionContext>for a session, so its result must not outlive the receiver. That predates this PR and is documented rather than changed.