Repository navigation
fix(kernel): preserve session routing for async readers - #1001
Conversation
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
GitHub Actions
Inactive
GitHub Actions
Inactive
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
GitHub Actions
Failure
GitHub Actions
Inactive
![peco-review-bot[bot]](https://avatars.githubusercontent.com/in/3801177?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: 1 Medium
Looks solid overall — the reader-derivation routing, lock-guarded owner/marker/in-flight bookkeeping, and error passthrough are correct and well-covered by the updated unit tests and the new realkernel tests. One medium concern: the new hard runtime dependency on ExecutedAsyncStatement.reader() isn't reflected in the ^1.3.0 floor in pyproject.toml, so an in-range older wheel will regress previously-working async re-fetch/concurrent-poll (including for standard sessions) with a NotSupportedError.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
Verdict: 1 Medium
Looks good — the session-routing change is sound (owner read under lock, reader derived outside it, all owner-present/absent × streamed/in-flight branches resolve correctly, and NotSupportedError propagates unchanged through wrap_kernel_exception so old wheels get a clear error). Unit + realkernel tests cover the derived-reader paths. One medium concern: the committed KERNEL_REV (cb5b11c1…) doesn't match the SHA named in the PR description (b2cda261…) — verify the committed revision actually exposes reader().
What type of PR is this?
Description
The kernel backend reattaches by statement ID for result re-fetches and concurrent polls. That loses the submitting session's routing when using warehouse session URLs for Lakehouse Real-Time.
Derive independent readers from the retained async owner for these reads. Serial status polling and the first result stream continue using the owner; IDs without a retained owner continue using ID-only attachment. Preserve owner reservations, retry behavior and statement lifetime.
Document same-connection async polling and re-fetch support, along with the Lakehouse Real-Time cross-connection restriction. Add native routing coverage for both standard and Real-Time sessions and run it in the source-built kernel workflow.
This depends on the reader API in databricks/databricks-sql-kernel#380.
KERNEL_REVpinscb5b11c1e6991e0b1a61f6ae67f7c7dcecfd8a23for source-built CI. The package dependency remains^1.3.0; older wheels withoutExecutedAsyncStatement.reader()raise a clearNotSupportedErrorwhen an independent reader is needed. Package distribution requires a reader-capable kernel release.How is this tested?
Unit tests
E2E Tests
Manually
N/A
python -m pytest tests/unit/test_kernel_client.py -m 'not realkernel' -q: 248 passed.python -m pytest tests/unit/test_kernel_async_readers.py -m realkernel -q: 4 passed, using the native wheel built from the pinned kernel revision and a local HTTP server. Covers repeated result fetches, overlapping status/materialization, inline chunk routing and owner cleanup for both URL families.Black checks for the backend and native test file, plus
git diff --check, passed.Live staging and resharding validation remain pending.
Related Tickets & Documents