Skip to content

Fix: render execution_time through a per-test generator instead of patching the shared dialect - #6031

Merged
cmgoffena13 merged 2 commits into
SQLMesh:mainfrom
niklasdohmen:fix/thread-safe-execution-time-rendering
Oct 21, 2026
Merged

cmgoffena13 merged 2 commits into
SQLMesh:mainfrom
niklasdohmen:fix/thread-safe-execution-time-rendering

Conversation

@niklasdohmen

@niklasdohmen niklasdohmen commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem

ModelTest implements execution_time by patching two process-global dictionaries on the dialect's generator class — TRANSFORMS and SQLGlot's cached dispatch table — inside _concurrent_render_context. The lock there covers only the model and CTE renders; setUp (fixture CREATE VIEW) and tearDown render SQL outside it.

With concurrent_tasks > 1 and at least one test that sets execution_time, another thread can hit the window in which patch.dict restores the dispatch table (clear, then refill) and fail with

ValueError: Unsupported expression type Create

while creating its fixture views. Less visibly, any other thread that renders CURRENT_DATE/CURRENT_TIMESTAMP during the patched window gets the first test's frozen time. We saw the crash on a 106-test Databricks project running 16-wide and had to serialise the whole suite (15 minutes instead of about 1).

Fix

  • Build the frozen-time transforms into a per-test generator subclass (_frozen_time_generator_class, cached per generator/time/dialect) instead of patching the shared class. SQLGlot caches one dispatch table per generator class, so the subclass gets its own and the shared one is never touched.
  • SQL model tests render the model query and the CTE queries through that subclass (_generate_sql). _concurrent_render_context keeps the lock only for what is genuinely process-global: time_machine.travel.
  • Python model tests can run arbitrary SQL through the engine adapter, whose generator cannot be swapped per test, so they keep the previous shared-dialect patch under the lock (patch_shared_dialect=True). That path is unchanged in behaviour.

Test

test_freeze_time_does_not_patch_shared_generator: inside a frozen test's render context, another thread renders CREATE VIEW and CURRENT_DATE through the shared duckdb dialect 200 times; asserts no error, that the shared dialect still renders CURRENT_DATE, that the test's own generator renders the frozen cast, and that the shared TRANSFORMS and dispatch table are byte-identical afterwards. On main the shared dialect renders the frozen cast inside the context, which this test rejects; passes with this change. tests/core/test_test.py otherwise unchanged in outcome.

… patch

ModelTest froze time by patching the dialect's generator TRANSFORMS and
SQLGlot's cached dispatch table, which are process-global, under a lock
that only covers the model and CTE renders. setUp (fixture CREATE VIEW)
and tearDown render outside that lock, so with concurrent_tasks > 1
another test could observe the dispatch table mid-restore and fail with
"Unsupported expression type Create", or silently render CURRENT_* at the
other test's frozen time.

Build the frozen-time transforms into a per-test generator subclass and
render SQL model tests through it; keep the lock for time_machine only.
Python model tests still patch the shared dialect under the lock because
their SQL goes through the engine adapter.

Signed-off-by: Niklas Dohmen <niklas@enam.co>
@niklasdohmen
niklasdohmen force-pushed the fix/thread-safe-execution-time-rendering branch from 15ba5cc to 55d2c86 Compare September 7, 2026 14:14
@cmgoffena13
cmgoffena13 self-requested a review October 21, 2026 04:57
@cmgoffena13 cmgoffena13 self-assigned this Oct 21, 2026

@cmgoffena13 cmgoffena13 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this does not address this issue with Python models, that is out of scope (at least to me). Approved

@cmgoffena13
cmgoffena13 merged commit 2026750 into SQLMesh:main Oct 21, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants