Skip to content

Store MySQL JSON documents as exact text (#1016) - #1161

Merged
rmcdaniel merged 2 commits into
mainfrom
feat/server-1016-mysql-json-text
Nov 12, 2026
Merged

rmcdaniel merged 2 commits into
mainfrom
feat/server-1016-mysql-json-text

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Fixes #1016. Native 3.7.0 headline. Do not merge before 3.6.0 is tagged (General, 2026-11-12). Auto-merge stays off until then.

MySQL's json type parses doubles into its own binary form, so history, heartbeat details and other JSON documents could come back one ULP off (1.4820142857142857 as …855). MariaDB, PostgreSQL and SQLite keep the text.

Change (MySQL only, schema 12 → 13)

  • rust/migrations/mysql/0013_json_text.sql: the 75 json columns in 36 tables become LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin with an enforced json_valid check. Check names follow the Compile MySQL JSON text upgrade table plan and required controls #1035 plan: dw_json_text_ plus the first 20 hex digits of SHA-256(table NUL column). There is one atomic ALTER TABLE per table.
  • Fresh bootstrap creates the final columns directly. schema-upgrade from 12 runs the ALTERs, and a resumed upgrade or PHP takeover accepts each table only in its exact 12 or 13 definition. The older steps (8–11) now build toward the schema-12 catalog. On real MySQL the fresh and upgraded catalogs are byte-identical.
  • Per-flavor schema version: MySQL 13, MariaDB stays at 12 with an unchanged bootstrap checksum (pinned in mariadb/0012_checksum.txt). The transition lock selects the version with IF(VERSION() LIKE '%MariaDB%',12,13), the upgrade report carries the target, and takeover marks the flavor's version. MySQL's retained schema-12 checksum is pinned in mysql/0012_checksum.txt and equals main's current one.
  • No query changes: MySQL already reads these columns with the JSON_EXTRACT/JSON_UNQUOTE SQL MariaDB runs on text, and sqlx binds Json as a string, so text is stored verbatim.
  • Parity: json-doubles keeps mysql_json_doubles for published PHP only; native expects exact history on MySQL.

General's gates

  1. Migration timing on MySQL 8.0 with a seeded history (qualification_json_text_upgrade_timing, DW_JSON_TEXT_TIMING_ROWS): MySQL 8.0.46, 2 CPUs, default 128 MiB buffer pool, 1,000,000 ~1 KiB history rows (2.65 GB data): upgrade 115 s, about 2 min per 1M history rows.
  2. Bit-exact doubles: new doubles_come_back_bit_exact_through_history_memo_search_attributes_and_heartbeats checks to_bits() equality for the memo (describe and WorkflowStarted), double search attributes (describe, list and WorkflowStarted) and heartbeat details, read through a fresh runtime. On MySQL it also asserts that workflow_history_events.payload is longtext. It passes on SQLite and real MySQL 8.0. The existing memo test now expects exact history on MySQL as well.
  3. MariaDB and PostgreSQL untouched: no MariaDB, PostgreSQL or SQLite migration or schema code changes. A unit test pins MariaDB's unchanged checksum and version 12. The qualification subset (bootstrap, upgrades 7–12, takeovers, transitions) passes on CI's MariaDB 10.11 image.
  4. Upgrade text (rust/README.md): MySQL only; stop every node, back up, run schema-upgrade, then start; downtime for the whole upgrade at about 2 min per million history rows; existing rows keep MySQL's rendering; rollback means restoring the backup. The 3.7.0 release notes and chart UPGRADING will repeat it.

Tests

  • MySQL 8.0.46: the full MySQL qualification suite passes (40, kill/resume included), plus the new qualification_version_12_json_text_upgrade_resumes_and_keeps_rows. It covers an upgrade before any change, resume after two committed tables, refusal of a foreign table definition, a 12-era row keeping MySQL's rendering, 75 enforced checks, new text kept exactly, and invalid JSON refused.
  • MySQL 8.4.13: bootstrap, upgrade, takeover and version 10–12 tests pass.
  • Execution tests on MySQL 8.0 (doubles, memo, search attributes, heartbeats): 31 pass. SQLite lib 483 and execution 777 pass. Node unit tests pass, except the 4 that need a local php.

🤖 Generated with Claude Code

MySQL's json type parses doubles into its own binary form, so history,
heartbeat details and other JSON documents could come back one ULP off
(1.4820142857142857 as ...855). MariaDB, PostgreSQL and SQLite keep the
text.

MySQL moves to native schema 13: the 75 json columns in 36 tables become
LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin with an enforced
json_valid check named dw_json_text_<sha256(table NUL column)[..20]>.
A fresh bootstrap creates them directly; schema-upgrade from 12 runs one
atomic ALTER per table, and a resumed upgrade or takeover accepts each
table only in its exact 12 or 13 definition. MariaDB stays at 12 with an
unchanged bootstrap checksum; the transition lock picks the version per
server. Stored documents keep MySQL's rendering.

Real MySQL 8.0.46 and 8.4.13 and MariaDB 10.11 runs pass the
qualification suites; 1M history rows (2.65 GB) upgrade in 115 s on
MySQL 8.0 with 2 CPUs. Doubles come back bit-exact on MySQL through
history, memo, search attributes and heartbeat details.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MySQL parity harness compares published PHP's physical catalog with
native's. projectMysqlJsonText maps exactly native's utf8mb4_bin text
column and named json_valid check back to PHP's json definition for
each PHP json column; any other difference still fails.

A registered JSON default now keeps its key order on MySQL, as on every
other backend: MySQL's json column used to sort object keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Update: 3.6.0 is released and promoted, so auto-merge is on. CI fixes in this push:

  • The MySQL parity harness compares published PHP's physical catalog with native's. New projectMysqlJsonText (ready-origin-schema.php, used by mysql-schema.php compare) maps exactly native's utf8mb4_bin text column and its named json_valid check back to PHP's json definition, for each PHP json column. Any other difference still fails. It is unit tested in NativeMysqlJsonTextSchemaTest (exact round trip from catalog 13 to 12, live CHARACTER SET spelling, MariaDB unchanged, refusals for a changed collation or a missing or renamed check), and I checked it with php:8.3-cli.
  • Behavior note for 3.7.0: a registered JSON default (and any JSON object) now keeps its key order on MySQL, as on every other backend. MySQL's json column used to sort object keys by length, then bytes. query_contract_arguments now expects the registered order on all backends.

@rmcdaniel
rmcdaniel enabled auto-merge November 12, 2026 04:19
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 9e473a6 Nov 12, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MySQL json columns round JSON doubles in history; store JSON losslessly (3.2.0)

1 participant