CSS API: Add a CSS Syntax Level 3 tokenizer by sirreal · Pull Request #14781 · WordPress/wordpress-develop

sirreal · 2026-10-23T13:02:23Z

Caution

This is a work in progress draft and is not ready for any serious review.

Add WP_CSS_Token_Processor, a forward-only CSS Syntax Level 3 tokenizer for replacing Core's ad hoc CSS parsers. #14782 adds a style attribute processor that uses it.

create() accepts UTF-8 CSS; next_token() advances through it. The cursor exposes each token's type, source span and decoded value. get_token() returns a WP_CSS_Token that remains valid after the cursor advances.

The tokenizer preserves source offsets for editing:

Comments are returned as tokens.
CSS input preprocessing runs while reading tokens, leaving the source bytes unchanged.
Invalid UTF-8 decodes to U+FFFD using the maximal subpart rule.
next_token() returns false at EOF; there is no EOF token.

The internal WP_CSS_Token_Serializer re-escapes decoded values and separates tokens that would otherwise combine, following CSS Syntax serialization. It also prevents accidental CDO/CDC formation, which the specification's pairwise table does not cover. Public declaration serialization is added in #14782.

Reported validation: 957 passing css-api tests, including the 185-case @rmenke/css-tokenizer-tests 1.3.0 corpus. The fixture README documents the import and three deliberate differences from that corpus; its MIT license is included.

Trac ticket: https://core.trac.wordpress.org/ticket/65738

Use of AI Tools

AI assistance: Yes
Tools: Codex, Claude Code
Models: GPT-5, Claude Opus, Claude Fable 5.1
Used for: implementation, tests, specification drafting, review and this description.

This pull request is for code review only. Keep other discussion in the Trac ticket. Do not merge. See GitHub Pull Requests for Code Review in the Core Handbook.

The corpus docblock now names @rmenke/css-tokenizer-tests, the package the test cases derive from.

CSS preprocessing replaces NUL with U+FFFD before tokenization, so an unquoted url() containing NUL is a url-token, not a bad-url-token. The source byte stays in the token for offsets; the value decodes it. Ported from WordPress/php-toolkit bb1c7069 with its CSSPreprocessingTest cases.

Ident sequences and unquoted url values advance over runs of bytes that need no special handling with one strspn() call instead of one PHP loop iteration per byte. Tokenizing 2,000 copies of a three-declaration style string (url, dimension, custom property) drops from 54 ms to 14 ms. Ported from WordPress/php-toolkit 8e411e7c.

decode_range() and its escape helpers take the CSS source as an argument, so code that holds the source and a token's offsets can decode the token after the cursor has moved on.

get_token() returns the current token as an object that keeps its values after the processor moves on: type, flag, value, unit, start, length, and whether it ended at its closing delimiter. The value is decoded from the source on the first call to get_value(). The processor stays a cursor; get_token_value() reads the same object, so the value decoding is in one place.

The token processor reads CSS; the new class writes it. serialize_string() and serialize_ident() move unchanged, with their tests.

WP_CSS_Token_Serializer::serialize() writes tokens read by the processor as CSS that reads as the same tokens. Comments are dropped, and each whitespace run, including one broken by comments, becomes one space. An empty comment, /**/, separates each token pair the §9 table marks when no whitespace is between them. Tokens are escaped again from their decoded values: a `\` delim keeps its newline, a hash is escaped by its type flag, and a unit that reads as an exponent has its `e` escaped. Strings and urls cut off by the end of the input are written closed, and blocks still open at the end are closed in reverse order. Bad-string and bad-url tokens have no value; they are written as a quote and a newline, and as `url(()`. serialize_string() still normalizes newlines in its argument. Token values are quoted without that step, so an escaped CR or FF in a string keeps its code point.

CSS Syntax 3 §9 allows a tokenizer to preserve comments when they have no effect on parsing. The processor emits them so callers can find comments by byte offset.

serialize_string() replaced CR, CRLF and FF in its argument with LF, as if the argument were CSS source. Its output is escaped and never preprocessed, so the replacement changed the value: a string token written from "a\D b" read back with a newline. CR and FF are now written as \D and \C, so the string reads back as the same value; NULL still becomes U+FFFD. A caller that wants normalized line endings normalizes before calling. serialize() writes string tokens with serialize_string() again, and the private quote_string() is removed.

The CSS Syntax 3 §9 table compares token types in pairs. It misses two sequences that the tokenizer reads as a CDO or CDC token: `<`, `!` and a token that starts with `--` spell ``. So `<!/**/--x` was written as ``. An empty comment is now written before the last token of each, so the output reads as the same tokens.

@access

A token list can hold tokens that have no valid CSS form, so the public way to write CSS will be at the declaration level, where the style attribute processor exposes only valid declarations. WP_CSS_Token_Serializer is marked @access private; its methods stay public so other core classes can call them. serialize_ident() and serialize_string() had no caller outside the serializer. They become escape_ident(), public for property names, and the private escape_string(); url values use the private escape_url(), which now returns the whole url token. CSSOM defines "serialize an identifier" and "serialize a string" with a different escape set, so the new names avoid suggesting those algorithms. The string escaping tests now run through serialize(). The class documentation states what the output guarantees: it parses to the same CSS as the tokens. It is not always valid CSS: bad-string and bad-url tokens are written as the same bad tokens.

escape_ident() repeated escape_name()'s loop with three changes for the first code point. It now handles a lone `-` and a leading digit, or `-` and a digit, and escapes the rest with escape_name(). Output is unchanged: 80,565 generated values of up to five code points escape the same as before.

Hash flag round-trips get a data provider, with the expected flag of each input checked on both sides. serialize() is tested with a generator, since it accepts any iterable, and with a DEL delim.

String tokens wrote U+0001 to U+001F and U+007F raw, except LF, CR and FF. They now escape every code point in that range, as CSSOM's "serialize a string" does, so a raw tab is written `\9 `. Identifiers, hash names and urls already escaped these code points.

CSSOM's "escape a character as code point" writes the hex digits in lowercase, so `<` is written `\3c ` instead of `\3C `. CSS reads either case, so the output parses as before.

CSS Syntax limits the non-ASCII code points an identifier may contain to those HTML allows in custom element names. The tokenizer, like Chrome, still reads every code point above U+007F as an identifier code point, and escape_name() wrote them all raw: `a\a0 b` was written with a raw no-break space, which a tokenizer that follows the spec reads as an ident, a delim and an ident. escape_name() now hex-escapes the code points outside the set, so identifiers, hash names and dimension units read the same under both rules.

github-actions · 2026-10-23T13:03:37Z

⚠️ PHPStan Static Analysis

PHPStan reported 2 errors when analyzing f3e7bff (workflow run). This comment is updated as new commits are pushed.

Errors

These are also shown as annotations on the changed files. They do not fail the check, but they should still be fixed before the pull request is committed. An error that is a false positive may instead warrant being added to a baseline.

Location	Error	Identifier	First reported
`src/wp-includes/css-api/class-wp-css-token-processor.php:431`	`Unsafe usage of new static().`	`new.static`	`5d86e10`
`src/wp-includes/css-api/class-wp-css-token-processor.php:1416`	`Strict comparison using !== between null and int will always evaluate to true.`	`notIdentical.alwaysTrue`	`5d86e10`

Running PHPStan locally

To run the analysis on your own machine, use npm run typecheck:php, or composer phpstan outside the Docker environment.

github-actions · 2026-10-23T13:06:39Z

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

All changes will be lost when closing a tab with a Playground instance.
All changes will be lost when refreshing the page.
A fresh instance is created each time the link below is clicked.
Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

…processor-foundations

dmsnell · 2026-10-23T21:36:24Z

Let’s make sure that we credit @adamziel for pioneering this work in the PHP Toolkit

dmsnell · 2026-10-23T21:45:38Z

+			return null;
+		}
+		return substr( $this->css, $this->token_starts_at, $this->token_length );
+	}


I haven’t looked to where this is used yet, but there might be some risk in making it such a prominent method. in the Block Processor we expose get_span() which seems risky enough. do we have strong justification that this function is necessary?

also, I wonder if “raw” would be a valuable name to include here vs. unnormalized and what the tradeoffs are between these on developer comprehension

dmsnell · 2026-10-23T21:47:09Z

+	 */
+	public function get_token_value(): ?string {
+		$token = $this->get_token();
+		return null === $token ? null : $token->get_value();


for functions returning properties of the WP_CSS_Token class, we have hidden allocations that may not be obvious. did you consider leaving this only as a method on WP_CSS_Token so that the chain of computation and allocation would be clear?

dmsnell · 2026-10-23T21:49:06Z

+	 */
+	public function is_token_terminated(): bool {
+		return null !== $this->token_type && $this->token_is_terminated;
+	}


this is a notable change in how this processor works compared to the HTML API and Block Processor, which expose paused_at_incomplete_token(). what does is mean if we provide consumers a convenient way to access syntax that we know is definitively broken?

dmsnell · 2026-10-23T21:50:42Z

+			// than one PHP iteration per byte. These ASCII bytes need no special
+			// handling in an unquoted URL. Quotes, parentheses, whitespace, escapes,
+			// and non-ASCII bytes fall through to the rules below.
+			// Only the cursor advances; source bytes stay unchanged.


who was assuming that the source bytes are modified?

dmsnell · 2026-10-23T21:52:01Z

+			if ( $plain > 0 ) {
+				$this->at += $plain;
+				continue;
+			}


given that this is at the start of the loop, there’s no need to continue no? because that will immediately run the strspn() again? apart from duplicating an $at < $this->length check, would it make sense to leave this is an early ASCII slide into the rest of the loop body?

…processor-foundations

The CSS API is not in 7.2.0. The `_doing_it_wrong()` call in consume_string() also names 7.3.0 instead of 1.0.0.

- `@link` tags hold URLs and `@see` tags name code. Related constants name each other: string and bad-string, url and bad-url, CDC and CDO, the hash token and its type flags, and the serializer's follower lists and `MARKED_PAIRS`. - Docblock examples are `php interactive` blocks that print their result. The class example that gathered bad strings had no string in its input and found none; it now has one. - Box drawing shows the token and value spans of the offset properties. - The UTF-8 note links the HTML chapter 3 of Unicode 18.0.0, section 3.9.6, U+FFFD Substitution of Maximal Subparts, instead of the Unicode 9.0.0 PDF. - Four CSS Syntax anchors did not exist: `typedef-CDC-token` and `typedef-CDO-token` are lowercase, `delim-token-diagram` is `typedef-delim-token`, and token values are defined under `tokenization`. - Comment closers are written as `*` `/`, CSS and HTML literals are in backticks, and comments use the impersonal voice.

The tokenizer stored value offsets for comment tokens that nothing read and that were wrong: the value started at the `/` of the opener, and its length was the token length minus 4 even when the input ended before the closer, so `/* a` had an empty value. The value now starts after `/*` and ends before `*/` or at the end of the input. WP_CSS_Token::get_value() and get_token_value() return that text for a comment token instead of null. Comments contain no escapes, so `/*\41*/` has the value `\41`; only input preprocessing applies, as for other values: CR, CRLF and FF become LF, and NUL and invalid UTF-8 become U+FFFD. The six comment tokens in the corpus and the normalization test now expect the text, and the corpus README describes the rule.

The docblock said a `\` followed by the end of the input is not a valid escape, but the function returns true there, as CSS Syntax requires: only a newline rejects it. The spec step that rejects a newline was written above the end-of-input branch, which returns true, and the newline check was labeled with the step that returns true. Each comment now precedes the code it describes.

get_normalized_token() decoded escapes in every token, so `/* \41 */` was normalized to `/* A*/`. CSS comments contain no escapes. The new WP_CSS_Token_Processor::preprocess_range() applies input preprocessing only: CR, CRLF and FF become LF, and NUL and invalid UTF-8 become U+FFFD. get_normalized_token() uses it for comment tokens, and WP_CSS_Token::get_value() uses it for comment values instead of its own copy of the same rules. Corpus case `tests/comment/0004` expected the decoding: `/*\*/` normalized to `/**/`. It now expects `/*\*/`, and the corpus README says escapes are decoded outside comments only.

The cursor keeps next_token(), get_token(), get_token_type(), get_token_start() and get_token_length(): it says what the current token is and where it is. WP_CSS_Token says what the token contains. Removed: get_token_value(), get_token_type_flag(), get_token_unit() and is_token_terminated(), each a copy of a WP_CSS_Token getter; get_unnormalized_token(), a substr() of the source by the offsets the cursor already exposes; and get_normalized_token(), the decoded text of the whole token span, which no code in src reads. The `normalized` column of the css-test-cases.json fixture existed only to check get_normalized_token(), so it is removed with it. The collect_tokens() test helper now takes the CSS source, since it needs the source to slice each token's raw text.

@access

Code that puts a PHP value into CSS has no public way to escape it: the serializer's escapers are internal, so each caller writes its own. WP_CSS_Builder writes a value as a quoted string, a url() function holding a quoted string, or an identifier, each reading back as the given value. The escape rules stay in WP_CSS_Token_Serializer, which may later escape for other contexts or with other character sets; the builder forwards to it. escape_string() becomes public and marked @access private, as escape_ident() already is. url() writes the quoted form, url("..."), so it needs only the string escapes. The unquoted url token needs its own escapes for parentheses, quotes and whitespace.

@ignore

The developer documentation site skips declarations marked @ignore. Every PHP-private member and every declaration marked @access private now has it, as the HTML API does. Private members without a @SInCE tag gain @SInCE 7.3.0 with it.

@see

The per-type description of a token's value and the termination rules were on WP_CSS_Token_Processor::get_token_value() and is_token_terminated(), which no longer exist. They move to WP_CSS_Token::get_value() and is_terminated(), which had one-line descriptions. The class example that echoed get_normalized_token() now echoes token types, and the @see tags naming removed methods are gone.

The parameter accepted only 'UTF-8' and returned null for any other value, so create() was typed static|null and every caller checked for a null it could not get. Every caller passes text that is already decoded: an attribute value or a string from the database. An optional trailing parameter can be added in a later release without breaking callers if the tokenizer ever reads other encodings; removing one after release cannot.

sirreal added 24 commits October 8, 2026 21:00

CSS API: Add CSS token processor foundations

a491bef

Harden CSS builder identifier serialization

a80bd98

Harden CSS token value updates

db896db

Point CSS API tests at Trac #65738 and set @SInCE 7.2.0

b2e6400

The corpus docblock now names @rmenke/css-tokenizer-tests, the package the test cases derive from.

Move string and ident serialization onto the token processor

6804231

Drop the token mutation API

c64ca47

Drop untested token value getters

59eece4

Document corpus provenance

52576f1

Report whether a token ended at its delimiter

a9d4b5d

Decode token text with a static method

dabd8c5

decode_range() and its escape helpers take the CSS source as an argument, so code that holds the source and a token's offsets can decode the token after the cursor has moved on.

Move string and ident serialization into WP_CSS_Token_Serializer

892cd25

The token processor reads CSS; the new class writes it. serialize_string() and serialize_ident() move unchanged, with their tests.

Document why comment tokens are kept

3458860

CSS Syntax 3 §9 allows a tokenizer to preserve comments when they have no effect on parsing. The processor emits them so callers can find comments by byte offset.

Fill serializer test gaps

06c6e5d

Hash flag round-trips get a data provider, with the expected flag of each input checked on both sides. serialize() is tested with a generator, since it accepts any iterable, and with a DEL delim.

Write hex escapes in lowercase

52216bc

CSSOM's "escape a character as code point" writes the hex digits in lowercase, so `<` is written `\3c ` instead of `\3C `. CSS reads either case, so the output parses as before.

This was referenced Oct 23, 2026

HTML API: Add WP_HTML_Style_Attribute_Processor #14782

Draft

KSES: Parse declarations in safecss_filter_attr() #14784

Draft

Block Supports: Append background and dimensions styles with the style attribute processor #14785

Draft

Merge remote-tracking branch 'upstream/trunk' into css-api/css-token-…

7900018

…processor-foundations

dmsnell reviewed Oct 23, 2026

View reviewed changes

sirreal added 11 commits October 27, 2026 15:44

Merge remote-tracking branch 'upstream/trunk' into css-api/css-token-…

11e4165

…processor-foundations

Set CSS API @SInCE to 7.3.0

c29e0c9

The CSS API is not in 7.2.0. The `_doing_it_wrong()` call in consume_string() also names 7.3.0 instead of 1.0.0.

Mark private CSS API declarations @ignore

f0020b5

The developer documentation site skips declarations marked @ignore. Every PHP-private member and every declaration marked @access private now has it, as the HTML API does. Private members without a @SInCE tag gain @SInCE 7.3.0 with it.

Navigation Menu

Repository navigation

CSS API: Add a CSS Syntax Level 3 tokenizer - #14781

CSS API: Add a CSS Syntax Level 3 tokenizer#14781
sirreal wants to merge 36 commits into
WordPress:trunkfrom
sirreal:css-api/css-token-processor-foundations

sirreal commented Oct 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 23, 2026

Uh oh!

dmsnell commented Oct 23, 2026

Uh oh!

dmsnell Oct 23, 2026

Uh oh!

dmsnell Oct 23, 2026

Uh oh!

dmsnell Oct 23, 2026

Uh oh!

dmsnell Oct 23, 2026

Uh oh!

dmsnell Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sirreal commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Use of AI Tools

Uh oh!

github-actions Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ PHPStan Static Analysis

Errors

Running PHPStan locally

Uh oh!

github-actions Bot commented Oct 23, 2026

Test using WordPress Playground

Some things to be aware of

Uh oh!

dmsnell commented Oct 23, 2026

Uh oh!

dmsnell Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

dmsnell Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

dmsnell Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

dmsnell Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

dmsnell Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sirreal commented Oct 23, 2026 •

edited

Loading

github-actions Bot commented Oct 23, 2026 •

edited

Loading