Repository navigation
Conversation
The corpus docblock now names @rmenke/css-tokenizer-tests, the package the test cases derive from.
CSS preprocessing replaces NUL with U+FFFD before tokenization, so an unquoted url() containing NUL is a url-token, not a bad-url-token. The source byte stays in the token for offsets; the value decodes it. Ported from WordPress/php-toolkit bb1c7069 with its CSSPreprocessingTest cases.
Ident sequences and unquoted url values advance over runs of bytes that need no special handling with one strspn() call instead of one PHP loop iteration per byte. Tokenizing 2,000 copies of a three-declaration style string (url, dimension, custom property) drops from 54 ms to 14 ms. Ported from WordPress/php-toolkit 8e411e7c.
decode_range() and its escape helpers take the CSS source as an argument, so code that holds the source and a token's offsets can decode the token after the cursor has moved on.
get_token() returns the current token as an object that keeps its values after the processor moves on: type, flag, value, unit, start, length, and whether it ended at its closing delimiter. The value is decoded from the source on the first call to get_value(). The processor stays a cursor; get_token_value() reads the same object, so the value decoding is in one place.
The token processor reads CSS; the new class writes it. serialize_string() and serialize_ident() move unchanged, with their tests.
WP_CSS_Token_Serializer::serialize() writes tokens read by the processor as CSS that reads as the same tokens. Comments are dropped, and each whitespace run, including one broken by comments, becomes one space. An empty comment, /**/, separates each token pair the §9 table marks when no whitespace is between them. Tokens are escaped again from their decoded values: a `\` delim keeps its newline, a hash is escaped by its type flag, and a unit that reads as an exponent has its `e` escaped. Strings and urls cut off by the end of the input are written closed, and blocks still open at the end are closed in reverse order. Bad-string and bad-url tokens have no value; they are written as a quote and a newline, and as `url(()`. serialize_string() still normalizes newlines in its argument. Token values are quoted without that step, so an escaped CR or FF in a string keeps its code point.
CSS Syntax 3 §9 allows a tokenizer to preserve comments when they have no effect on parsing. The processor emits them so callers can find comments by byte offset.
serialize_string() replaced CR, CRLF and FF in its argument with LF, as if the argument were CSS source. Its output is escaped and never preprocessed, so the replacement changed the value: a string token written from "a\D b" read back with a newline. CR and FF are now written as \D and \C, so the string reads back as the same value; NULL still becomes U+FFFD. A caller that wants normalized line endings normalizes before calling. serialize() writes string tokens with serialize_string() again, and the private quote_string() is removed.
The CSS Syntax 3 §9 table compares token types in pairs. It misses two sequences that the tokenizer reads as a CDO or CDC token: `<`, `!` and a token that starts with `--` spell `<!--`, and an ident `--` followed by `>` spells `-->`. So `<!/**/--x` was written as `<!--x` (CDO, ident `x`) and `--/**/>` as `-->`. An empty comment is now written before the last token of each, so the output reads as the same tokens.
A token list can hold tokens that have no valid CSS form, so the public way to write CSS will be at the declaration level, where the style attribute processor exposes only valid declarations. WP_CSS_Token_Serializer is marked @access private; its methods stay public so other core classes can call them. serialize_ident() and serialize_string() had no caller outside the serializer. They become escape_ident(), public for property names, and the private escape_string(); url values use the private escape_url(), which now returns the whole url token. CSSOM defines "serialize an identifier" and "serialize a string" with a different escape set, so the new names avoid suggesting those algorithms. The string escaping tests now run through serialize(). The class documentation states what the output guarantees: it parses to the same CSS as the tokens. It is not always valid CSS: bad-string and bad-url tokens are written as the same bad tokens.
escape_ident() repeated escape_name()'s loop with three changes for the first code point. It now handles a lone `-` and a leading digit, or `-` and a digit, and escapes the rest with escape_name(). Output is unchanged: 80,565 generated values of up to five code points escape the same as before.
Hash flag round-trips get a data provider, with the expected flag of each input checked on both sides. serialize() is tested with a generator, since it accepts any iterable, and with a DEL delim.
String tokens wrote U+0001 to U+001F and U+007F raw, except LF, CR and FF. They now escape every code point in that range, as CSSOM's "serialize a string" does, so a raw tab is written `\9 `. Identifiers, hash names and urls already escaped these code points.
CSSOM's "escape a character as code point" writes the hex digits in lowercase, so `<` is written `\3c ` instead of `\3C `. CSS reads either case, so the output parses as before.
CSS Syntax limits the non-ASCII code points an identifier may contain to those HTML allows in custom element names. The tokenizer, like Chrome, still reads every code point above U+007F as an identifier code point, and escape_name() wrote them all raw: `a\a0 b` was written with a raw no-break space, which a tokenizer that follows the spec reads as an ident, a delim and an ident. escape_name() now hex-escapes the code points outside the set, so identifiers, hash names and dimension units read the same under both rules.
…al-style-attribute
…e-attribute-append
…-style-attr-processor
The parameter accepted only 'UTF-8' and returned null for any other value, so create() was typed static|null and every caller checked for a null it could not get. Every caller passes text that is already decoded: an attribute value or a string from the database. An optional trailing parameter can be added in a later release without breaking callers if the tokenizer ever reads other encodings; removing one after release cannot.
create() no longer returns null, so the four checks could never be true.
…al-style-attribute
…e-attribute-append
…processor-foundations
…-style-attr-processor
…al-style-attribute
…e-attribute-append
Replace the static create() factory and private constructor with a public __construct( string $css ). A factory is useful when creation can fail, has several modes, or picks the returned class. Since the encoding argument was removed, create() does none of these. WP_HTML_Tag_Processor and WP_Block_Processor also take their input through a public constructor. This also removes the PHPStan new.static error: a subclass constructor with another signature would make new static( $css ) throw a TypeError.
utf8_ord() returns an int, and _wp_scan_utf8() has already confirmed that the bytes are one valid code point.
…-style-attr-processor
utf8_ord() returns an int, and _wp_scan_utf8() has already confirmed that the bytes are one valid code point.
Replace the private constructor and the static create() method with a public constructor, as WP_CSS_Token_Processor now has. The parameter stays untyped so that PHP 7.4 can accept `true` and `null`. WP_HTML_Tag_Processor::get_attribute() returns `true` for a boolean style attribute and `null` for a missing one. The constructor reads both as an empty style, so its result can be passed in directly. Any other type that is not a string throws a TypeError, as create() did.
A successful mutation parsed the whole style twice, on the same text.
The verification step built the edited text, parsed it as a candidate
processor and checked the candidate's declarations. Then
apply_lexical_updates() rebuilt that text from the update queue and
parsed it again.
The verification methods now return the parsed candidate, or null where
they returned false, and the mutators take its style text, tokens and
declarations. The checks are unchanged, and so is the cursor handling.
Every mutator queued one update and applied it before returning, so the
queue never held more than one update: queue_lexical_update() always
found it empty, and has_queued_append_at() always returned false. The
test suite passed with assertions for both, and with an assertion that
the sorted and merged output equals the single replacement. With the
candidate's text as the updated style, the queue, its sort and merge,
and the separator check that read it are removed. get_updated_style()
returns the current text.
Timings in µs per call, before and after. PHP 8.5.12 CLI, opcache and
JIT off. N declarations of five kinds, 36 bytes each; 10 calls per run
(remove_declaration() at N=5: 5 calls, removing all); median of runs
within 1 s, one process per variant; median of 3 interleaved rounds.
N=5 N=20 N=100
set_value() 124 -> 74 395 -> 212 1787 -> 916
set_important() 111 -> 55 383 -> 191 1789 -> 889
remove_declaration() 42 -> 21 264 -> 132 1670 -> 831
append_declaration() 180 -> 105 476 -> 263 1989 -> 1103
The before and after runs produce the same output for every case.
The processor has no method that sets a property. The class docblock now shows how: remove the property's declarations in one named next_declaration() loop, then append the new declaration. It covers setting several properties in one unnamed pass, since the cursor moves forward only, and how the appended declaration's position interacts with shorthands in the cascade. A test confirms that remove_declaration() in a named loop removes every matching declaration, including ones that follow a removed one. The example outputs were checked by running the examples as written.
next_token() advances the cursor, so two calls on one processor can return different results. Without the annotation PHPStan reuses the first call's result for the second, which reports a later "if ( $processor->next_token() )" as always true and its else branch as unreachable. WP_HTML_Tag_Processor::next_token() carries the same annotation.
…-style-attr-processor
…al-style-attribute
…e-attribute-append
The processor's constructor treats true and null, which get_attribute() returns for a valueless or missing attribute, as an empty style.
…al-style-attribute
…e-attribute-append
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caution
This is a work in progress draft and is not ready for any serious review.
Use
WP_HTML_Style_Attribute_Processor::append_declaration()when background and dimensions block supports add generated styles to an existingstyleattribute.Appending a semicolon and CSS text does not end an open string, comment or block. The generated declaration can become part of the preceding value. The processor closes that value before appending:
Well-formed styles change only in formatting. Existing wrapper expectations use
assertEqualHTML()to compare declarations.Depends on #14783. This PR adds the two commits after
977555cadf.Reported validation: 27 background support tests and 5 dimensions support tests pass.
Trac ticket: TBD
Use of AI Tools
AI assistance: Yes
Tools: Codex, Claude Code
Models: GPT-5, Claude Opus, Claude Fable 5.1
Used for: implementation, tests, specification drafting, review and this description.
This pull request is for code review only. Keep other discussion in the Trac ticket. Do not merge. See GitHub Pull Requests for Code Review in the Core Handbook.