Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
sanitizeEncodingacceptsascii,latin1,binary,base64andbase64url, butgenerateStringpasses every non-hex encoding tonew TextDecoder(encoding):binary,base64,base64url: TextDecoder doesn't support them, soparse()throwsRangeError.latin1,ascii: WHATWG maps both to windows-1252, so byte0x80decodes to"€"instead of"\x80"(NodeBufferbehavior).Repro:
Fix
Handle these encodings the same way
hexalready is. Two small self-contained helpers are passed in throughctx.addImport. NoBufferis used, so it still works in browsers:latin1/binary:String.fromCharCode.applyover 8192-byte chunks (bytesToLatin1)ascii: clear the high bit of each byte in a loop, like Node, then UTF-8TextDecoder, which is exact because every byte is now < 0x80 (bytesToAscii)base64:btoa(latin1);base64url: same with+/mapped to-_and padding removed (same as Node)I skipped
Uint8Array.prototype.toBase64because Node 22 (the engine floor) doesn't ship it unflagged.btoais available in every supported runtime.generateStringnow builds the decode expression in one place for all four length modes, so the hex/TextDecoder ternary isn't repeated four times.The README said "all encodings supported by TextDecoder", but the allow-list rejects things like
utf-16leandshift_jis. I updated the README to list what's actually accepted. Widening the allow-list is a separate change.Regression tests in
test/primitive_parser.tscheck every encoding againstBuffer#toString()output, including a 20000-byte buffer that spans several chunks.Decode time (Node 26, µs per call), per-byte
Array.fromvs this PR:.map+ chunked)