Skip to content

Fix string decoding for latin1, binary, ascii, base64 and base64url - #324

Open
keichi wants to merge 4 commits into
masterfrom
fix-string-encodings
Open

keichi wants to merge 4 commits into
masterfrom
fix-string-encodings

Conversation

@keichi

@keichi keichi commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Problem

sanitizeEncoding accepts ascii, latin1, binary, base64 and base64url, but generateString passes every non-hex encoding to new TextDecoder(encoding):

  • binary, base64, base64url: TextDecoder doesn't support them, so parse() throws RangeError.
  • latin1, ascii: WHATWG maps both to windows-1252, so byte 0x80 decodes to "€" instead of "\x80" (Node Buffer behavior).

Repro:

Parser.start().string("s", { length: 1, encoding: "base64" }).parse(Uint8Array.of(0x41));
// RangeError: The "base64" encoding is not supported
Parser.start().string("s", { length: 1, encoding: "latin1" }).parse(Uint8Array.of(0x80)).s;
// "€" (expected "\x80")

Fix

Handle these encodings the same way hex already is. Two small self-contained helpers are passed in through ctx.addImport. No Buffer is used, so it still works in browsers:

  • latin1/binary: String.fromCharCode.apply over 8192-byte chunks (bytesToLatin1)
  • ascii: clear the high bit of each byte in a loop, like Node, then UTF-8 TextDecoder, which is exact because every byte is now < 0x80 (bytesToAscii)
  • base64: btoa(latin1); base64url: same with +/ mapped to -_ and padding removed (same as Node)

I skipped Uint8Array.prototype.toBase64 because Node 22 (the engine floor) doesn't ship it unflagged. btoa is available in every supported runtime.

generateString now builds the decode expression in one place for all four length modes, so the hex/TextDecoder ternary isn't repeated four times.

The README said "all encodings supported by TextDecoder", but the allow-list rejects things like utf-16le and shift_jis. I updated the README to list what's actually accepted. Widening the allow-list is a separate change.

Regression tests in test/primitive_parser.ts check every encoding against Buffer#toString() output, including a 20000-byte buffer that spans several chunks.

Decode time (Node 26, µs per call), per-byte Array.from vs this PR:

size latin1 Array.from latin1 chunked ascii (.map + chunked) ascii (loop + TextDecoder)
1 KB 20.2 2.8 7.1 0.8
64 KB 1293 155 447 39

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant