Add 40 selective languages and reduce HTML rendering overhead

.changeset/selective-languages-and-rendering.md

@@ -0,0 +1,9 @@
+---
+'@tanstack/highlight': minor
+---
+
+Add 40 independently importable language definitions, bringing the built-in registry to 79 including plaintext. Named language-barrel imports and direct language subpaths are checked for tree shaking. New definitions provide lightweight lexical coverage using shared scanning helpers and vendored Highlight.js vocabularies, without adding runtime dependencies. The full registry grows; existing selective language imports do not retain the new definitions.
+
+Serialize ordinary HTML blocks directly and avoid replacement work for text that needs no HTML escaping. Keep token output and decorated rendering unchanged. Measured local docs highlighting takes 20% less time in Node and 27% less in Bun for 73 additional gzip bytes in core.
+
+Preserve all existing language names and aliases, including `cmd` for `shell`. The new Windows Batch definition uses `batch`, `bat`, and `dos`.

NOTICE.md

@@ -0,0 +1,35 @@
+# Third-party notices
+
+The vocabulary lists in the following language definitions are adapted from Highlight.js 11.11.1 (https://github.com/highlightjs/highlight.js/tree/11.11.1/src/languages). The scanning helpers and lightweight lexical definitions are maintained here; the Highlight.js runtime is not included.
+
+ada, applescript, awk, batch, c, clojure, coffeescript, crystal, d, elixir, elm, erlang, fortran, fsharp, glsl, graphql, groovy, haskell, ini, julia, latex, less, lisp, makefile, matlab, nix, objectivec, ocaml, pascal, powershell, properties, protobuf, r, scala, scss, vbnet, verilog, vhdl, wat, x86asm.
+
+BSD 3-Clause License
+
+Copyright (c) 2006, Ivan Sagalaev.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -68,7 +68,7 @@ const result = highlighter.highlight(`const node = <Button />`, {
 
 `html` does not import script or style languages. Registering `js`, `ts`, or `css` enables delegation for matching embedded regions without increasing the standalone HTML module.
 
-For convenience, the root entry contains every shipped language:
+For convenience, the root entry contains every shipped language. Use selective registration above when browser size matters:
 
 ```ts
 import { highlight } from '@tanstack/highlight'
@@ -195,9 +195,9 @@ Available themes: Aurora X, Dracula, GitHub Dark, GitHub Light, Gruvbox Dark, Gr
 
 ## Languages
 
-`apache`, `cmake`, `cpp`, `csharp`, `css`, `dart`, `diff`, `dockerfile`, `ejs`, `env`, `go`, `html`, `http`, `java`, `js`, `json`, `jsx`, `kotlin`, `lua`, `markdown`, `mermaid`, `nginx`, `perl`, `php`, `plaintext`, `python`, `ruby`, `rust`, `scheme`, `shell`, `sql`, `svelte`, `swift`, `toml`, `ts`, `tsrx`, `tsx`, `vue`, and `yaml`.
+`ada`, `apache`, `applescript`, `awk`, `batch`, `c`, `clojure`, `cmake`, `coffeescript`, `cpp`, `crystal`, `csharp`, `css`, `d`, `dart`, `diff`, `dockerfile`, `ejs`, `elixir`, `elm`, `env`, `erlang`, `fortran`, `fsharp`, `glsl`, `go`, `graphql`, `groovy`, `haskell`, `html`, `http`, `ini`, `java`, `js`, `json`, `jsx`, `julia`, `kotlin`, `latex`, `less`, `lisp`, `lua`, `makefile`, `markdown`, `matlab`, `mermaid`, `nginx`, `nix`, `objectivec`, `ocaml`, `pascal`, `perl`, `php`, `plaintext`, `powershell`, `properties`, `protobuf`, `python`, `r`, `ruby`, `rust`, `scala`, `scheme`, `scss`, `shell`, `sql`, `svelte`, `swift`, `toml`, `ts`, `tsrx`, `tsx`, `vbnet`, `verilog`, `vhdl`, `vue`, `wat`, `x86asm`, `yaml`.
 
-Each language is available from `@tanstack/highlight/languages/<name>`. The aggregate `@tanstack/highlight/languages` entry can tree-shake, while direct subpaths make isolation explicit. Importing only core helpers from the root entry also removes unused language registrations in a compatible bundler.
+All 79 definitions are independently importable. Each language is available from `@tanstack/highlight/languages/<name>`. The aggregate `@tanstack/highlight/languages` entry can tree-shake, while direct subpaths make isolation explicit. Importing only core helpers from the root entry also removes unused language registrations in a compatible bundler.
 
 ## Output Contract
 
@@ -214,13 +214,15 @@ Local browser bundles, minified with esbuild and compressed independently. KB us
 
 | Registration | Minified | Gzip | Brotli |
 | --- | ---: | ---: | ---: |
-| Core, no languages | 3.84 KB | 1.82 KB | 1.66 KB |
-| Core + TSX | 10.11 KB | 4.26 KB | 3.90 KB |
-| Octane MDX + TypeScript | 13.86 KB | 5.59 KB | 5.17 KB |
-| Nine-language docs set | 16.04 KB | 6.20 KB | 5.66 KB |
-| All 39 languages | 52.74 KB | 17.73 KB | 15.93 KB |
+| Core, no languages | 4.03 KB | 1.89 KB | 1.72 KB |
+| Core + TSX | 10.34 KB | 4.39 KB | 4.01 KB |
+| Octane MDX + TypeScript | 14.09 KB | 5.72 KB | 5.23 KB |
+| Nine-language docs set | 16.27 KB | 6.31 KB | 5.77 KB |
+| All 79 languages | 82.09 KB | 27.75 KB | 24.61 KB |
 
-The following comparison was measured before the 1.0 property-context correction. Re-run the comparison commands below for current timings.
+Ordinary HTML blocks skip the intermediate render tree and only escape text when needed. A local comparison of identical output measured 20% less time for the docs corpus in Node and 27% less in Bun, adding 73 gzip bytes to core and 67 to the TSX bundle. [Measurements and reproduction](reports/language-expansion/README.md).
+
+The following external-library comparison was measured before the 1.0 property-context correction. Re-run the comparison commands below for current timings.
 
 On 80 real JavaScript/TypeScript/JSX/TSX TanStack docs fixtures repeated across 5,040 blocks, using the median of three runs after warmup:
 

docs/guides/performance.md

@@ -8,15 +8,15 @@ CI measures selective browser bundles and highlighting performance on real docum
 
 ## Bundle profiles
 
-`pnpm run size` builds twenty-six browser profiles with esbuild and measures minified, gzip, and Brotli bytes independently. It also checks that helper, adapter, and selective language imports retain only the requested modules.
+`pnpm run size` builds 66 browser profiles with esbuild and measures minified, gzip, and Brotli bytes independently. It also checks that helper, adapter, and selective language imports retain only the requested modules.
 
 | Profile | Registered languages | Current gzip | CI budget |
 | --- | --- | ---: | ---: |
-| Core | None | 1.82 KB | 2.0 KB |
-| TSX | TSX | 4.26 KB | 4.35 KB |
-| Octane | TypeScript plus Octane MDX adapter | 5.59 KB | 5.7 KB |
-| Docs | CSS, HTML, JS, JSON, JSX, Markdown, Shell, TS, TSX | 6.20 KB | 6.3 KB |
-| All | All 39 definitions | 17.73 KB | 18.0 KB |
+| Core | None | 1.89 KB | 2.10 KB |
+| TSX | TSX | 4.39 KB | 4.45 KB |
+| Octane | TypeScript plus Octane MDX adapter | 5.72 KB | 5.80 KB |
+| Docs | CSS, HTML, JS, JSON, JSX, Markdown, Shell, TS, TSX | 6.31 KB | 6.40 KB |
+| All | All 79 definitions | 27.75 KB | 28.00 KB |
 
 KB uses 1,000 bytes. Core helpers imported from the root tree-shake to the same engine size. The standalone theme helper is 695 gzip bytes.
 
@@ -26,19 +26,24 @@ Selective profiles are the primary metric. The all-language profile exists to pr
 
 The committed corpus contains 334 real code fences sampled from TanStack documentation, with up to twenty samples per normalized language.
 
-`pnpm run bench` measures tokenization, HTML, Markdown, HAST, line numbers, long decorated blocks, and dedicated C#, C++, CMake, Dart, Java, Kotlin, Lua, Perl, PHP, Ruby, Rust, and Swift samples. Each profile reports the median of three samples after two warmup passes, with a 1.2 second CI budget. The main highlighting profile processes at least 10,000 blocks.
+`pnpm run bench` measures tokenization, HTML, Markdown, HAST, line numbers, long decorated blocks, the 40 additional language fixtures, and dedicated C#, C++, CMake, Dart, Java, Kotlin, Lua, Perl, PHP, Ruby, Rust, and Swift samples. Each profile reports the median of three samples after two warmup passes, with a 1.2 second CI budget. The main highlighting profile processes at least 10,000 blocks.
 
-A local before-and-after review used the same minified bundle settings, fixtures, and benchmark harness on macOS arm64 with Node 24.15.0:
+The 40 added definitions each cost 2.8 to 3.8 KB gzip including core. Each stays below a 4 KB gzip budget. Every definition is tested through both its direct subpath and a named barrel import, and unrelated language code must be absent from the resulting bundle.
 
-| Workload | Blocks | Before | After |
-| --- | ---: | ---: | ---: |
-| Highlight | 10,020 | 229 ms | 118 ms |
-| TanStack Markdown | 10,020 | 244 ms | 111 ms |
-| HAST | 5,010 | 148 ms | 40 ms |
-| 1,000-line numbered blocks | 50 | 358 ms | 68 ms |
-| 1,000-line decorated blocks | 50 | 426 ms | 150 ms |
+## Rendering tradeoff
 
-Generated HTML byte totals were unchanged. The core avoids rescanning earlier tokens for each line, and HAST adapters skip HTML serialization. Highlighter bundles grew by 41 to 173 gzip bytes across the five profiles, while root helper imports and theme CSS generation became smaller. The Octane gzip budget increased from 5.2 KB to 5.5 KB to accommodate correct fence metadata and attribute preservation.
+Ordinary HTML blocks serialize tokens directly instead of allocating a render tree. Line numbers and decorations retain the structured rendering path. Escaping first checks whether the string contains an HTML-sensitive character. The token arrays and generated HTML remain identical.
+
+On an Apple M5 Pro, fifteen alternating samples after warmup measured:
+
+| Workload | Node 24.15.0 time reduction | Bun 1.3.14 time reduction |
+| --- | ---: | ---: |
+| All 334 docs snippets | 20% | 27% |
+| 1,000-line TypeScript block | 29% | 32% |
+
+Core grows by 73 gzip bytes and the TSX bundle by 67. Core-dependent budgets allow up to 200 extra minified bytes and 100 compressed bytes. The all-language budget grows separately to cover the 40 additional definitions. Selective TSX imports retain none of those definitions or their scanner.
+
+The comparison uses the same minified browser bundle settings, asserts identical tokens and HTML, warms each implementation for 200 ms, and measures fifteen alternating samples in batches of at least 30 ms. These local measurements exclude module initialization and browser layout. The goal is inexpensive synchronous highlighting, not matching a larger engine's maximum throughput.
 
 ## Comparison scripts
 
@@ -75,5 +80,3 @@ Context-aware fixes are welcome when they solve common docs code. A change shoul
 - Generated HTML size when output structure changes
 
 The correct response to a crossed budget is to inspect the behavior and architecture. Budgets can move when a measured quality improvement justifies the bytes, but the tradeoff must be explicit.
-
-The 1.0 property-context correction adds roughly 230 gzip bytes to the TSX profile without changing core. Local Node 26 gzip results differ slightly from CI compression: the CI docs profile is 6,217 bytes against a 6,300-byte budget, and the all-language budget keeps a comparable allowance above its local measurement.

docs/installation.md

@@ -60,7 +60,7 @@ export const highlighter = createHighlighter({
 })
 ```
 
-The root entry is useful for prototypes, server-only scripts, or sites where the roughly 18 KB gzip all-language build is acceptable:
+The root entry is useful for prototypes, server-only scripts, or sites where the roughly 28 KB gzip all-language build is acceptable:
 
 ```ts
 import { highlight } from '@tanstack/highlight'