Repository navigation
Terminal: count East Asian Wide characters as two columns - #2894
Conversation
e36f2ef to
94faeaf
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Combining characters are discarded, and supplementary or edited wide characters can corrupt storage, rendering, cursor positioning, and copied text.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 4
Open (5)
Combining marks are discarded instead of retained with base characters · New Wide-character filler detection misclassifies blanks after surrogate pairs · New Repaint fails to recognize the second cell of supplementary characters · New Proportional rendering draws surrogate pairs as separate characters · New Copy logic incorrectly treats blanks after emoji as filler cells · New
What changed in this PR
Adds East Asian wide-character support to the terminal’s model, emulator, rendering, copying, and repaint behavior.
Changes:
- Introduces Unicode width classification and filler cells.
- Updates emulator positioning, wrapping, insertion, rendering, and copying.
- Adds width and emulator tests plus the required bundle version increment.
| File | Description |
|---|---|
CharWidth.java |
Defines Unicode display widths and filler detection. |
VT100EmulatorBackend.java |
Adds cell-aware writing and wide-character handling. |
TextLineRenderer.java |
Renders wide and supplementary characters. |
TextCanvas.java |
Expands repaint ranges across wide glyphs. |
AbstractTextCanvasModel.java |
Removes fillers from copied text. |
CharWidthTest.java |
Tests width classification and fillers. |
AllTestSuite.java |
Registers the new width tests. |
VT100EmulatorBackendTest.java |
Tests placement, wrapping, overwriting, and insertion. |
META-INF/MANIFEST.MF |
Increments the bundle service version. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Rebased on current master and addressed the Copilot review in 5bb7086. The GitHub Actions failures on the previous head were not from this change: the build stopped in org.eclipse.core.filesystem with "Baseline and reactor have the same fully qualified version, but different content", because the old base still had 1.11.500 while master has moved to 1.12.0. The rebase should clear it. @akurtakov thanks for triggering the review. |
A supplementary character is stored as its two surrogates, one per cell, so it covers both of its cells itself. The null after it is an ordinary empty cell, not a filler, and copying must keep it as a space. A repaint that starts on the low surrogate is widened to the high one, like a repaint starting on a filler. The proportional font path draws the surrogate pair as one string instead of one half at a time. Found by review of eclipse-platform#2894.
|
I would like to be able to test this myself before going on further. Do you think you can give me clear instruction how to test and see the effect of this change? |
|
Hi @akurtakov, just checking whether the steps above worked for you, or if a short screen recording or a ready-built bundle would make testing easier. Happy to adjust the PR if anything looks off. |
|
For context, this is a long-standing user-facing problem, not just a spec detail:
Both are the case this PR fixes: a wide character took one column, so everything after it on the line was drawn in the wrong place. The same happens with Korean and Japanese, and with tools that print them, such as Claude Code or Codex CLI in Korean. |
A supplementary character is stored as its two surrogates, one per cell, so it covers both of its cells itself. The null after it is an ordinary empty cell, not a filler, and copying must keep it as a space. A repaint that starts on the low surrogate is widened to the high one, like a repaint starting on a filler. The proportional font path draws the surrogate pair as one string instead of one half at a time. Found by review of eclipse-platform#2894.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved wide-character handling defects can crash one-column terminals, corrupt text during wrapping and editing, and break cursor rendering and word selection.
6 open issues
Reserve insert space after resolving the destination row · New Clear stale last-cell content before wrapping · New Use a single-cell replacement when wide characters exceed terminal width · New Preserve wide-character fillers during hover word scanning · New Clean up wide-character boundaries before erase, insert, and delete · New Draw the complete owning glyph for wide-character cursor overlays · New
3 resolved since last review
Review effort: Balanced
|
I was really busy with other things, sorry for the delay. Copilot found more problems, some of them look genuine, others like the emoji look like preexisting problems. |
Erasing, inserting or deleting characters moved or cleared single cells, so an operation that touched one cell of a wide character left the other behind: a glyph with its filler erased, a filler with nothing in front of it, or a glyph pushed into the last column with its filler pushed off the line. As in xterm, an operation that touches either cell of a wide character, or of a surrogate pair, now blanks the whole character first, and a character whose second cell an insertion pushes past the margin is blanked. Found by review of eclipse-platform#2894.
In insert mode the room for the whole string was made on the cursor's line before anything was written, so a wide character that then wrapped to the next line overwrote what was there while the room sat unused on the line above. The room is now made for each run where it is written, after any wrap, which is what xterm does. A wide character that does not fit in the last column wraps to the next line, and the cell it leaves unused is now erased instead of keeping what was there before, a wide character half in it included. In a terminal one column wide a character beyond the BMP cannot be stored, as its two chars do not fit in the one cell, and writing it threw an IllegalArgumentException. It is written as U+FFFD instead; a wide character of the BMP keeps its one cell, without a filler. Found by review of eclipse-platform#2894.
The word found for a hover or a double click ended at every null, and the second cell of a wide character holds one, so a path such as /tmp/한글.txt was cut at its first syllable. The filler of a wide character is now part of the word, as the character is, and is left out of the word's text; cell coordinates are unchanged. Found by review of eclipse-platform#2894.
The cursor drew only the one cell it was on: on the second cell of a wide character it painted a reversed blank over the right half of the glyph, and on a character beyond the BMP it drew half a surrogate pair. On either cell of a wide character the cursor now covers both, drawing the whole glyph reversed and clipped to its two cells, as xterm and Windows Terminal do. When the cursor appears or is switched on or off, the cells beside it are repainted too, as they already were when it moved away or blinked, so no half of a two cell cursor is left behind. Found by review of eclipse-platform#2894.
|
Thank you for the review, and no worries about the delay. I went through all six Copilot findings: each one was real, including the one-column case, which this PR introduced (a surrogate pair was written into a single cell and threw). They are fixed in four new commits, each with a regression test that fails without the fix:
All terminal tests pass (326) at every commit. |
There was a problem hiding this comment.
🟡 Changes recommended
Zero-count edits can corrupt wide characters, and wide-whitespace filler cells are misclassified, causing inconsistent word selection.
3 open issues
5 resolved since last review
Draw the complete owning glyph for wide-character cursor overlays Clean up wide-character boundaries before erase, insert, and delete Use a single-cell replacement when wide characters exceed terminal width Clear stale last-cell content before wrapping Reserve insert space after resolving the destination row
Review effort: Balanced
Inserting, deleting or erasing zero characters still blanked a wide character under or next to the cursor, since the wide character cleanup ran before the count was looked at. A count of zero now returns at once. The filler in the second cell of a wide character takes the boundary status of its character instead of always being part of a word, so both cells of an ideographic space (U+3000) end a word, while a path with Hangul in it is still one word. Found by review of eclipse-platform#2894.
|
The two new findings were real as well and are fixed in a635667, each with a regression test: a count of zero now inserts, deletes or erases nothing (before, the wide-character cleanup ran first and could blank a neighbouring half), and the filler of a wide character takes the boundary status of its character, so both cells of an ideographic space end a word while a path with Hangul in it stays one word. All 328 terminal tests pass. |
|
As each of the intermediate commits have issues identified, please squash them in one and for future changes amend. |
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed for subtle interactions where wide-character padding truncates path selection, partial-glyph selection introduces artificial blanks, and zero-width input prematurely triggers wrapping or scrolling.
2 resolved since last review
Review effort: Balanced
|
Thanks, fixed in 9b32b3e: a selection that starts on the second cell of a wide character now starts on its first, and one that ends on the first cell ends on the second, so dragging from the right half of 한 in 한글 copies 한글 and the highlight covers the whole glyph. A new test in TextCanvasModelTest covers it. The commits are squashed into one, and further changes will be amended into it. |
The emulator assumed every character occupies one cell, so Hangul, Han and Kana text, fullwidth forms and emoji were placed one column short per character and the screen fell apart as soon as a program laid text out for a real terminal (line editors, curses UIs, Ink based CLIs). Add CharWidth, a UAX eclipse-platform#11 East Asian Width lookup: Wide and Fullwidth count as two columns, combining marks and controls as zero, Ambiguous as one, as UAX eclipse-platform#11 recommends outside an East Asian legacy context. The emulator advances the cursor by that width and stores a NUL filler in the second cell of a wide character, never splits one across the right margin and erases the margin cell it leaves unused. Overwriting, erasing, inserting or deleting either half of a wide character blanks the whole character, and a count of zero edits nothing. Insert mode makes room in cells, run by run, on the line the text is written to. A character beyond the BMP takes two cells and leaves no filler in a one-column terminal. The renderer skips the fillers so a fixed width font draws a wide glyph over both cells, falls back to placing each character at its own cell when the font does not advance exactly one cell per column, draws a character beyond the BMP whole and draws the block cursor over both cells. A partial repaint that starts on the second cell of a wide character is widened to its first, and copying drops the fillers. A selection that starts or ends on half of a wide character covers the whole character, both when copied and when highlighted, and the filler takes the word boundary status of its character when selecting the word under the mouse. Tests cover the width table, placement, the margin, overwriting and erasing halves, insert and delete, zero counts, the cursor, selection and word selection.
|
This one is already significant improvement! Thanks for that! I'm merging it in. Please rebase whichever PR is next on top of master with details how to test. |




The terminal emulator assumes every character occupies one cell. Hangul, Han and Kana text, fullwidth forms and emoji are East Asian Wide and occupy two, so every such character put the cursor one column short and the screen fell apart as soon as a program laid text out for a real terminal — line editors, curses UIs, Ink-based CLIs.
This adds
CharWidth, a UAX #11 East Asian Width lookup (Wide/Fullwidth = 2, combining marks and controls = 0, Ambiguous = 1 as UAX #11 recommends outside an East Asian legacy context), and makes the emulator and renderer follow it:Tests:
CharWidthTest(width table) and four cases inVT100EmulatorBackendTest(placement, margin, overwriting halves, insert mode). All existing terminal tests pass.Verified on Windows 11 with a Korean shell session, vim, htop and Claude Code inside the Eclipse Terminal; compared against Windows Terminal for column alignment.