Skip to content

Terminal: count East Asian Wide characters as two columns - #2894

Merged
akurtakov merged 1 commit into
eclipse-platform:masterfrom
insjang:eaw-width
Nov 3, 2026
Merged

akurtakov merged 1 commit into
eclipse-platform:masterfrom
insjang:eaw-width

Conversation

@insjang

@insjang insjang commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

The terminal emulator assumes every character occupies one cell. Hangul, Han and Kana text, fullwidth forms and emoji are East Asian Wide and occupy two, so every such character put the cursor one column short and the screen fell apart as soon as a program laid text out for a real terminal — line editors, curses UIs, Ink-based CLIs.

This adds CharWidth, a UAX #11 East Asian Width lookup (Wide/Fullwidth = 2, combining marks and controls = 0, Ambiguous = 1 as UAX #11 recommends outside an East Asian legacy context), and makes the emulator and renderer follow it:

  • the cursor advances by width; the second cell of a wide character holds a NUL filler
  • a wide character is never split across the right margin
  • overwriting either half of a wide character blanks the other half
  • insert mode counts cells, not characters
  • the renderer skips fillers so a fixed-width font draws the glyph over both cells, falls back to placing each character at its own cell when the font does not advance exactly one cell per column, and draws characters beyond the BMP whole
  • a partial repaint starting on the second cell of a wide character is widened to its first
  • copying drops the fillers

Tests: CharWidthTest (width table) and four cases in VT100EmulatorBackendTest (placement, margin, overwriting halves, insert mode). All existing terminal tests pass.

Verified on Windows 11 with a Korean shell session, vim, htop and Claude Code inside the Eclipse Terminal; compared against Windows Terminal for column alignment.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Combining characters are discarded, and supplementary or edited wide characters can corrupt storage, rendering, cursor positioning, and copied text.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 4 Medium severity · 1 Low severity

Open (5)
What changed in this PR

Adds East Asian wide-character support to the terminal’s model, emulator, rendering, copying, and repaint behavior.

Changes:

  • Introduces Unicode width classification and filler cells.
  • Updates emulator positioning, wrapping, insertion, rendering, and copying.
  • Adds width and emulator tests plus the required bundle version increment.
File Description
CharWidth.java Defines Unicode display widths and filler detection.
VT100EmulatorBackend.java Adds cell-aware writing and wide-character handling.
TextLineRenderer.java Renders wide and supplementary characters.
TextCanvas.java Expands repaint ranges across wide glyphs.
AbstractTextCanvasModel.java Removes fillers from copied text.
CharWidthTest.java Tests width classification and fillers.
AllTestSuite.java Registers the new width tests.
VT100EmulatorBackendTest.java Tests placement, wrapping, overwriting, and insertion.
META-INF/​MANIFEST.MF Increments the bundle service version.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

    54 files  ± 0      54 suites  ±0   40m 31s ⏱️ + 1m 41s
 4 950 tests +23   4 928 ✅ +23   22 💤 ±0  0 ❌ ±0 
12 741 runs  +69  12 587 ✅ +69  154 💤 ±0  0 ❌ ±0 

Results for commit 34baa14. ± Comparison against base commit c519dfe.

♻️ This comment has been updated with latest results.

@insjang

insjang commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master and addressed the Copilot review in 5bb7086.

The GitHub Actions failures on the previous head were not from this change: the build stopped in org.eclipse.core.filesystem with "Baseline and reactor have the same fully qualified version, but different content", because the old base still had 1.11.500 while master has moved to 1.12.0. The rebase should clear it.

@akurtakov thanks for triggering the review.

insjang added a commit to insjang/eclipse.platform that referenced this pull request Sep 30, 2026
A supplementary character is stored as its two surrogates, one per cell,
so it covers both of its cells itself. The null after it is an ordinary
empty cell, not a filler, and copying must keep it as a space. A repaint
that starts on the low surrogate is widened to the high one, like a
repaint starting on a filler. The proportional font path draws the
surrogate pair as one string instead of one half at a time.

Found by review of eclipse-platform#2894.
@akurtakov

Copy link
Copy Markdown
Member

I would like to be able to test this myself before going on further. Do you think you can give me clear instruction how to test and see the effect of this change?

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@insjang

insjang commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for looking at it! Here is a way to see the effect.

Setup

Run an Eclipse application with org.eclipse.terminal.control from this branch (or install the built bundle), open a local terminal (bash or WSL), and compare with the same commands in any other terminal (xterm, GNOME Terminal, Windows Terminal, VS Code). The terminal font needs CJK glyphs; the default monospace font falls back to a system CJK font on Windows and Linux.

1. Column alignment: every | should end up in the same column, since each Korean/Japanese/full-width character takes two cells:

printf '%s|\n' 'abcd' '가나' '日本' 'ab'

Without the change the | characters are not aligned and wide characters overlap the following text.

2. Cursor movement over wide characters: move the cursor 4 cells to the right from the line start and overwrite:

printf 'ab가cd\r\033[4CX\n'

Expected output is ab가Xd (a, b and 가 fill 4 cells). Without the change the X lands on the wrong cell.

3. Line wrap: this wraps at the right edge, after half as many characters as the terminal has columns:

python3 -c "print('가' * 100)"

Without the change the line does not wrap: each character is drawn one cell after the previous one, so they overlap and only about half of them can be read.

4. Interactive: type echo 한글 test at the prompt, then press Left, Backspace and Delete a few times over the Korean letters. The cursor should stay on the character it edits. The same applies to programs that draw full-screen layouts such as vim, tmux (split borders) or ls with CJK file names.

The model tests in the PR (CharWidthTest and the wide character cases in VT100EmulatorBackendTest) cover the same cases without the UI.

Before and after, with the three commands above (Linux, D2Coding 11pt):
eaw-before-after

@insjang

insjang commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

Hi @akurtakov, just checking whether the steps above worked for you, or if a short screen recording or a ready-built bundle would make testing easier. Happy to adjust the PR if anything looks off.

@insjang

insjang commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

For context, this is a long-standing user-facing problem, not just a spec detail:

  • TM terminal display issue eclipse-cdt/cdt#806 (open since 2024): Chinese text over SSH breaks the line, and the reporter already suspected the column count.
  • Eclipse Terminal on the Marketplace, review from 2018: "Tm-terminal can't display Chinese characters correctly, only half of Chinese characters can be displayed" (UTF-8).

Both are the case this PR fixes: a wide character took one column, so everything after it on the line was drawn in the wrong place. The same happens with Korean and Japanese, and with tools that print them, such as Claude Code or Codex CLI in Korean.

akurtakov pushed a commit to insjang/eclipse.platform that referenced this pull request Oct 30, 2026
A supplementary character is stored as its two surrogates, one per cell,
so it covers both of its cells itself. The null after it is an ordinary
empty cell, not a filler, and copying must keep it as a space. A repaint
that starts on the low surrogate is widened to the high one, like a
repaint starting on a filler. The proportional font path draws the
surrogate pair as one string instead of one half at a time.

Found by review of eclipse-platform#2894.
@akurtakov
akurtakov requested a balanced review from Copilot October 30, 2026 12:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akurtakov

Copy link
Copy Markdown
Member

I was really busy with other things, sorry for the delay. Copilot found more problems, some of them look genuine, others like the emoji look like preexisting problems.

insjang added a commit to insjang/eclipse.platform that referenced this pull request Oct 31, 2026
Erasing, inserting or deleting characters moved or cleared single cells,
so an operation that touched one cell of a wide character left the other
behind: a glyph with its filler erased, a filler with nothing in front of
it, or a glyph pushed into the last column with its filler pushed off the
line. As in xterm, an operation that touches either cell of a wide
character, or of a surrogate pair, now blanks the whole character first,
and a character whose second cell an insertion pushes past the margin is
blanked.

Found by review of eclipse-platform#2894.
insjang added a commit to insjang/eclipse.platform that referenced this pull request Oct 31, 2026
In insert mode the room for the whole string was made on the cursor's
line before anything was written, so a wide character that then wrapped
to the next line overwrote what was there while the room sat unused on
the line above. The room is now made for each run where it is written,
after any wrap, which is what xterm does.

A wide character that does not fit in the last column wraps to the next
line, and the cell it leaves unused is now erased instead of keeping
what was there before, a wide character half in it included.

In a terminal one column wide a character beyond the BMP cannot be
stored, as its two chars do not fit in the one cell, and writing it
threw an IllegalArgumentException. It is written as U+FFFD instead; a
wide character of the BMP keeps its one cell, without a filler.

Found by review of eclipse-platform#2894.
insjang added a commit to insjang/eclipse.platform that referenced this pull request Oct 31, 2026
The word found for a hover or a double click ended at every null, and
the second cell of a wide character holds one, so a path such as
/tmp/한글.txt was cut at its first syllable. The filler of a wide
character is now part of the word, as the character is, and is left
out of the word's text; cell coordinates are unchanged.

Found by review of eclipse-platform#2894.
insjang added a commit to insjang/eclipse.platform that referenced this pull request Oct 31, 2026
The cursor drew only the one cell it was on: on the second cell of a
wide character it painted a reversed blank over the right half of the
glyph, and on a character beyond the BMP it drew half a surrogate pair.
On either cell of a wide character the cursor now covers both, drawing
the whole glyph reversed and clipped to its two cells, as xterm and
Windows Terminal do. When the cursor appears or is switched on or off,
the cells beside it are repainted too, as they already were when it
moved away or blinked, so no half of a two cell cursor is left behind.

Found by review of eclipse-platform#2894.
@insjang

insjang commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for the review, and no worries about the delay. I went through all six Copilot findings: each one was real, including the one-column case, which this PR introduced (a surrogate pair was written into a single cell and threw). They are fixed in four new commits, each with a regression test that fails without the fix:

  1. Insert mode: room for inserted text is now made run by run, on the line the text is actually written to, so a wide character that wraps no longer overwrites the next line. It now matches xterm: content shifts right, and cells pushed past the margin are lost.
  2. Early wrap: when a wide character does not fit in the last column, the cell it leaves unused is erased, together with any wide-character half in it, instead of keeping what was there.
  3. One-column terminal: a character beyond the BMP no longer throws IllegalArgumentException. U+FFFD is written instead, and a wide BMP character keeps its single cell with no filler.
  4. Word selection: the filler in a wide character's second cell no longer ends the word under the mouse, so hover, double-click and word drag select paths like /tmp/한글.txt whole.
  5. Editing: erase, insert and delete of characters, and erase in line/display, blank the whole wide character (or surrogate pair) when they touch either of its cells, as xterm does.
  6. Cursor: on either cell of a wide character, the block cursor covers both cells and draws the whole glyph reversed. The neighbouring cells are repainted when the cursor moves or blinks, so no half cursor is left behind.

All terminal tests pass (326) at every commit.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

insjang added a commit to insjang/eclipse.platform that referenced this pull request Oct 31, 2026
Inserting, deleting or erasing zero characters still blanked a wide
character under or next to the cursor, since the wide character cleanup
ran before the count was looked at. A count of zero now returns at once.

The filler in the second cell of a wide character takes the boundary
status of its character instead of always being part of a word, so both
cells of an ideographic space (U+3000) end a word, while a path with
Hangul in it is still one word.

Found by review of eclipse-platform#2894.
@insjang

insjang commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

The two new findings were real as well and are fixed in a635667, each with a regression test: a count of zero now inserts, deletes or erases nothing (before, the wide-character cleanup ran first and could blank a neighbouring half), and the filler of a wide character takes the boundary status of its character, so both cells of an ideographic space end a word while a path with Hangul in it stays one word. All 328 terminal tests pass.

@akurtakov

Copy link
Copy Markdown
Member

As each of the intermediate commits have issues identified, please squash them in one and for future changes amend.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed for subtle interactions where wide-character padding truncates path selection, partial-glyph selection introduces artificial blanks, and zero-width input prematurely triggers wrapping or scrolling.

1 open issue
2 resolved since last review

Review effort: Balanced

@akurtakov

Copy link
Copy Markdown
Member

One more find. It's not a regression as the old code was far worse but still it would be nice to have it fixed here too.
A selection that starts on the second cell of a wide character isn't moved back to its first cell, as hover, the cursor and repaints are. extractSelectedText() cuts the line at the start column before scrubLine() runs, so the filler has no wide character in front of it and is copied as a space. Dragging from the right half of 한 in 한글 copies 글. The highlight in TextLineRenderer.drawLine() has the same problem and paints over the right half of the 한 glyph.
The highlight problem is visible in this screenshot
image

@insjang

insjang commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, fixed in 9b32b3e: a selection that starts on the second cell of a wide character now starts on its first, and one that ends on the first cell ends on the second, so dragging from the right half of 한 in 한글 copies 한글 and the highlight covers the whole glyph. A new test in TextCanvasModelTest covers it. The commits are squashed into one, and further changes will be amended into it.

The emulator assumed every character occupies one cell, so Hangul, Han
and Kana text, fullwidth forms and emoji were placed one column short
per character and the screen fell apart as soon as a program laid text
out for a real terminal (line editors, curses UIs, Ink based CLIs).

Add CharWidth, a UAX eclipse-platform#11 East Asian Width lookup: Wide and Fullwidth
count as two columns, combining marks and controls as zero, Ambiguous as
one, as UAX eclipse-platform#11 recommends outside an East Asian legacy context.

The emulator advances the cursor by that width and stores a NUL filler
in the second cell of a wide character, never splits one across the
right margin and erases the margin cell it leaves unused. Overwriting,
erasing, inserting or deleting either half of a wide character blanks
the whole character, and a count of zero edits nothing. Insert mode
makes room in cells, run by run, on the line the text is written to. A
character beyond the BMP takes two cells and leaves no filler in a
one-column terminal.

The renderer skips the fillers so a fixed width font draws a wide glyph
over both cells, falls back to placing each character at its own cell
when the font does not advance exactly one cell per column, draws a
character beyond the BMP whole and draws the block cursor over both
cells. A partial repaint that starts on the second cell of a wide
character is widened to its first, and copying drops the fillers. A
selection that starts or ends on half of a wide character covers the
whole character, both when copied and when highlighted, and the filler
takes the word boundary status of its character when selecting the word
under the mouse.

Tests cover the width table, placement, the margin, overwriting and
erasing halves, insert and delete, zero counts, the cursor, selection
and word selection.
@akurtakov

Copy link
Copy Markdown
Member

This one is already significant improvement! Thanks for that! I'm merging it in. Please rebase whichever PR is next on top of master with details how to test.
P.S. I might not be fastest to review it (others are welcome to beat me on that !) but I'll get to it .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants