Terminal: grapheme clusters, with the cell layout kept by the text data by insjang · Pull Request #2904 · eclipse-platform/eclipse.platform

insjang · 2026-09-03T01:10:24Z

Depends on #2894 (East Asian Width): builds on CharWidth and the two-cell placement of wide characters. This PR shows that commit as well until #2894 is merged; only the second commit is new here.

A combining mark has no cell of its own. It was dropped, which lost data: an accent typed as a separate mark, a Hangul syllable sent as conjoining jamo, a Japanese voicing mark. A mark now composes with the character before it (NFC) whenever the two have a single form, which covers all of those.

What has no composed form is a grapheme cluster: an emoji joined to others with a zero-width joiner, one with a skin tone, a character with a presentation selector, a keycap, a flag made of two regional indicators. Windows Terminal (1.22+) and the libraries programs use to lay text out give such a cluster two cells however many characters it runs to, so the emulator has to count the same way or the columns drift.

The cells keep the cluster's first character, which is what gets drawn; the whole cluster is kept beside the line and copied out whole, so what the user copies is what the program wrote.
Writing either cell forgets the cluster; it moves with its line when lines scroll or are copied, and through snapshots.
A narrow character asked to be shown as an emoji (VS16, as in ❤️) takes a second cell. Regional indicator pairs join into one flag.
Marks that compose still compose first, so Hangul and accented Latin are unchanged.

API: ITerminalTextDataReadOnly.getCluster / ITerminalTextData.setCluster (default methods, @since 1.2, bundle version 1.2.0).

Tests: combining marks and clusters in VT100EmulatorBackendTest; cluster bookkeeping (overwrite, scroll, clean, copy, window) in AbstractITerminalTextDataTest for every data implementation.

github-actions · 2026-11-04T04:58:53Z

Test Results

54 files ± 0 54 suites ±0 40m 23s ⏱️ + 6m 4s
5 009 tests +24 4 987 ✅ +24 22 💤 ±0 0 ❌ ±0
12 900 runs +54 12 746 ✅ +54 154 💤 ±0 0 ❌ ±0

Results for commit 0f0b239. ± Comparison against base commit 63d22c8.

This pull request removes 2 and adds 26 tests. Note that renamed tests count towards both.

org.eclipse.terminal.internal.model.CharWidthTest ‑ testIsFiller
org.eclipse.terminal.internal.model.CharWidthTest ‑ testIsSecondCell

org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testClusterBoundaries
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testClusterEdgeCases
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testClusterEdits
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testClusterGrowingWideScrollsWithIt
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testClusterKeptPastTheMarginAndEndedByAMove
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testClustersAcrossWritesAndWraps
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testCombiningMarks
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testGraphemeClusters
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testInsertKeepsAClusterNextToTheMargin
org.eclipse.terminal.internal.emulator.VT100EmulatorBackendTest ‑ testNothingHiddenAndNoEndlessCluster
…

♻️ This comment has been updated with latest results.

insjang · 2026-11-04T05:32:59Z

Rebased on master, now that #2894 is in: one commit, 30d40d1, the earlier commits squashed; all terminal tests pass. The writing loop combines this PR's handling of marks and clusters with #2894's final placement (room made run by run in insert mode, the margin cell erased on an early wrap); a mark or cluster part that goes onto the character before it makes no room.

How to test, in a runtime Eclipse with this org.eclipse.terminal.control, Terminal view, local shell:

A combining mark composes: printf 'e\xcc\x81 | \xe1\x84\x92\xe1\x85\xa1\xe1\x86\xab\n' (e + U+0301, and 한 as conjoining jamo)
Expected: é and 한, in one and two cells, and the | right after é. Copying gives the composed characters. Before this change the marks were dropped.
Clusters take two cells: printf '👍🏽|👨‍👩‍👧|🇰🇷|1️⃣|❤️|\n'
Expected: each is drawn in two cells, the bars line up below a line of ab|ab|ab|ab|ab|, and copying the line gives back exactly what was printed.
Overwriting a cluster forgets it: printf '👨‍👩‍👧\rxy\n' then copy the line: it gives xy.
Scrolling and history: print a line with clusters, seq 1 100, scroll back and copy it: the clusters are intact.
Hangul and accented Latin written as precomposed text are unchanged: echo 한글 café.

Unit tests: testCombiningMarks and testGraphemeClusters in VT100EmulatorBackendTest; the cluster bookkeeping (overwrite, scroll, clean, copy, window) in AbstractITerminalTextDataTest for every data implementation.

Copilot

🟡 Changes recommended

Unresolved grapheme-cluster defects can lose or corrupt text during composition, editing and copying, while also mishandling cursor adjacency and right-margin wrapping.

9 open issues

Narrow-base combining marks and keycaps are dropped · New
Cursor movement leaves stale ZWJ join state · New
VS16 promotion overwrites text in insert mode · New
Promoted heart fillers are missed by wide-character checks · New
VS16 promotion at the margin clears pending wrap · New
Combining marks discard earlier retained marks · New
Character shifts discard cluster metadata · New
Self-copy clears cluster metadata before reading it · New
Selections omit promoted heart clusters · New

What changed in this PR

Extends the terminal’s two-cell character support from #2894 with combining-mark composition and grapheme-cluster storage, aiming to preserve column alignment and copied text.

Changes:

Adds NFC composition and cluster handling for joined emoji, modifiers, presentation selectors and flags.
Carries cluster metadata through terminal data implementations, snapshots and selection copying.
Adds emulator and model tests, plus public cluster APIs and a bundle version bump.

File	Description
terminal/tests/org.eclipse.terminal.test/src/org/eclipse/terminal/internal/model/TerminalTextDataWindowTest.java	Tests window-relative cluster access and invalidation.
terminal/tests/org.eclipse.terminal.test/src/org/eclipse/terminal/internal/model/AbstractITerminalTextDataTest.java	Tests cluster overwrite, scrolling, cleaning and copying.
terminal/tests/org.eclipse.terminal.test/src/org/eclipse/terminal/internal/emulator/VT100EmulatorBackendTest.java	Tests composition, emoji clusters and deferred wrapping.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/model/ITerminalTextDataReadOnly.java	Adds the default cluster-read API.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/model/ITerminalTextData.java	Adds the default cluster-write API.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/internal/textcanvas/AbstractTextCanvasModel.java	Includes stored clusters in copied selections.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/internal/model/TerminalTextDataWindow.java	Translates cluster access into window coordinates.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/internal/model/TerminalTextDataStore.java	Stores, invalidates and transfers cluster metadata.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/internal/model/TerminalTextDataSnapshot.java	Exposes snapshot cluster data.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/internal/model/TerminalTextDataFastScroll.java	Maps cluster access through scrolling offsets.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/internal/model/TerminalTextData.java	Delegates cluster access and notifies snapshots.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/internal/model/SynchronizedTerminalTextData.java	Synchronizes cluster reads and writes.
terminal/bundles/org.eclipse.terminal.control/src/org/eclipse/terminal/internal/emulator/VT100EmulatorBackend.java	Implements composition, joining and two-cell promotion.
terminal/bundles/org.eclipse.terminal.control/META-INF/MANIFEST.MF	Increments bundle and model-package versions for the APIs.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Active defects cause grapheme data loss across input boundaries and wrapping, corrupt cluster placement during scrolling, and break hover reconstruction after resizing or across empty cells.

6 open issues

8 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Unresolved combining-cluster bugs can corrupt cell contents, trigger a cursor-rendering exception in one-column terminals, and prevent composition with characters moved into scrollback.

5 open issues

5 resolved since last review

_{Review effort: Balanced}

insjang · 2026-11-05T11:40:19Z

I owe you an apology for the last few rounds on this PR. Each time Copilot found an edge case I patched that one place instead of asking why it was possible. The cause is in the structure: whether a cell is the second half of a character is decided in about ten places (emulator, text store, selection, hover, copy, canvas, renderer, cursor), each with its own conditions, and every patch added another. That is why every review turned up new holes, and it cost you review time and notifications. I'm sorry.

I'm rewriting the cluster support instead of patching it further:

One class answers every question about the cells of a line (how many cells a character takes, where it starts, what text it holds), and every reader and writer goes through it; the scattered checks are removed.
One method writes a character into its cells, so fillers and cluster data cannot get out of step.
The emulator keeps a single record of the last cluster it wrote, instead of reading state back from the cells.
Grapheme cluster widths follow Windows Terminal, including clusters made only of zero width code points, so nothing the program writes is dropped.
A randomized test mixes writes (split at every point), cursor moves, erases, insertions, deletions, scrolling and resizing, and checks the cells against that one class after every step, on top of the Unicode conformance data (GraphemeBreakTest, emoji-test).

I'll push it as one commit when it passes all of that, and not before. Until then there is no need to look at this PR or request another Copilot review; the open Copilot comments will be answered by the rewrite.

insjang · 2026-11-05T12:36:02Z

The rewrite is pushed as one commit, eb42d69. How many cells a character takes is now kept by the text data when the character is written (TerminalTextDataStore, read through getCellWidth/getCluster/isSimpleLine), and everything that reads cells goes through it; the scattered checks are gone. Cluster boundaries follow UAX #29 (BreakIterator), widths follow Windows Terminal, and nothing the program writes is dropped. Besides the unit tests there is a randomized test that mixes writes, cursor moves, erases, insert/delete, scrolling, resizing and the alternate screen and checks the cells, selection, copy and drawn text after every step. Writing and drawing CJK text is about 1.8 times as fast as on master; ASCII is written up to about 12 percent slower. I have answered the open Copilot comments; each is covered by the rewrite.

Copilot

🟡 Changes recommended

Unresolved cluster-handling defects discard retained off-screen layout during line copying, reuse stale continuation state after cursor movement, and cause hover selection to cross visible gaps.

3 open issues

5 resolved since last review

_{Review effort: Balanced}

+		// read before writing: the source may be this very line
+		Layout layout = null;
+		boolean simple = source.isSimpleLine(sourceLine);
+		for (int col = 0; !simple && chars != null && col < chars.length && col < source.getWidth(); col++) {


+		if (fLastText == null || fCursorLine != fAfterLine || fCursorColumn != fAfterColumn
+				|| fWrapPending != fAfterPending || fLastColumn >= fColumns) {
+			return -1;


+			return true;
 		}
-		return scrubbed.toString();
+		char c = text.charAt(0);


A combining mark has no cell of its own. It was dropped, which lost data: an accent typed as a separate mark, a Hangul syllable sent as conjoining jamo, a Japanese voicing mark. More than marks go with the character before them: a zero width joiner and what it joins, a skin tone, a presentation selector, the second regional indicator of a flag. What a reader takes for one character is a grapheme cluster, and where one ends is now decided by the Unicode rules (UAX eclipse-platform#29) through java.text.BreakIterator, over the text as it is written, the cluster written last going first, so a cluster goes on across writes but not across a cursor move. A cluster takes the cells its code points would, at least one and at most two, summed as Windows Terminal sums them: a mark takes none, VS16 makes it wide. Grown wide on the last cell it moves whole to the next line, as a wide character does; in insert mode its second cell is made room for. A cluster that starts with a code point taking no cell goes onto the one before it, or takes a cell of its own when there is none. Left out, before the clusters are found, are control characters, the format characters no cluster needs (a zero width space, bidirectional marks, overrides and isolates), and what only hides text: tag characters outside the tag sequence of a flag (UTS eclipse-platform#51), a variation selector that does not follow a character, a second prefix sign. None is drawn, and in copied text they could make a command read differently from how it runs or carry text no one saw. A cluster keeps at most 32 chars (the longest emoji sequence is 15), so a flood of marks cannot grow one cell without bound. How many cells each character takes was worked out again, from the chars around it, wherever it was needed: in the emulator, the selection, copying, the word under the pointer, the canvas and the renderer, each with its own conditions. The text data now keeps it. TerminalTextDataStore records, per line, the cells that are the second of a character (written with it in one call: a wide character and its filler, a surrogate pair, or a cluster made wide) and the clusters beside them; lines of one-cell characters keep nothing extra. Reading it is a lookup (getCellWidth, getCluster, isSimpleLine, new default methods of ITerminalTextDataReadOnly, hence the version bump), and everything that reads cells goes through it and through Cells, which puts characters back together; the old inference in CharWidth is gone. The emulator writes every character through one method, moves characters whole when cells are inserted or deleted, and keeps one record of the cluster written last. The cells hold what is drawn, the character composed with its marks (NFC) where it has such a form; the cluster is kept as the program wrote it, and it is what is drawn over its cells (a flag, a family, a keycap as one) and what selecting and copying give. Text of one-cell characters is drawn as before, from the line's chars. Runs of wide characters are written in one go, the boundary rules are consulted only for characters they could join (found once per character from BreakIterator itself), and character widths of the BMP are looked up. Writing CJK text is about 1.8 times as fast as before, and drawing it faster too; other text is drawn at least as fast and written up to about 12 percent slower, the cost of finding clusters. Tests: CellsTest, and a randomized test that mixes writes (split at any point), cursor moves, erases, insertions, deletions, scrolling, resizing and the alternate screen, checking after every step that the cells, the selection, the copy and the drawn text agree, and that written text is copied back unchanged.

insjang force-pushed the grapheme-clusters branch from 66aa868 to 4a0de20 Compare September 28, 2026 07:43

insjang mentioned this pull request Sep 28, 2026

Terminal: count East Asian Wide characters as two columns #2894

Merged

insjang force-pushed the grapheme-clusters branch from 4a0de20 to 2676b31 Compare September 30, 2026 08:06

insjang mentioned this pull request Oct 11, 2026

Terminal: DEC private mode lists, DECTCEM and bracketed paste #2901

Open

insjang force-pushed the grapheme-clusters branch from 2676b31 to 30d40d1 Compare November 4, 2026 04:33

insjang force-pushed the grapheme-clusters branch 3 times, most recently from 9f6edf5 to 7a7a5cd Compare November 5, 2026 01:35

insjang mentioned this pull request Nov 5, 2026

Terminal: follow the VT100 and ECMA-48 parser rules, and support the VT100 commands vttest checks #3059

Open

insjang force-pushed the grapheme-clusters branch from 7a7a5cd to 93754ea Compare November 5, 2026 07:18

akurtakov requested a balanced review from Copilot November 5, 2026 09:00

Copilot started reviewing on behalf of akurtakov November 5, 2026 09:00

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

insjang force-pushed the grapheme-clusters branch from 93754ea to 304dd72 Compare November 5, 2026 10:31

akurtakov requested a balanced review from Copilot November 5, 2026 10:33

Copilot started reviewing on behalf of akurtakov November 5, 2026 10:33

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

insjang force-pushed the grapheme-clusters branch from 304dd72 to c3aef36 Compare November 5, 2026 10:49

akurtakov requested a balanced review from Copilot November 5, 2026 11:15

Copilot started reviewing on behalf of akurtakov November 5, 2026 11:15

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

insjang force-pushed the grapheme-clusters branch from c3aef36 to 6240c74 Compare November 5, 2026 12:34

insjang force-pushed the grapheme-clusters branch 2 times, most recently from f2e115d to 4e8c737 Compare November 5, 2026 12:40

insjang changed the title ~~Terminal: compose combining marks, and keep a grapheme cluster in two cells~~ Terminal: grapheme clusters, with the cell layout kept by the text data Nov 5, 2026

insjang force-pushed the grapheme-clusters branch from 4e8c737 to eb42d69 Compare November 5, 2026 12:44

akurtakov requested a balanced review from Copilot November 5, 2026 13:12

Copilot started reviewing on behalf of akurtakov November 5, 2026 13:12

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

insjang force-pushed the grapheme-clusters branch from eb42d69 to 0f0b239 Compare November 5, 2026 15:24

Conversation

insjang commented Sep 3, 2026

Uh oh!

github-actions Bot commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

insjang commented Nov 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

insjang commented Nov 5, 2026

Uh oh!

insjang commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Nov 4, 2026 •

edited

Loading

insjang commented Nov 5, 2026 •

edited

Loading