ckVision Text Width and Grapheme Segmentation

ckv::text is ckVision's deterministic text-layout policy. It implements extended grapheme clusters from the Unicode Standard Annex #29 rules and a fixed terminal-column policy derived from Unicode 15.1.0 character data. Terminal-specific rendering agreement remains a term-layer concern under the decision log D-019; this module does not read the locale, environment, or terminal state.

Pinned Unicode data

The generated header include/cvision/core/generated_unicode_15_1.hpp and the conformance include tests/generated_unicode_grapheme_break_15_1.inc are generated by tools/unicode/generate_unicode_tables.py. They are checked in so every supported platform builds against identical, reviewable data. They are not edited by hand.

The release is Unicode 15.1.0. The generator accepts only these published Unicode Character Database files:

The source URLs are rooted at https://www.unicode.org/Public/15.1.0/ucd/. The named files, their hashes, and the generator are the complete provenance chain for generated data.

To regenerate after intentionally changing the pinned Unicode release, fetch those five files into a fresh directory (flattened to the filenames in the table), verify their hashes, then run:

python3 tools/unicode/generate_unicode_tables.py \
  --input-dir /path/to/unicode-15.1.0 \
  --tables include/cvision/core/generated_unicode_15_1.hpp \
  --grapheme-tests tests/generated_unicode_grapheme_break_15_1.inc

Review the generated diff, update this document's version, URLs and hashes, and run the conformance test and full required gates. Changing the Unicode release is a normative behavior change and requires a decision log record.

Grapheme segmentation

grapheme_end implements UAX #29 extended-grapheme rules GB3–GB13, GB9c and GB999 using the complete 15.1 property assignments needed by those rules: CR, LF, Control, Extend, ZWJ, Regional_Indicator, Prepend, SpacingMark, Hangul L/V/T/LV/LVT, Extended_Pictographic, and the Indic conjunct-break Consonant, Linker, and Extend properties. split_graphemes is its allocation-convenience wrapper; display and editing code normally iterates with grapheme_end.

tests/test_unicode_grapheme_break.cpp executes all 1,187 cases from the pinned official GraphemeBreakTest.txt. Project tests additionally exercise Arabic, Hebrew, South and Southeast Asian marks, rare East-Asian-wide scripts, emoji/ZWJ/variation selectors, malformed UTF-8, and the library's width policy.

Malformed UTF-8 is decoded one invalid byte at a time as U+FFFD for purposes of segmentation and width. This provides deterministic forward progress and matches utf8::decode; callers that are crossing into a rendered Cell use sanitize_display_text to materialize replacement characters.

Callers that must reject malformed data before display processing can use utf8::is_valid. It accepts only well-formed encodings of Unicode scalar values and distinguishes malformed input from a legitimate encoded U+FFFD replacement character.

Column width

codepoint_width returns zero for grapheme-control code points (Control, CR, LF, Extend, and ZWJ). All other scalar values are two columns only when their Unicode 15.1 East_Asian_Width property is W or F; they are one column otherwise. In particular, A (East-Asian Ambiguous) is deliberately one column by default, as specified by D-019.

grapheme_width applies the cluster-level policy needed for stable layout:

• a cluster containing VS16 is two columns and one containing VS15 is one;
• a leading regional-indicator pair (a flag) is two columns;
• an extended-pictographic ZWJ sequence is two columns; and
• a cluster that begins with an emoji modifier (U+1F3FB..U+1F3FF) takes the modifier's own East_Asian_Width, two columns, although the modifier is an Extend code point that adds nothing after a base (D-084).

Every other cluster is as wide as its first code point.

These are ckVision's deterministic layout choices, not an assertion that every terminal will render every sequence identically. A future terminal capability may adapt presentation at the term boundary without changing this default library policy.

Known terminal-divergent cases

Terminals disagree with this policy, and with each other, on the classes below (the decision log D-019). Each row is one case of the dedicated suite tests/test_text_width_divergent.cpp, which pins the width ckVision chooses. ctest -L d019 runs exactly that suite, and cvision_tests --suite test_text_width_divergent.cpp --list names its cases. The doc_widget_coverage gate fails when this table and the suite name different cases.

Safe clipping and elision

clip_to_width(text, columns) returns the greatest complete-grapheme prefix that fits the supplied terminal-cell budget. elide_to_width returns the text unchanged when it fits, otherwise reserves cells for a complete marker (U+2026 by default) and appends it after a complete-grapheme prefix. A non-positive budget returns empty text; a marker too wide for the budget is itself safely clipped.

These are the only text-truncation operations for widget display. Button, Window title, Menu, StatusLine, ListView, Table and Memo rendering use them or retain text as explicit grapheme elements. Painter clipping remains the final geometry guard, but it is not a substitute for deciding truncation in cells: a list row or table cell stops short of the scrollbar column it shares with the bar, because the bar paints over that column after the text and would leave half of a wide glyph behind. Inputs, labels, trees, help, static text and text views do not truncate by byte: they either maintain grapheme-indexed content, wrap by grapheme/cell, or pass full text to Painter's grapheme-aware cell clip.

tests/test_clip_sweep_golden.cpp holds every text-drawing widget family to this. Label, StaticText, Button, InputLine, ListView, Table, a DropdownMenu item, a StatusLine item, TextView and a TabControl caption each draw one string of ASCII, a combining sequence, a CJK ideograph, an emoji ZWJ sequence and a flag at every width from 0 until all of it shows. The frames are pinned as tests/golden/clip_sweep_<family>.dump, one file per family with its widths in order, and regenerated on every host by generated_golden_bytes. The test checks every cell of every width: no cell holds part of a cluster, no wide glyph keeps its lead without its continuation or the reverse, and nothing is drawn outside the widget.

Sanitization

sanitize_display_text replaces C0 and C1 controls, DEL, and invalid UTF-8 with U+FFFD before application text becomes Cell content. It also replaces tab and newline: a Cell is single-line, so multi-line widgets split lines before this boundary. This is the text-to-Cell neutralization required by the architecture §12. sanitize_clipboard_text keeps tab and line breaks, which are document content on a clipboard. sanitize_osc_text drops controls rather than marking them (malformed UTF-8 still becomes U+FFFD), for text that goes inside an OSC command string such as a window title or a hyperlink target; see OSC emission safety.

tests/golden/hostile_display_text.dump pins the result for hostile strings painted through Painter::draw_text: an erase-display CSI, an OSC 52 write, U+009B, BEL, DEL, NUL, tab and newline, raw 8-bit CSI and ST bytes, and malformed UTF-8. Each control and each malformed byte becomes one U+FFFD cell in place, and the rest of each sequence stays as inert text. The same suite presents that frame and checks the Presenter's raw bytes: every control in the stream is an ESC introducing the Presenter's own CSI, no OSC is opened, and the stream is well-formed UTF-8 with no C1 code point.

The library never consults the process locale. The <cctype> functions classify and fold case by the locale a host program may have set, and under a single-byte locale std::tolower rewrites UTF-8 lead bytes. Mnemonic matching, type-ahead, filters and the other ASCII comparisons use cvision/core/ascii.hpp instead, which touches only A–Z, a–z and 0–9. The layer_check gate rejects <cctype> and the other locale, clock and stream headers in core, scene, ui and widgets.

clip_to_width_view(text, columns) applies the same complete-grapheme boundary as clip_to_width and borrows the input without allocating. Keep the input alive while using the returned view. is_sanitized_display_text(text) validates that sanitization would preserve every byte, including valid U+FFFD; malformed UTF-8, C0/C1 controls and DEL fail validation. Painter borrows validated display text and sanitizes other input, preserving D-040 in both paths.