std/unicode

Unicode beyond the bytes of a str: code points and grapheme clusters, the properties of characters, case conversion and case folding, normalization, word and line boundaries, the width text takes in a terminal, and offsets in UTF-16, as the language server protocol counts them. A str stays bytes, indexed by bytes; these functions read it as UTF-8, with U+FFFD for a byte that does not start a valid character, and the offsets they take and give are byte offsets unless the name says otherwise.

import "std/unicode"

print(graphemes("e\u{301}👍🏽!"))            // three: "é", "👍🏽", "!"
print(upper_case("straße"))                 // STRASSE
print(normalize("e\u{301}", "NFC") == "é")  // true
print(display_width("日本"))                // 4

Function	Result
`code_points(text)`, `from_code_points(codes)`	the code points of text, as an `int[]`, and back
`code_point_at(text, offset)`	the code point at a byte offset and how many bytes it takes
`code_point_text(code)`, `code_point_count(text)`	the text of one code point, and how many a text has
`graphemes(text)`, `grapheme_count(text)`	the grapheme clusters: what a reader takes as one character, as a letter with its accents or an emoji with its modifiers
`grapheme_boundaries(text)`, `next_grapheme(text, offset)`, `previous_grapheme(text, offset)`	where the clusters start, and the next and previous one, as a cursor moves
`words(text)`	the words: the pieces between word boundaries that hold a letter or a digit
`word_boundaries(text)`, `next_word_boundary(text, offset)`, `previous_word_boundary(text, offset)`	the word boundaries, as an editor moves by words
`line_breaks(text)`	where lines may break, as `LineBreak` items with an `offset` and whether the break is `mandatory`
`general_category(code)`, `script(code)`, `combining_class(code)`	`"Lu"`, `"Latin"`, and the canonical combining class
`is_letter`, `is_mark`, `is_number`, `is_digit`, `is_punctuation`, `is_symbol`, `is_upper`, `is_lower`, `is_whitespace`, `is_pictographic`	properties of a code point
`upper_case(text)`, `lower_case(text)`, `title_case(text)`	text in another case, with the mappings of more than one character, as `ß` to `SS`
`fold_case(text)`, `equal_ignoring_case(a, b)`	case folding, for comparing text whatever its case
`normalize(text, form)`, `is_normalized(text, form)`	the forms `"NFC"`, `"NFD"`, `"NFKC"`, and `"NFKD"`
`code_point_width(code)`, `display_width(text)`	the columns a terminal gives a character and a text
`utf16_length(text)`, `utf16_offset(text, offset)`, `byte_offset_from_utf16(text, offset)`	UTF-16 lengths and offsets
`code_point_offset(text, offset)`, `byte_offset_from_code_point(text, index)`	code point offsets

The boundaries follow UAX 29 for grapheme clusters and words and UAX 14 for lines, without tailoring; normalization follows UAX 15. A capital sigma at the end of a word is lowercase ς; other mappings that depend on the language, as Turkish dotted and dotless i, are not made. display_width counts a grapheme cluster at a time: wide characters, such as ideographs and most emoji, take 2 columns, marks and characters that are not shown take none, an emoji asked for with U+FE0F takes 2, and one asked for as text with U+FE0E takes 1.

The tables come from the Unicode Character Database of the version UNICODE_VERSION gives, 17.0.0, through tools/unicode.cz, and are read the first time a function needs them.