utf8

The complete utf8 library for Unicode text. Byte-oriented string functions (string) see multi-byte characters as bytes; these see codepoints.

Encoding

utf8.char(...)

Builds a string from codepoints.

print(utf8.char(112, 246, 122, 111))  --> pozo
print(utf8.char(0x2603))              --> ☃

utf8.charpattern

Not a function — the Lua pattern matching exactly one UTF-8 character: [\0-\x7F\xC2-\xFD][\x80-\xBF]*. Use it wherever a pattern needs “one character”.

for ch in string.gmatch("pozo", utf8.charpattern) do
    print(ch)  --> p / ö / z / o
end

Decoding

utf8.codepoint(s, i?, j?)

Codepoints of the characters in byte range i..j (default: first character).

print(utf8.codepoint("pozo"))         --> 246
print(utf8.codepoint("pozo", 1, 2))   --> 112  246

utf8.codes(s)

Iterator yielding byte-offset and codepoint pairs — the idiomatic walk over a string’s characters.

for pos, cp in utf8.codes("a☃b") do
    print(pos, cp)
end
--> 1  97
--> 2  9731
--> 5  98

Length and offsets

utf8.len(s, i?, j?)

Number of characters in byte range i..j. Returns nil + the position of the first invalid byte if the range isn’t valid UTF-8.

print(utf8.len("pozo"))   --> 4       (string.len gives 5)
print(utf8.len("a☃b"))    --> 3

utf8.offset(s, n, i?)

Byte offset of the n-th character starting at byte i (default 1). n == 0 gives the offset of the character containing i; negative n counts backward.

print(utf8.offset("a☃b", 2))   --> 2
print(utf8.offset("a☃b", 3))   --> 5
print(utf8.offset("a☃b", -1))  --> 5