std.utf
Encode and decode UTF-8, UTF-16 and UTF-32 strings. For Win32 systems, the C wchar_t type is UTF-16 and corresponds to the D wchar type. For linux systems, the C wchar_t type is UTF-32 and corresponds to the D utf.dchar type. UTF character support is restricted to (\u0000 <= character <= \U0010FFFF). See Also:Wikipedia
http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
http://anubis.dkuug.dk/JTC1/SC2/WG2/docs/n1335 License:
Boost License 1.0. Authors:
Walter Bright Source:
std/utf.d
- Exception class that is thrown upon any errors.
- Test if c is a valid UTF-32 character.
\uFFFE and \uFFFF are considered valid by this function,
as they are permitted for internal use by an application,
but they are not allowed for interchange by the Unicode standard.
Returns:
true if it is, false if not. - stride() returns the length of a UTF-8 sequence starting at index i
in string s.
Returns:
The number of bytes in the UTF-8 sequence. Throws:
UtfException if s[i] is not the start of the UTF-8 sequence. - strideBack() returns the length of a UTF-8 sequence ending before index i
in string s.
Returns:
The number of bytes in the UTF-8 sequence. Throws:
UtfException if s[i-1] is not the end of the UTF-8 sequence. - stride() returns the length of a UTF-16 sequence starting at index i in string s.
- strideBack() returns the length of a UTF-16 sequence ending before index i in string s.
- stride() returns the length of a UTF-32 sequence starting at index i
in string s.
Returns:
The return value will always be 1. - strideBack() returns the length of a UTF-32 sequence ending before index i
in string s.
Returns:
The return value will always be 1. - Given an index i into an array of characters s[], and assuming that index i is at the start of a UTF character, determine the number of UCS characters up to that index i.
- Given a UCS index n into an array of characters s[], return the UTF index.
- Decodes and returns character starting at s[idx]. idx is advanced past the decoded character. If the character is not well formed, a UtfException is thrown and idx remains unchanged.
- Encodes character c into fixed-size array s. Returns the actual length of the encoded character (a number between 1 and 4 for char[4] buffers, and between 1 and 2 for wchar[2] buffers).
- Encodes character c and appends it to array s[].
- Returns the number of code units that are required to encode the code point
c when C is the character type used to encode it.
Examples:
assert(codeLength!char('a') == 1); assert(codeLength!wchar('a') == 1); assert(codeLength!dchar('a') == 1); assert(codeLength!char('\U0010FFFF') == 4); assert(codeLength!wchar('\U0010FFFF') == 2); assert(codeLength!dchar('\U0010FFFF') == 1);
- Checks to see if string is well formed or not. S can be an array of char, wchar, or dchar. Throws a UtfException if it is not. Use to check all untrusted input for correctness.
- Encodes string s into UTF-8 and returns the encoded string.
- Encodes string s into UTF-16 and returns the encoded string.
- Scheduled for deprecation in February 2012. Please use instead. Encodes string s into UTF-16 and returns the encoded string. toUTF16z is suitable for calling the 'W' functions in the Win32 API that take an LPWSTR or LPCWSTR argument.
- Encodes string s into UTF-32 and returns the encoded string.
- Returns a C-style zero-terminated string equivalent to str. str
must not contain embedded '\0''s as any C function will treat the first
'\0' that it sees a the end of the string. If str.empty is
true, then a string containing only '\0' is returned.
toUTFz accepts any type of string and is templated on the type of
character pointer that you wish to convert to. It will avoid allocating a
new string if it can, but there's a decent chance that it will end up having
to allocate a new string - particularly when dealing with character types
other than char.
Warning 1: If the result of toUTFz equals str.ptr, then if
anything alters the character one past the end of str (which is the
'\0' character terminating the string), then the string won't be
zero-terminated anymore. The most likely scenarios for that are if you
append to str and no reallocation takes place or when str is a
slice of a larger array, and you alter the character in the larger array
which is one character past the end of str. Another case where it could
occur would be if you had a mutable character array immediately after
str in memory (for example, if they're member variables in a
user-defined type with one declared right after the other) and that
character array happened to start with '\0'. Such scenarios will never
occur if you immediately use the zero-terminated string after calling
toUTFz and the C function using it doesn't keep a reference to it.
Also, they are unlikely to occur even if you save the zero-terminated string
(the cases above would be among the few examples of where it could happen).
However, if you save the zero-terminate string and want to be absolutely
certain that the string stays zero-terminated, then simply append a
'\0' to the string and use its ptr property rather than calling
toUTFz.
Warning 2: When passing a character pointer to a C function, and the
C function keeps it around for any reason, make sure that you keep a
reference to it in your D code. Otherwise, it may go away during a garbage
collection cycle and cause a nasty bug when the C code tries to use it.
Examples:
auto p1 = toUTFz!(char*)("hello world"); auto p2 = toUTFz!(const(char)*)("hello world"); auto p3 = toUTFz!(immutable(char)*)("hello world"); auto p4 = toUTFz!(char*)("hello world"d); auto p5 = toUTFz!(const(wchar)*)("hello world"); auto p6 = toUTFz!(immutable(dchar)*)("hello world"w);
- Returns the total number of code points encoded in a string.
The input to this function MUST be validly encoded.
Supercedes:
This function supercedes std.utf.toUCSindex(). Standards:
Unicode 5.0, ASCII, ISO-8859-1, WINDOWS-1252 Parameters:s the string to be counted