UTF-16
(Try to clarify BOM usage.) |
(→Format) |
||
| Line 12: | Line 12: | ||
For higher code points, first subtract 0x10000 to get a 20-bit number. Add the 10 most significant bits of that number to 0xd800, and use that as the first code unit. Add the 10 least significant bits to 0xdc00, and use that as the second code unit. | For higher code points, first subtract 0x10000 to get a 20-bit number. Add the 10 most significant bits of that number to 0xd800, and use that as the first code unit. Add the 10 least significant bits to 0xdc00, and use that as the second code unit. | ||
| + | |||
| + | Descriptions of UTF-16 often use the term ''surrogate pair''. This tends to create unnecessary confusion, perhaps because whether a ''surrogate'' is a ''codepoint'' or not depends on context in subtle ways. See [[UCS-2#Surrogate pairs|UCS-2]] for the historical reasons that this term exists. | ||
== Byte-oriented encodings == | == Byte-oriented encodings == | ||
Revision as of 01:27, 11 September 2013
UCS Transformation Format—16-bit (UTF-16) is a 16-bit Unicode character encoding. It encodes a sequence of Unicode code points in a sequence of unsigned 16-bit integers (which we'll call code units).
Code points up to U+FFFF use one code unit, while higher code points use two. UTF-16 is not capable of encoding code points in the reserved range U+D800 to U+DFFF, but that range is not required for full Unicode support.
Contents |
Format
For code points up to U+FFFF, the encoding is done in the trivial way (U+0000=0x0000, U+0001=0x0001, ..., U+FFFF=0xffff).
For higher code points, first subtract 0x10000 to get a 20-bit number. Add the 10 most significant bits of that number to 0xd800, and use that as the first code unit. Add the 10 least significant bits to 0xdc00, and use that as the second code unit.
Descriptions of UTF-16 often use the term surrogate pair. This tends to create unnecessary confusion, perhaps because whether a surrogate is a codepoint or not depends on context in subtle ways. See UCS-2 for the historical reasons that this term exists.
Byte-oriented encodings
There are two byte-oriented flavors of UTF-16: UTF-16BE (big-endian) and UTF-16LE (little-endian). Each code unit uses 2 bytes, so each code point is encoded using either 2 or 4 bytes.
The code units are encoded according to the desired byte order, and the order of the code units is not changed. (This means that for UTF-16LE, 4-byte code points use sort of a mixed-endian byte order.)
Files encoded in UTF-16BE or UTF-16LE often begin with a Byte Order Mark (BOM) to distinguish them from each other, and to help distinguish them from other character encodings. However, the Unicode standard says[1] that a file with a BOM should not be labeled with a charset of "UTF-16BE" or "UTF-16LE". Instead, the charset name should be simply "UTF-16".
Relationship with UCS-2
UTF-16 is compatible with UCS-2, in the sense that a UTF-16 decoder can correctly decode UCS-2, provided that the UCS-2 data does not use any reserved Unicode code points. The reverse is not true: a UCS-2 decoder cannot correctly decode UTF-16.
Specifications
- RFC 2781 (2000-02)
- Unicode 6.0, Chapter 3 (2011) — §3.9 D91; §3.10 D96–D98, Table 3-9
- ISO/IEC 10646:2003 Annex Q (2003)
