Strings and encodings in Qt
En Ar Bg De El Es Fa Fi Fr Hi Hu It Ja Kn Ko Ms Nl Pl Pt Ru Th Tr Uk Zh
This is a short introduction to encodings, and how not to fail at using them inside Qt (and in general inside a C/C++ project).
The contents of this page need updating to Qt 5. Qt 5 makes more use of UTF-8. In doubt, always consult the manual.
The short recipe
- There Ain't No Such Thing As Plain Text.
- Encode your source code files using UTF-8. Set all your tools (editors, etc.) to read and write UTF-8 files. Do it once, for all your code base, and make your colleagues stick with it.
- Make your compiler read UTF-8 files. Changing your locale to an UTF-8 locale should be enough (otherwise, see your compiler's manual).
- In your code use QStringLiteral / QString::fromUtf8() for all string literals, and tr() (in Qt 4: trUtf8() / QT_TR_UTF8() / etc.) for all user-visible strings.
- If you really want to use the QString(const char *) constructor, in Qt 4, remember to call QTextCodec::setCodecForCStrings() passing the UTF-8 text codec (QTextCodec::codecForName("UTF-8")). Being an application wide setting, this might break plugins or 3rd party Qt libraries. Note that in Qt 5 this constructor expects a UTF-8 string already.
The long story
Encodings in C/C++
What is the encoding of string literals in C/C+?
The C and C+ specifications don't say much about the encoding of string literals inside a program — or about the encoding of a program's source code itself. They just specify that an input character set and an execution character set do exist, with some simple requirements.
A compiler has to employ several techniques to deal with this problem. For starters, it must properly decode the bytes of the source file into a C/C++ program (so that the parser can parse the program). This is usually done using the current locale settings, or falling back to UTF-8 (it's ASCII compatible); if you are using GCC you can pass the -finput-charset= command line option to specify a certain encoding for the input files.
Once a source file is parsed, the compiler keeps the string literals in an internal, private representation, which is not really interesting — as compiler users, it suffices for us that this internal representation is able to represent any possible symbol in any alphabet. This usually means that this internal representation is actually an Unicode encoding.
In the end, the compiler has to emit some assembly code for the strings, that is, sequences of raw bytes; therefore, it has to encode the strings using an execution encoding. Again, if you're using GCC you can pass the -fexec-charset= and -fwide-exec-charset= command line options to control which encoding is used for the char literals and the wchar_t literals respectively (by default they're UTF-8 and UTF-16 or UTF-32, depending on the size of wchar_t).
What's the scope of such command line options?
The invocation of the compiler in which they appear, so one translation unit — yes, you can change the input and execution charsets for every translation unit, if you want.
Wait a minute… so what does o exactly?
The escape sequence inside a string literal encodes the code point U+XXXX using the execution character set (similarly, the encodes the code point U+XXXXXXXX). Note that s, in general, a prefix for an universal character, and you're allowed to use it even outside string literals.
How does Qt know the encoding of a byte array (a char *), for instance, when I build a QString from it?
It's simple: Qt doesn't know it. For instance:
- the QString(const char *) constructor in Qt 5 expects UTF-8 and UTF-8 only. In Qt 4, it uses the QTextCodec::codecForCStrings() if one was set, or falls back to Latin-1 — so if you pass data in any other encoding, including UTF-8, it may horribly break;
- the static QString::fromLatin1() method builds a string from Latin-1 encoded data;
- the static QString::fromUtf8() method builds a string from UTF-8 encoded data;
- the tr() method for translation expects UTF-8 in Qt 5 (in Qt 4 the QTextCodec::codecForTr() if one was set, or, again, falls back to L