 is parsed, the compiler keeps the string literals in an internal, private representation, which is not really interesting — as compiler users, it suffices for us that this internal representation is able to represent any possible symbol in any alphabet. This usually means that this internal representation is actually an Unicode encoding.
In the end, the compiler has to emit some assembly code for the strings, that is, sequences of raw bytes; therefore, it has to encode the strings using an execution encoding. Again, if you're using GCC you can pass the -fexec-charset= and -fwide-exec-charset= command line options to control which encoding is used for the char literals and the wchar_t literals respectively (by default they're UTF-8 and UTF-16 or UTF-32, depending on the size of wchar_t).
What's the scope of such command line options?
The invocation of the compiler in which they appear, so one translation unit — yes, you can change the input and execution charsets for every translation unit, if you want.
Wait a minute… so what does o exactly?
The escape sequence inside a string literal encodes the code point U+XXXX using the execution character set (similarly, the encodes the code point U+XXXXXXXX). Note that s, in general, a prefix for an universal character, and you're allowed to use it even outside string literals.
How does Qt know the encoding of a byte array (a char *), for instance, when I build a QString from it?
It's simple: Qt doesn't know it. For instance:
- the QString(const char *) constructor in Qt 5 expects UTF-8 and UTF-8 only. In Qt 4, it uses the QTextCodec::codecForCStrings() if one was set, or falls back to Latin-1 — so if you pass data in any other encoding, including UTF-8, it may horribly break;
- the static QString::fromLatin1() method builds a string from Latin-1 encoded data;
- the static QString::fromUtf8() method builds a string from UTF-8 encoded data;
- the tr() method for translation expects UTF-8 in Qt 5 (in Qt 4 the QTextCodec::codecForTr() if one was set, or, again, falls back to Latin-1);
- in Qt 4 the lupdate tool uses the CODECFORTR and CODECFORSRC in your .pro file to know the encoding of the strings inside tr() and of your source files;
- for any other encoding schemes you have to build a proper QTextCodec (by using QTextCodec::codecForName() or similar methods), and then use its toUnicode() method.
QString is a re-entrant, implicitly shared container for an UTF-16 encoded string. This means (in no particular order):
- All methods that take an index, as well as the iterators, work in terms of code units, not code points.
- A QString might contain invalid data (for instance a broken surrogate pair).
But what's wrong with the QString(const char *) constructor?
In Qt 4: that it decodes the bytes in the passed string using the fromAscii() function, which — despite its name — uses the codec returned by QTextCodec::codecForCStrings(), or Latin-1 if you didn't set one. Thus, if you pass UTF-8 data to it without doing anything else, QString is NOT going to decode your data properly.
Note also the codec set is application-wide — once you set it, you change the behaviour of fromAscii() in your code, in libraries you are using, in plugins you may have loaded, etc. If they're very poorly coded you may even break them by doing so (BT;DT).
So what should I use instead?
There are a number of available options:
- Switch all of your code editors to use UTF-8, convert all of your codebase (f.i. by using the iconv(1) tool), set your compiler input and execution charsets to UTF-8, then in your code use QString::fromUtf8(), trUtf8(), etc.; this way, will do the right thing™.
QString str = QStringLiteral("ßàéöø")
- If your source must be 7-bit ASCII, you can still make use of QString::fromUtf8 by either using the u prefix, or performing the encoding by hand and using the escape sequence. Be sure that your execution charset is UTF-8 (or that string literals pass through unchanged). For instance,
QString str = QString::fromUtf8("3\xb7"); // str is "÷"
- If your source is Latin-1 and you don't want to risk it