last day (15 days later) » 

15:52
219
A: How well is Unicode supported in C++11?

R. Martinho Fernandes How well does the C++ standard library support unicode? Terribly. A quick scan through the library facilities that might provide Unicode support gives me this list: Strings library Localization library Input/output library Regular expressions library I think all but the first one provid...

@Uflex maybe. I don't know if following advice you don't understand is a good idea.
@graham.reeds haha, thanks, but I was aware of that. Check the "Acknowledgments" section ;)
No. wstring_convert always converts from what the standard calls a 'byte' string to what the standard calls a 'wide' string. This choice of words is quite confusing (I should reword the answer, I guess). A better choice would be 'serialized' and 'deserialized'. 'Serialized' is always as bytes meant for use external to your program (a file, the network, whatever), and 'deserialized' is always as whatever your program wants to use internally. In the list of conversions I gave the left is serialized and the right is deserialized. There is no provided way to deserialize anything into UTF-8...
... no matter how you want to store it (but wchar_t would be stupid). Unless, of course, it's an identity conversion. This whole thing just feels yucky. Whoever designed this had no idea what they were doing, and the committee approved it :(
@Dan does that make it clearer? If so, I'll put it in the answer.
Thanks for the feedback! I'll update the answer when I'm not browsing from a phone. But you misunderstood (could be my fault). The bytes end is always with char and is the external end. The "wide" end can be of any type and is the internal end. "Narrow/wide" is really not the same thing. The standard just uses really confusing terminology for orthogonal concepts. I'll be extra careful updating the answer to make sure I steer clear of that confusion.
@DanNissenbaum I edited the answer, tiptoeing around the terminology.
@Alexandros thanks for noticing! I fixed it. (You can do 1-char edits after a certain rep level, if I recall correctly)
u8"🍌" is a bad example imho, because it can be anything as it relies on implementation defined behavior. 🍌 is outside of character set defined in C++ ISO(2.3). Implementation can probably map u8"🍌" to u8"\U0001F34C" (or to anything it likes; and I'm not certain about one to many mapping, because I can't find "map" definition in the standard). The whole point about 🍌 still holds though ^_^
@Sergey.quixoticaxis.Ivanov what do you mean by "C++ ISO(2.3)"? There's no leeway for implementation defined behaviour here. See eel.is/c++draft/lex.string#9. If you can type it, your implementation must make it { 0xF0, 0x9F, 0x8D, 0x8C, 0 }. UTF-8 doesn't care what the character looks like: it encodes any thing from U+0000 to U+10FFFF (excluding U+D800 to U+DFFF), regardless of what meaning it has. So, unless you're using a source encoding that can't encode U+1F34C BANANA, there's no implementation-defined behaviour.
If you are using a source encoding that can't encode bananas, then you "can't type it", so there's no behaviour to discuss. If you mean that the current normative references refer to ISO/IEC 10646-1:1993, which is pre-Unicode 2.0, and thus only covers code points up to U+FFFF, then yes, technically the standard doesn't prescribe the behaviour. However, it's pretty clear that ISO/IEC 10646-1:1993 is intended to be included with all amendments, since the standard refers to things like "UTF-8", "UTF-16", "surrogate pairs", and the 0000-10FFFF range. This is a defect we (SG16) are working on atm.
@bit2shift There's some confusion. Note I said UTF-8 isn't supported as the "deserialized" form (implying that UTF-16 is the "serialized" end of the conversion, meaning it comes as bytes, not as 16-bit units). I edited the text for clarity now to say "from a UTF-16 byte stream". If you have a stream of bytes, you can't decode those to UTF-8 (you can't even decode those to a u16string because codecvt_utf16<char16_t> converts to UCS-2, against all odds)
(Well, you can decode to a u16string, but not to a UTF-16 u16string :D)
@R.MartinhoFernandes [lex.charset] defines all characters that are usable by the language, everything from the source file is mapped to this character set according to implementation defined rules ([lex.phases]). Anything outside this set is converted to universal character name. Maybe I miss something, but I can't see any rule that prevents my implementation from mapping 🍌 character to, for example, letter 'a' thus exchanging u8"🍌" to u8"a". As far as I understand, [lex.string] speaks about language after the first phase of physical file mapping.
@R.MartinhoFernandes sorry about 2.3. I was reading a copy of C++11 standard and [lex.charset] is lex 2.3. My bad.
@R.MartinhoFernandes the same goes for non-unicode. Nothing prevents an implementation from mapping "abc" in the physical file to "that is my favorite string". Because my physical file can be in any form (for example, sound or pictures), it only needs to be mapped to [lex.charset].
The problem here is the word "character" (sigh, something else we need to comb the standard for to fix). The C-word, when discussing Unicode. Abused to mean anything and everything :(.
When the standard says "Physical source file characters are mapped (...)" it means... bytes.
So the translation will map some bytes to A or to 🍌, but it won't map 🍌 to anything, because the input is bytes, not "characters".
15:53
So by character it means char?
Well, that explains things, thank you.
One little question if you don't mind? Is there any explicit statement about mapping?
We'll want to reword large sections of the standard because of this stuff.
@Sergey.quixoticaxis.Ivanov what kind of mapping you mean?
15:57
Physical files to language. If by characters in the source file mean bytes, as far as [lex.charset] and [lex.phases] go it's not obvious what exact bytes are considered character set.
Ah, that's the implementation-defined bit.
The implementation can pick any encoding.
(In practice some implementations let you pick with flags)
Yeap, they do. Well, hope the next iteration of the standard would move things forward a bit. It's a bit tricky to try and implement so platform and culture agnostic code (at least as my experience goes, and I'm a mostly using .Net with its Unicode source code).
Thank you for your time.
*so = some
Yes, things look hopeful now. Since last month there is an actual official study group with people who know about this stuff working on this.

last day (15 days later) »