last day (14 days later) » 

10:23 AM
2
Q: Convert UChar* to wchar_t* using ICU in Linux for Supplementary Plane characters

D3XT3RWe have a C++ application deployed on RHEL using ICU. We have a situation where in we need to convert UChar* to wchar_t* on linux. We use u_strToWCS to perform the conversion. #include <iostream> #include <wchar.h> #include "unicode/ustring.h" void convertUnicodeStringtoWideChar(const UChar* ...

 
"It fails..." -- Input, observed output, expected output? A main() giving those infos, and wrapping this function (plus necessary #include statements) into a compilable example?
I took the liberty of slapping on some crude main() to make it MCVE. Output is 61, e4, 160, 20ac, 2f929 -- which is what I would expect. (Note the last unit, which is a non-BMP / UTF-16 surrogate pair in the input.) Repeating the question to the OP, inhowfar does it "fail"?
As UCS-2 did not know anything beyond the BMP, it's kind of hard to imagine how the OP could have encoded any non-BMP characters in it. ;-)
 
OK. I was looking at wrong directions. u_strToWCS works fine. The problem arises because I need to pass that wide string to a java application on windows using CORBA. Since wchar_t in linux is 32bit, I need to find a way to convert 32bit wchar_t to 16bit wchar_t
 
@D3XT3R: You're mixing up container datatype and encoding here, something that makes for less-than-precise communication and, potentially, misunderstanding. There are various encodings -- UTF-8, UTF-16 and UTF-32 being the "useful" ones -- and various datatypes that can be used for storing them -- std::string or char[] for UTF-8; UChar[], icu::UnicodeString, std::u16string and (Windows) wchar_t for UTF-16; UChar32[], std::u32string and (Unix) wchar_t for UTF-32. Each combination has its own pros and cons, unfortunately.
(ctd.) The UCS-2 that Davislor mentioned is what became UTF-16, before there was anything beyond the BMP. It's, basically, UTF-16 without the surrogate pairs. (Warning, oversimplification.) It's what Microsoft was looking at when opting for a 16-bit definition of wchar_t. Note I didn't list std::wstring above. 16 vs. 32 bit is evil enough when handled in terms of wchar_t; once you wrap it into std::wstring it becomes toxic. Stick to std::u16string / std::u32string if you want the standard library, and to the ICU side of things if you want the whole bells & whistles of Unicode.
 
for input of 乕乭𠁄𠁉乺丕, i received 4e55 4e6d 20044 20049 4e7a 4e15 on Linux and 4e55 4e6d d840 dc44 d840 dc49 4e7a 4e15 on Windows.
 
And what would have been your expected output? As I said, on Windows wchar_t is 16bit, so your input is "converted" to the appropriate encoding (UTF-16, which admittedly is not a "wide" encoding in the literal meaning of the term). On Linux (32bit wchar_t) you get UTF-32 encoding from this function.
Note that the contents of an ICU UChar[] are already in UTF-16 encoding. So if UTF-16 is what you are aiming for, you don't need to convert at all.
(You see the mess that the different definitions of wchar_t brought about? That's why the standard was extended by the "fixed" types, char16_t and char32_t...)
 
10:32 AM
Let me explain you my situation. I have a java web application which connects to C++ application (Linux/Windows) using CORBA. In the Java application, i input '乕乭𠁄𠁉乺丕', c++ application does some processing and returns the original text entered along with some other additional data. Now, when I see the original text which has been returned back by c++, I see '乕乭DI乺丕'. My 2 supplementary characters are garbled.
 
What is the encoding of the text in the Java application?
Is it still in that encoding when the C++ code receives it?
What is the encoding passed back from C++ to the Java code?
Is it still in that encoding when the Java code receives it?
(Check your assumptions. Find out exactly which step garbles your strings.)
 
I fetch the data from Java application in CORBA::WChar*, convert it into UChar* using u_strFromWCS, do some processing, convert UChar* back to wchar_t using u_strToWCS and then dup that processed text to CORBA::WString_out using CORBA::wstring_dup
 
That's the datatypes you're using. I was asking about the encoding. ;-)
I understand that the Windows implementation works fine, while the Linux version doesn't?
 
yes Linux doesnt
 
(Checking about CORBA:WChar, which I am not familiar with...)
The point is, it is not clear at which point of your conversion chain the breakage occurs, and which parts are really necessary for you, making it a bit hard to give advice.
 
10:45 AM
I will check all the encoding queries you raised and get back
 
{thumbs up}
 
Is it still in that encoding when the C++ code receives it? I receive it in UTF-16 (4e55 4e6d d840 dc44 d840 dc49 4e7a 4e15)
What is the encoding passed back from C++ to the Java code? UTF-32 (4e55 4e6d 20044 20049 4e7a 4e15)
What is the encoding of the text in the Java application? UTF-16
 
11:28 AM
OK.... not exactly the info I was looking for, but, anyway...

Notice that you don't *need* any conversion `u_strFromWCS` -- Java delivers in UTF-16, which is exactly what ICU processes in `UChar` arrays. Also, on Linux (where "wide character string" implies UTF-32), `u_strFromWCS` would be the *wrong* function to call in the first place. I *assume* that is where your actual problem is -- that, on Linux, you are using `u_strFromWCS` on UTF-16 input where the function expects UTF-32 input.
 

last day (14 days later) »