[scribus-dev] Change to UTF-32

Craig Bradney cbradney at scribus.info
Tue May 31 17:20:14 UTC 2016


> On 31 May 2016, at 13:13, Khaled Hosny <khaledhosny at eglug.org> wrote:
> 
> On Tue, May 31, 2016 at 05:36:59AM -0400, William Bader wrote:
>> 
>> 
>>> To: scribus-dev at lists.scribus.net
>>> From: jghali at libertysurf.fr
>>> Date: Tue, 31 May 2016 08:53:16 +0200
>>> Subject: Re: [scribus-dev] Change to UTF-32
>>> 
>>> Le 31/05/2016 à 07:29, Khaled Hosny a écrit 
>>>>> Jeans not convinced that we need to move to UTF32 at this stage in
>>>>> CTL, i.e. too early, not required. That we will see regressions in
>>>>> unknown places and find it hard to prove that the code is working
>>>>> properly for testing etc.
>>>> The fact that we already have a few reports at GitHub about supporting
>>>> characters outside BMP says otherwise.
>>> 
>>> Supporting Emojis or historic scripts is certainly something nice, but supporting them is
>>> not in any way required at this point. So I maintain my position. UTF-32 has advantages
>>> but this is something which can wait after CTL code merge.
>> 
>> 
>> Even with UTF-32, there are still combining characters
>> https://en.wikipedia.org/wiki/Combining_character that require several
>> code points per character. Doesn't that raise issues similar to
>> characters outside BMP in UTF-16? Doing UTF-32 correctly might require
>> touching all of the places that have BMP issues in UTF-16 because they
>> will still need to deal with characters that require multiple
>> integers, and Qt will be messier if you can't use QString.
> 
> Combining characters is already handled such fine. We are not talking
> here about user-perceived characters (grapheme clusters in Unicode
> speak) but Unicode code points. Both UTF-8 and UTF-16 are variable size
> encodings, so a code point can be made of one or more code units. The
> implication of this is that if you process code units as if it were a
> fixed length encoding you can end with illegal Unicode text, which is a
> serious issue. With a fixed width encoding if you want to access the Nth
> character you just access the Nth position in the array, with variable
> length encodings you need to process the whole stream, and so on.
> 
> Any way, people are free to reject the parts of the patches they don’t
> like. I don’t have this weird notion that some characters or scripts are
> first class citizen and others we can think about them later. UTF-16 is
> the worst of encodings (as I explained in the IRC discussion), it
> encourages developers to write bad code, and the sorry state of non-BMP
> in Scribus is a great testimony about this.
> 


Any ideas how we can break down the arguments for and against and come to a decision?

For me, the priority is to get CTL into trunk in a state that is as reliable as possible for existing functionality (i.e. Latin text) and extending to whatever scripts are already ok by CTL.

Maybe the UTF32 change is part of that, maybe not. If not, then the work should be either a) fix UTF16 handling or b) move to UTF32 as the step after merging.

Thoughts? Khaled? Avon? Jean?

Craig




More information about the scribus-dev mailing list