[scribus-dev] Change to UTF-32

[Khaled Hosny]

On Tue, May 31, 2016 at 05:36:59AM -0400, William Bader wrote:
> 
> 
> > To: scribus-dev at lists.scribus.net
> > From: jghali at libertysurf.fr
> > Date: Tue, 31 May 2016 08:53:16 +0200
> > Subject: Re: [scribus-dev] Change to UTF-32
> > 
> > Le 31/05/2016 à 07:29, Khaled Hosny a écrit 
> > >> Jeans not convinced that we need to move to UTF32 at this stage in
> > >> CTL, i.e. too early, not required. That we will see regressions in
> > >> unknown places and find it hard to prove that the code is working
> > >> properly for testing etc.
> > > The fact that we already have a few reports at GitHub about supporting
> > > characters outside BMP says otherwise.
> > 
> > Supporting Emojis or historic scripts is certainly something nice, but supporting them is
> > not in any way required at this point. So I maintain my position. UTF-32 has advantages
> > but this is something which can wait after CTL code merge.
> 
> 
> Even with UTF-32, there are still combining characters
> https://en.wikipedia.org/wiki/Combining_character that require several
> code points per character. Doesn't that raise issues similar to
> characters outside BMP in UTF-16? Doing UTF-32 correctly might require
> touching all of the places that have BMP issues in UTF-16 because they
> will still need to deal with characters that require multiple
> integers, and Qt will be messier if you can't use QString.

Combining characters is already handled such fine. We are not talking
here about user-perceived characters (grapheme clusters in Unicode
speak) but Unicode code points. Both UTF-8 and UTF-16 are variable size
encodings, so a code point can be made of one or more code units. The
implication of this is that if you process code units as if it were a
fixed length encoding you can end with illegal Unicode text, which is a
serious issue. With a fixed width encoding if you want to access the Nth
character you just access the Nth position in the array, with variable
length encodings you need to process the whole stream, and so on.

Any way, people are free to reject the parts of the patches they don’t
like. I don’t have this weird notion that some characters or scripts are
first class citizen and others we can think about them later. UTF-16 is
the worst of encodings (as I explained in the IRC discussion), it
encourages developers to write bad code, and the sorry state of non-BMP
in Scribus is a great testimony about this.

Regards,
Khaled