[scribus-dev] Change to UTF-32

[William Bader]

> From: cbradney at scribus.info
> Date: Tue, 31 May 2016 19:20:14 +0200
> To: scribus-dev at lists.scribus.net
> Subject: Re: [scribus-dev] Change to UTF-32
> 
> 
> > On 31 May 2016, at 13:13, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > 
> > On Tue, May 31, 2016 at 05:36:59AM -0400, William Bader wrote:
> >> 
> >> 
> >>> To: scribus-dev at lists.scribus.net
> >>> From: jghali at libertysurf.fr
> >>> Date: Tue, 31 May 2016 08:53:16 +0200
> >>> Subject: Re: [scribus-dev] Change to UTF-32
> >>> 
> >>> Le 31/05/2016 à 07:29, Khaled Hosny a écrit 
> >>>>> Jeans not convinced that we need to move to UTF32 at this stage in
> >>>>> CTL, i.e. too early, not required. That we will see regressions in
> >>>>> unknown places and find it hard to prove that the code is working
> >>>>> properly for testing etc.
> >>>> The fact that we already have a few reports at GitHub about supporting
> >>>> characters outside BMP says otherwise.
> >>> 
> >>> Supporting Emojis or historic scripts is certainly something nice, but supporting them is
> >>> not in any way required at this point. So I maintain my position. UTF-32 has advantages
> >>> but this is something which can wait after CTL code merge.
> >> 
> >> 
> >> Even with UTF-32, there are still combining characters
> >> https://en.wikipedia.org/wiki/Combining_character that require several
> >> code points per character. Doesn't that raise issues similar to
> >> characters outside BMP in UTF-16? Doing UTF-32 correctly might require
> >> touching all of the places that have BMP issues in UTF-16 because they
> >> will still need to deal with characters that require multiple
> >> integers, and Qt will be messier if you can't use QString.
> > 
> > Combining characters is already handled such fine. We are not talking
> > here about user-perceived characters (grapheme clusters in Unicode
> > speak) but Unicode code points. Both UTF-8 and UTF-16 are variable size
> > encodings, so a code point can be made of one or more code units. The
> > implication of this is that if you process code units as if it were a
> > fixed length encoding you can end with illegal Unicode text, which is a
> > serious issue. With a fixed width encoding if you want to access the Nth
> > character you just access the Nth position in the array, with variable
> > length encodings you need to process the whole stream, and so on.
> > 
> > Any way, people are free to reject the parts of the patches they don’t
> > like. I don’t have this weird notion that some characters or scripts are
> > first class citizen and others we can think about them later. UTF-16 is
> > the worst of encodings (as I explained in the IRC discussion), it
> > encourages developers to write bad code, and the sorry state of non-BMP
> > in Scribus is a great testimony about this.
> > 
> 
> 
> Any ideas how we can break down the arguments for and against and come to a decision?
> 
> For me, the priority is to get CTL into trunk in a state that is as reliable as possible for existing functionality (i.e. Latin text) and extending to whatever scripts are already ok by CTL.
> 
> Maybe the UTF32 change is part of that, maybe not. If not, then the work should be either a) fix UTF16 handling or b) move to UTF32 as the step after merging.
> 
> Thoughts? Khaled? Avon? Jean?
> 
> Craig


I think that it is usually better to divide big changes into the smallest steps possible to make it easier to find bugs using bisection.
If Khaled's CTL code is mostly working and already depends on UTF32, though, it would probably be easier to take it at once instead of trying to split it into separate CTL and UTF32 patch sets.
If UTF32 is the eventual goal, I think that it isn't worth trying to fix UTF16 issues that would go away with the move to UTF32.
William


 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160531/996c99b9/attachment.html>