[scribus-dev] Change to UTF-32

[Andreas Vox]

> Am 31.05.2016 um 19:20 schrieb Craig Bradney <cbradney at scribus.info>:
> 
> 
> 
> Any ideas how we can break down the arguments for and against and come to a decision?
> 
> For me, the priority is to get CTL into trunk in a state that is as reliable as possible for existing functionality (i.e. Latin text) and extending to whatever scripts are already ok by CTL.
> 
> Maybe the UTF32 change is part of that, maybe not. If not, then the work should be either a) fix UTF16 handling or b) move to UTF32 as the step after merging.
> 
> Thoughts? Khaled? Avon? Jean?
> 


PRO:

- unit = code point
- no surrogate pairs used
- easier to code, less error prone

But:
- we should still handle characters that use more than one code point, even in UTF32
- what should we do with surrogate pairs that show up in UTF32 by error?


AGAINST:

- not all consequences understood
- we can still change later
- Qt and ICU use UTF16
- we can use QString without recoding


Non-Issues:

- memory requirements: we stop much more than 4 bytes per character, anyway (flags, glyph positions)
- speed: recoding is much faster than other stuff we do with text


Did I miss any arguments?


DATA:

Here’s a summary of what code is affected by Khaled’s patch  beside text layout/shaping:

* pageitem_textframe: 
  the layout code frequently accesses individual chars, also queries which char produced a certain glyph

* pdflib_core: 
  Test for == QChar(13). This is really bogus, since the code should use boxes and layout flags

* scribus150format:
 Currently some characters are stored as Unicode numbers (Unicode=„nnn“) instead of chars (ch=„xyz“). This should change to proper Unicode in UTF-8

* scriptplugin: 
  iterate over chars for selection - this is really old bad code and can result in splitting surrogate pairs. Selection should be supported by Storytext directly without iteration

* short-words: 
  replaceChar - should probably be reviewed and changed to use a search/replace facility in StoryText

* pslib: 
  Test for == QChar(13), == QChar(10), see pdflib above

* storytext:
  cursorposition, search
  These are important functions that should not only handle surrogate pairs but also multi-unit characters, language dependent collation etc.
  If we use ICU for that, using UTF16 might be better.

* bookmwin: 
  Tests == QChar(13), == QChar(10) - more bogus code

* util: 
  wordAndPara - this function should be implemented in StoryText in a language aware manner


IMHO it doesn’t matter if we use UTF32 code points, UTF16 code points or Unicode characters as unit for Storytext - in all cases we need code to handle situations where unit of storage != user perceived character for cursor positioning and selection. This should be handled by robust functions in StoryText. Instead of querying single chars, other Scribus code should use nextChar/nextWord/nextSentence or search functions and use the returned text range for text extraction or replacing. Using QString for these functions is probably safer than handling lists of UTF32 code points. 


/Andreas