[scribus-dev] Change to UTF-32

[Craig Bradney]

> On 31 May 2016, at 00:41, Andreas Vox <andreas.vox at gmail.com> wrote:
> 
> 
>> Am 30.05.2016 um 22:10 schrieb Craig Bradney <cbradney at scribus.info>:
>> 
>> 
>>> On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
>>> 
>>> On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
>>>> 
>>>>> 
>>>> 
>>>> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
>>> 
>>> Converting a QString read from 25MB text file to UTF-32 takes <
>>> 0.4 seconds, so that does not seem to be a concern. The current slowness
>>> is mainly bad handling of text spanning multiple frames more than
>>> anything else and Andreas is working on a fix for this.
>>> 
>>> 
>> 
>> 
>> http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.
>> 
> 
> I wonder what they’d say if they saw our code in ScText_Shared…
> 
> As long as we pack every Char in an ScText struct, 8 vs. 16 vs. 32 bit doesn’t matter. A better structure would use a rope data structure or a list of strings for text and store style information as runs outside the text (coming soon).
> 
> My main issue is that we won’t be using QString as a model for story text. String has many handy member functions that QLsit<uint> doesn’t. It’s also possible that QList<uint> is slower for many tasks.
> 
> But if everyone is fine with the change, we can make it work. Text search and nextCharacter() have to be rewritten anyway in order to handle non-latin text correctly.
> 
> The file format will probably use UTF8 anyway. Importing/exporting text will always handle character encoding as needed.

Jeans not convinced that we need to move to UTF32 at this stage in CTL, i.e. too early, not required. That we will see regressions in unknown places and find it hard to prove that the code is working properly for testing etc.

The assumption we’re only using the first half of UTF16 at this point would seem to cover our needs for now and doesn’t involve the changes describe, even if its a bad assumption to ignore the upper half. "UTF-32 is also supposed to use more than one unit for big code points, but in practical, it only requires one unit to store all code points of Unicode 6.0” http://unicodebook.readthedocs.io/unicode_encodings.html

I am a little concerned with the QString vs QList<uint> idea…. and speed. If its basic units and internal to StoryText are we better ending up with QByteArrays, basic arrays (or some other structure that doesn’t need manual memory management)?

"QString stores a string of 16-bit QChars, where each QChar corresponds one Unicode 4.0 character. (Unicode characters with code values above 65535 are stored using surrogate pairs, i.e., two consecutive Chars.)” - Does it really not deal with UTF32 properly or if its dealing with the surrogate pairs properly, do we really need to do more?.. 


Just some more 2c

Craig