[scribus-dev] Change to UTF-32

[Khaled Hosny]

On Tue, May 31, 2016 at 07:11:47AM +0200, Craig Bradney wrote:
> 
> > On 31 May 2016, at 00:41, Andreas Vox <andreas.vox at gmail.com> wrote:
> > 
> > 
> >> Am 30.05.2016 um 22:10 schrieb Craig Bradney <cbradney at scribus.info>:
> >> 
> >> 
> >>> On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
> >>> 
> >>> On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
> >>>> 
> >>>>> 
> >>>> 
> >>>> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
> >>> 
> >>> Converting a QString read from 25MB text file to UTF-32 takes <
> >>> 0.4 seconds, so that does not seem to be a concern. The current slowness
> >>> is mainly bad handling of text spanning multiple frames more than
> >>> anything else and Andreas is working on a fix for this.
> >>> 
> >>> 
> >> 
> >> 
> >> http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.
> >> 
> > 
> > I wonder what they’d say if they saw our code in ScText_Shared…
> > 
> > As long as we pack every Char in an ScText struct, 8 vs. 16 vs. 32 bit doesn’t matter. A better structure would use a rope data structure or a list of strings for text and store style information as runs outside the text (coming soon).
> > 
> > My main issue is that we won’t be using QString as a model for story text. String has many handy member functions that QLsit<uint> doesn’t. It’s also possible that QList<uint> is slower for many tasks.
> > 
> > But if everyone is fine with the change, we can make it work. Text search and nextCharacter() have to be rewritten anyway in order to handle non-latin text correctly.
> > 
> > The file format will probably use UTF8 anyway. Importing/exporting text will always handle character encoding as needed.
> 
> Jeans not convinced that we need to move to UTF32 at this stage in
> CTL, i.e. too early, not required. That we will see regressions in
> unknown places and find it hard to prove that the code is working
> properly for testing etc.

The fact that we already have a few reports at GitHub about supporting
characters outside BMP says otherwise.

> The assumption we’re only using the first half of UTF16 at this point
> would seem to cover our needs for now and doesn’t involve the changes
> describe, even if its a bad assumption to ignore the upper half.
> "UTF-32 is also supposed to use more than one unit for big code
> points, but in practical, it only requires one unit to store all code
> points of Unicode 6.0”
> http://unicodebook.readthedocs.io/unicode_encodings.html

UTF-32 can have 2^32 characters, that is 4,294,967,296. There is no way
that Unicode will ever have that number of characters. By contrast
Unicode 9.0 (to be released yet) has only 128,172 characters.

> I am a little concerned with the QString vs QList<uint> idea…. and
> speed. If its basic units and internal to StoryText are we better
> ending up with QByteArrays, basic arrays (or some other structure that
> doesn’t need manual memory management)?

SoryText does not use QString even in trunk, it uses a QList of
something ScText which stores the characters as a QChar among other
things. What I did is replacing that QChar with uint and adapted the
code to work without.

Regards,
Khaled