[scribus-dev] Change to UTF-32

[Khaled Hosny]

On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
> 
> > On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > 
> > On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
> >> 
> >>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
> >>> 
> >>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> >>>> 
> >>>>> On 30 May 2016, at 17:26, Andreͮaͦsͯ <andreas.vox at gmail.com> wrote:
> >>>>> 
> >>>>> Hi!
> >>>>> 
> >>>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> >>>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> >>>>> 
> >>>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
> >>>>> 
> >>>>> Best regards
> >>>>> /Andreas
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>>> I don’t really see the difference because I don’t know the code. If
> >>>> the APIs enable me, or anyone who doesn’t know the text code to do
> >>>> stuff like insert text at a certain point for example, the it would
> >>>> seem ok.
> >>>> One thing that is mentioned is for example inserting a QChar.. so how
> >>>> does one programmatically insert 1 character?
> >>> 
> >>> StoryText has no API to insert single character, inserting QChar
> >>> apparently works because it can be cast to QString or some such. You can
> >>> now either insert a QString and the code will convert it to UTF-32, or a
> >>> QVector<uint> of UTF-32 code points. We might have a friendly typedef
> >>> for this vector.
> >>> 
> >>>> I don’t have an opinion on the 16 vs 32 as long as it works so we can
> >>>> do all operations.. read/write to file, display it, type it, export it
> >>>> to various formats.
> >>> 
> >>> The big difference is that UTF-16 is variable length, a Unicode
> >>> character can be represented by one or two UTF-16 code units, but
> >>> Scribus was treating it as a fixed length encoding which is wrong.
> >>> UTF-32, on the other hand, is fixed length encoding, so the assumption
> >>> all over Scribus codebase can be kept.
> >>> 
> >> 
> >> So… you don’t want to use UTF16 as its a variable length encoding (1
> >> or 2 16 bit code units) and we’d need to deal with the unknown length
> >> in all the code..
> > 
> > Right.
> > 
> >> And the downside to UTF32 is? Higher memory requirement?
> > 
> > Not sure about that, since we are not replacing 16-bit int by 32-bit
> > int, but QChar by uint, whether uint takes less or more memory than
> > QChar this needs to be measured (no idea how, though).
> > 
> >> Inability to
> >> directly use QStrings in storage of text? (even though there is
> >> QString::toUcs4 to facilitate conversion)
> > 
> > We can (and still) use QString in many places, mainly when interfacing
> > with APIs that use QString. We just need to be careful and not assume
> > that QString length is the same as StoryText length.
> > 
> 
> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).

Converting a QString read from 25MB text file to UTF-32 takes <
0.4 seconds, so that does not seem to be a concern. The current slowness
is mainly bad handling of text spanning multiple frames more than
anything else and Andreas is working on a fix for this.