[scribus-dev] Change to UTF-32

[Khaled Hosny]

On Mon, May 30, 2016 at 10:10:08PM +0200, Craig Bradney wrote:
> 
> > On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > 
> > On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
> >> 
> >>> On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org> wrote:
> >>> 
> >>> On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
> >>>> 
> >>>>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
> >>>>> 
> >>>>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> >>>>>> 
> >>>>>>> On 30 May 2016, at 17:26, Andreͮaͦsͯ <andreas.vox at gmail.com> wrote:
> >>>>>>> 
> >>>>>>> Hi!
> >>>>>>> 
> >>>>>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> >>>>>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> >>>>>>> 
> >>>>>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
> >>>>>>> 
> >>>>>>> Best regards
> >>>>>>> /Andreas
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> I don’t really see the difference because I don’t know the code. If
> >>>>>> the APIs enable me, or anyone who doesn’t know the text code to do
> >>>>>> stuff like insert text at a certain point for example, the it would
> >>>>>> seem ok.
> >>>>>> One thing that is mentioned is for example inserting a QChar.. so how
> >>>>>> does one programmatically insert 1 character?
> >>>>> 
> >>>>> StoryText has no API to insert single character, inserting QChar
> >>>>> apparently works because it can be cast to QString or some such. You can
> >>>>> now either insert a QString and the code will convert it to UTF-32, or a
> >>>>> QVector<uint> of UTF-32 code points. We might have a friendly typedef
> >>>>> for this vector.
> >>>>> 
> >>>>>> I don’t have an opinion on the 16 vs 32 as long as it works so we can
> >>>>>> do all operations.. read/write to file, display it, type it, export it
> >>>>>> to various formats.
> >>>>> 
> >>>>> The big difference is that UTF-16 is variable length, a Unicode
> >>>>> character can be represented by one or two UTF-16 code units, but
> >>>>> Scribus was treating it as a fixed length encoding which is wrong.
> >>>>> UTF-32, on the other hand, is fixed length encoding, so the assumption
> >>>>> all over Scribus codebase can be kept.
> >>>>> 
> >>>> 
> >>>> So… you don’t want to use UTF16 as its a variable length encoding (1
> >>>> or 2 16 bit code units) and we’d need to deal with the unknown length
> >>>> in all the code..
> >>> 
> >>> Right.
> >>> 
> >>>> And the downside to UTF32 is? Higher memory requirement?
> >>> 
> >>> Not sure about that, since we are not replacing 16-bit int by 32-bit
> >>> int, but QChar by uint, whether uint takes less or more memory than
> >>> QChar this needs to be measured (no idea how, though).
> >>> 
> >>>> Inability to
> >>>> directly use QStrings in storage of text? (even though there is
> >>>> QString::toUcs4 to facilitate conversion)
> >>> 
> >>> We can (and still) use QString in many places, mainly when interfacing
> >>> with APIs that use QString. We just need to be careful and not assume
> >>> that QString length is the same as StoryText length.
> >>> 
> >> 
> >> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
> > 
> > Converting a QString read from 25MB text file to UTF-32 takes <
> > 0.4 seconds, so that does not seem to be a concern. The current slowness
> > is mainly bad handling of text spanning multiple frames more than
> > anything else and Andreas is working on a fix for this.
> > 
> > 
> 
> http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.

That seems like red herring mostly, if it were that bad we should have
been using UTF-8 from the start. A 1 million character string takes ~2MB
of memory as UTF-16 and a whopping ~4MB as UTF-32. I fail to see what
kind of slowing this might give us in 2016. Either this page is talking
about processing hundreds of gigabytes of text (not something Scribus
does) or computers from the last century.