[scribus-dev] Change to UTF-32

Louis Desjardins louis.desjardins at gmail.com
Mon May 30 20:29:51 UTC 2016


2016-05-30 16:24 GMT-04:00 Khaled Hosny <khaledhosny at eglug.org>:

> On Mon, May 30, 2016 at 10:10:08PM +0200, Craig Bradney wrote:
> >
> > > On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > >
> > > On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
> > >>
> > >>> On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org>
> wrote:
> > >>>
> > >>> On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
> > >>>>
> > >>>>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org>
> wrote:
> > >>>>>
> > >>>>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> > >>>>>>
> > >>>>>>> On 30 May 2016, at 17:26, Andreͮaͦsͯ <andreas.vox at gmail.com>
> wrote:
> > >>>>>>>
> > >>>>>>> Hi!
> > >>>>>>>
> > >>>>>>> Khaled recently committed a change to the HOST-Oman/ctl branch
> that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> > >>>>>>>
> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> > >>>>>>>
> > >>>>>>> Since we have divided opinions on this issue, would you mind
> reviewing the discussion in the comments and add your own thoughts?
> > >>>>>>>
> > >>>>>>> Best regards
> > >>>>>>> /Andreas
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> I don’t really see the difference because I don’t know the code.
> If
> > >>>>>> the APIs enable me, or anyone who doesn’t know the text code to do
> > >>>>>> stuff like insert text at a certain point for example, the it
> would
> > >>>>>> seem ok.
> > >>>>>> One thing that is mentioned is for example inserting a QChar.. so
> how
> > >>>>>> does one programmatically insert 1 character?
> > >>>>>
> > >>>>> StoryText has no API to insert single character, inserting QChar
> > >>>>> apparently works because it can be cast to QString or some such.
> You can
> > >>>>> now either insert a QString and the code will convert it to
> UTF-32, or a
> > >>>>> QVector<uint> of UTF-32 code points. We might have a friendly
> typedef
> > >>>>> for this vector.
> > >>>>>
> > >>>>>> I don’t have an opinion on the 16 vs 32 as long as it works so we
> can
> > >>>>>> do all operations.. read/write to file, display it, type it,
> export it
> > >>>>>> to various formats.
> > >>>>>
> > >>>>> The big difference is that UTF-16 is variable length, a Unicode
> > >>>>> character can be represented by one or two UTF-16 code units, but
> > >>>>> Scribus was treating it as a fixed length encoding which is wrong.
> > >>>>> UTF-32, on the other hand, is fixed length encoding, so the
> assumption
> > >>>>> all over Scribus codebase can be kept.
> > >>>>>
> > >>>>
> > >>>> So… you don’t want to use UTF16 as its a variable length encoding (1
> > >>>> or 2 16 bit code units) and we’d need to deal with the unknown
> length
> > >>>> in all the code..
> > >>>
> > >>> Right.
> > >>>
> > >>>> And the downside to UTF32 is? Higher memory requirement?
> > >>>
> > >>> Not sure about that, since we are not replacing 16-bit int by 32-bit
> > >>> int, but QChar by uint, whether uint takes less or more memory than
> > >>> QChar this needs to be measured (no idea how, though).
> > >>>
> > >>>> Inability to
> > >>>> directly use QStrings in storage of text? (even though there is
> > >>>> QString::toUcs4 to facilitate conversion)
> > >>>
> > >>> We can (and still) use QString in many places, mainly when
> interfacing
> > >>> with APIs that use QString. We just need to be careful and not assume
> > >>> that QString length is the same as StoryText length.
> > >>>
> > >>
> > >> What about speed? We already need to speed up the text engine
> massively, including rendering (or especially rendering).
> > >
> > > Converting a QString read from 25MB text file to UTF-32 takes <
> > > 0.4 seconds, so that does not seem to be a concern. The current
> slowness
> > > is mainly bad handling of text spanning multiple frames more than
> > > anything else and Andreas is working on a fix for this.
> > >
> > >
> >
> > http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong
> wording about slow down from using UTF32.
>
> That seems like red herring mostly, if it were that bad we should have
> been using UTF-8 from the start. A 1 million character string takes ~2MB
> of memory as UTF-16 and a whopping ~4MB as UTF-32. I fail to see what
> kind of slowing this might give us in 2016. Either this page is talking
> about processing hundreds of gigabytes of text (not something Scribus
> does) or computers from the last century.
>

Correct me if I’m wrong but that is maybe why there used to be 2 versions,
one Latin and one CJK, for the commercial DTP apps. That’s now history.
Back then, the computer were, as you point out, much (much) slower, hence
the necessity of this kind of solution.

Louis

>
> _______________________________________________
> scribus-dev mailing list
> scribus-dev at lists.scribus.net
> http://lists.scribus.net/mailman/listinfo/scribus-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scribus.net/pipermail/scribus-dev/attachments/20160530/254fa2c5/attachment.html>


More information about the scribus-dev mailing list