[scribus-dev] Change to UTF-32

[Craig Bradney]

> On 30 May 2016, at 22:08, Khaled Hosny <khaledhosny at eglug.org> wrote:
> 
> On Mon, May 30, 2016 at 09:39:11PM +0200, Craig Bradney wrote:
>> 
>>> On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org> wrote:
>>> 
>>> On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
>>>> 
>>>>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
>>>>> 
>>>>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
>>>>>> 
>>>>>>> On 30 May 2016, at 17:26, Andreͮaͦsͯ <andreas.vox at gmail.com> wrote:
>>>>>>> 
>>>>>>> Hi!
>>>>>>> 
>>>>>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
>>>>>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
>>>>>>> 
>>>>>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
>>>>>>> 
>>>>>>> Best regards
>>>>>>> /Andreas
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> I don’t really see the difference because I don’t know the code. If
>>>>>> the APIs enable me, or anyone who doesn’t know the text code to do
>>>>>> stuff like insert text at a certain point for example, the it would
>>>>>> seem ok.
>>>>>> One thing that is mentioned is for example inserting a QChar.. so how
>>>>>> does one programmatically insert 1 character?
>>>>> 
>>>>> StoryText has no API to insert single character, inserting QChar
>>>>> apparently works because it can be cast to QString or some such. You can
>>>>> now either insert a QString and the code will convert it to UTF-32, or a
>>>>> QVector<uint> of UTF-32 code points. We might have a friendly typedef
>>>>> for this vector.
>>>>> 
>>>>>> I don’t have an opinion on the 16 vs 32 as long as it works so we can
>>>>>> do all operations.. read/write to file, display it, type it, export it
>>>>>> to various formats.
>>>>> 
>>>>> The big difference is that UTF-16 is variable length, a Unicode
>>>>> character can be represented by one or two UTF-16 code units, but
>>>>> Scribus was treating it as a fixed length encoding which is wrong.
>>>>> UTF-32, on the other hand, is fixed length encoding, so the assumption
>>>>> all over Scribus codebase can be kept.
>>>>> 
>>>> 
>>>> So… you don’t want to use UTF16 as its a variable length encoding (1
>>>> or 2 16 bit code units) and we’d need to deal with the unknown length
>>>> in all the code..
>>> 
>>> Right.
>>> 
>>>> And the downside to UTF32 is? Higher memory requirement?
>>> 
>>> Not sure about that, since we are not replacing 16-bit int by 32-bit
>>> int, but QChar by uint, whether uint takes less or more memory than
>>> QChar this needs to be measured (no idea how, though).
>>> 
>>>> Inability to
>>>> directly use QStrings in storage of text? (even though there is
>>>> QString::toUcs4 to facilitate conversion)
>>> 
>>> We can (and still) use QString in many places, mainly when interfacing
>>> with APIs that use QString. We just need to be careful and not assume
>>> that QString length is the same as StoryText length.
>>> 
>> 
>> What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).
> 
> Converting a QString read from 25MB text file to UTF-32 takes <
> 0.4 seconds, so that does not seem to be a concern. The current slowness
> is mainly bad handling of text spanning multiple frames more than
> anything else and Andreas is working on a fix for this.
> 
> 


http://unicode.org/faq/utf_bom.html#UTF32 has some fairly strong wording about slow down from using UTF32.


Craig