[scribus-dev] Change to UTF-32

[Craig Bradney]

> On 30 May 2016, at 21:36, Khaled Hosny <khaledhosny at eglug.org> wrote:
> 
> On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
>> 
>>> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
>>> 
>>> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
>>>> 
>>>>> On 30 May 2016, at 17:26, Andreͮaͦsͯ <andreas.vox at gmail.com> wrote:
>>>>> 
>>>>> Hi!
>>>>> 
>>>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
>>>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
>>>>> 
>>>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
>>>>> 
>>>>> Best regards
>>>>> /Andreas
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> I don’t really see the difference because I don’t know the code. If
>>>> the APIs enable me, or anyone who doesn’t know the text code to do
>>>> stuff like insert text at a certain point for example, the it would
>>>> seem ok.
>>>> One thing that is mentioned is for example inserting a QChar.. so how
>>>> does one programmatically insert 1 character?
>>> 
>>> StoryText has no API to insert single character, inserting QChar
>>> apparently works because it can be cast to QString or some such. You can
>>> now either insert a QString and the code will convert it to UTF-32, or a
>>> QVector<uint> of UTF-32 code points. We might have a friendly typedef
>>> for this vector.
>>> 
>>>> I don’t have an opinion on the 16 vs 32 as long as it works so we can
>>>> do all operations.. read/write to file, display it, type it, export it
>>>> to various formats.
>>> 
>>> The big difference is that UTF-16 is variable length, a Unicode
>>> character can be represented by one or two UTF-16 code units, but
>>> Scribus was treating it as a fixed length encoding which is wrong.
>>> UTF-32, on the other hand, is fixed length encoding, so the assumption
>>> all over Scribus codebase can be kept.
>>> 
>> 
>> So… you don’t want to use UTF16 as its a variable length encoding (1
>> or 2 16 bit code units) and we’d need to deal with the unknown length
>> in all the code..
> 
> Right.
> 
>> And the downside to UTF32 is? Higher memory requirement?
> 
> Not sure about that, since we are not replacing 16-bit int by 32-bit
> int, but QChar by uint, whether uint takes less or more memory than
> QChar this needs to be measured (no idea how, though).
> 
>> Inability to
>> directly use QStrings in storage of text? (even though there is
>> QString::toUcs4 to facilitate conversion)
> 
> We can (and still) use QString in many places, mainly when interfacing
> with APIs that use QString. We just need to be careful and not assume
> that QString length is the same as StoryText length.
> 

What about speed? We already need to speed up the text engine massively, including rendering (or especially rendering).

Craig