[scribus-dev] Change to UTF-32

[Craig Bradney]

> On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
> 
> On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
>> 
>>> On 30 May 2016, at 17:26, Andreͮaͦsͯ <andreas.vox at gmail.com> wrote:
>>> 
>>> Hi!
>>> 
>>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
>>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
>>> 
>>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
>>> 
>>> Best regards
>>> /Andreas
>>> 
>>> 
>> 
>> 
>> I don’t really see the difference because I don’t know the code. If
>> the APIs enable me, or anyone who doesn’t know the text code to do
>> stuff like insert text at a certain point for example, the it would
>> seem ok.
>> One thing that is mentioned is for example inserting a QChar.. so how
>> does one programmatically insert 1 character?
> 
> StoryText has no API to insert single character, inserting QChar
> apparently works because it can be cast to QString or some such. You can
> now either insert a QString and the code will convert it to UTF-32, or a
> QVector<uint> of UTF-32 code points. We might have a friendly typedef
> for this vector.
> 
>> I don’t have an opinion on the 16 vs 32 as long as it works so we can
>> do all operations.. read/write to file, display it, type it, export it
>> to various formats.
> 
> The big difference is that UTF-16 is variable length, a Unicode
> character can be represented by one or two UTF-16 code units, but
> Scribus was treating it as a fixed length encoding which is wrong.
> UTF-32, on the other hand, is fixed length encoding, so the assumption
> all over Scribus codebase can be kept.
> 

So… you don’t want to use UTF16 as its a variable length encoding (1 or 2 16 bit code units) and we’d need to deal with the unknown length in all the code..

And the downside to UTF32 is? Higher memory requirement? Inability to directly use QStrings in storage of text? (even though there is QString::toUcs4 to facilitate conversion)

http://unicode.org/faq/utf_bom.html#utf32-2:
“
Q: Should I use UTF-32 (or UCS-4) for storing Unicode strings in memory?

A: This depends. If you frequently need to access APIs that require string parameters to be in UTF-32, it may be more convenient to work with UTF-32 strings all the time. However, the downside of UTF-32 is that it forces you to use 32-bits for each character, when only 21 bits are ever needed. The number of significant bits needed for the average character in common texts is much lower, making the ratio effectively that much worse. In many situations that does not matter, and the convenience of having a fixed number of code units per character can be the deciding factor.

Increasing the storage for the same number of characters does have its cost in applications dealing with large volume of text data: it can mean exhausting cache limits sooner; it can result in noticeably increased read/write times or in reaching bandwidth limits; and it requires more space for storage. What a number of implementations do is to represent strings with UTF-16,	but individual character values with UTF-32.

The chief selling point for Unicode is providing a representation for all the world’s characters, eliminating the need for juggling multiple character sets and avoiding the associated data corruption problems. These features were enough to swing industry to the side of using Unicode (UTF-16). While a UTF-32 representation does make the programming model somewhat simpler, the increased average storage size has real drawbacks, making a complete transition to UTF-32 less compelling.
“

Jean?

Avox: why not UTF32 in your opinion?

Craig