[scribus-dev] Change to UTF-32

[Khaled Hosny]

On Mon, May 30, 2016 at 09:22:43PM +0200, Craig Bradney wrote:
> 
> > On 30 May 2016, at 21:14, Khaled Hosny <khaledhosny at eglug.org> wrote:
> > 
> > On Mon, May 30, 2016 at 08:56:06PM +0200, Craig Bradney wrote:
> >> 
> >>> On 30 May 2016, at 17:26, Andreͮaͦsͯ <andreas.vox at gmail.com> wrote:
> >>> 
> >>> Hi!
> >>> 
> >>> Khaled recently committed a change to the HOST-Oman/ctl branch that changes the text coding from UTF_16 (QChar/QString) to UTF-32:
> >>> https://github.com/HOST-Oman/scribus/commit/b2886e64b4aba660e4939d6cdaba066aed39d081
> >>> 
> >>> Since we have divided opinions on this issue, would you mind reviewing the discussion in the comments and add your own thoughts?
> >>> 
> >>> Best regards
> >>> /Andreas
> >>> 
> >>> 
> >> 
> >> 
> >> I don’t really see the difference because I don’t know the code. If
> >> the APIs enable me, or anyone who doesn’t know the text code to do
> >> stuff like insert text at a certain point for example, the it would
> >> seem ok.
> >> One thing that is mentioned is for example inserting a QChar.. so how
> >> does one programmatically insert 1 character?
> > 
> > StoryText has no API to insert single character, inserting QChar
> > apparently works because it can be cast to QString or some such. You can
> > now either insert a QString and the code will convert it to UTF-32, or a
> > QVector<uint> of UTF-32 code points. We might have a friendly typedef
> > for this vector.
> > 
> >> I don’t have an opinion on the 16 vs 32 as long as it works so we can
> >> do all operations.. read/write to file, display it, type it, export it
> >> to various formats.
> > 
> > The big difference is that UTF-16 is variable length, a Unicode
> > character can be represented by one or two UTF-16 code units, but
> > Scribus was treating it as a fixed length encoding which is wrong.
> > UTF-32, on the other hand, is fixed length encoding, so the assumption
> > all over Scribus codebase can be kept.
> > 
> 
> So… you don’t want to use UTF16 as its a variable length encoding (1
> or 2 16 bit code units) and we’d need to deal with the unknown length
> in all the code..

Right.

> And the downside to UTF32 is? Higher memory requirement?

Not sure about that, since we are not replacing 16-bit int by 32-bit
int, but QChar by uint, whether uint takes less or more memory than
QChar this needs to be measured (no idea how, though).

> Inability to
> directly use QStrings in storage of text? (even though there is
> QString::toUcs4 to facilitate conversion)

We can (and still) use QString in many places, mainly when interfacing
with APIs that use QString. We just need to be careful and not assume
that QString length is the same as StoryText length.

Regards,
Khaled