[scribus-dev] Change to UTF-32

[Lukas Sommer]

> but i'm not that worried. if we document
> how the included scripts had to be changed,
> at the time the new version will be stable,
> they have clear recipes for fixing their scripts.

If I have observed it correctly, the python 2.7 script interface
_before_ the UTF32 patch does the following:

– For text selection, it counts in UTF16 code units. So it is possible
(without error message), to select half a surrogate pair. This is an
ill-formed sequenze.

– For get text of this selection – surprise – it adds silently the
missing half of the surrogate pair and converts  the (now well-formed)
sequence to UTF8. This makes really unpredictable the number of
Unicode scalar values that you select/get with a given function call.
And it makes practically impossible to interate easily through a text
frame.

Because of this incoherency personally I would consider this script
interface as yet broken for characters outside BMP.

(The situation gets worser because the python 2.7 build that scribus
uses on Windows uses internally UTF16, while the python 2.7 build that
scribus uses on most Linux systems uses internally UTF32. So your
script has additionally to deal with this problem. But this is
unrelated to Scribus’ internal representation.)

I think it is even necessary to break with the current (broken) script
interface to support correctly characters outside BMP in scripts.

2016-05-31 20:12 GMT, Craig Bradney <cbradney at scribus.info>:
>
>> On 31 May 2016, at 20:52, Fahad Al-Saidi <fahad.alsaidi at gmail.com> wrote:
>>
>> My suggestion is not merging the whole CTL branch at once. Then you can
>> revert any bad decision later on. The problem is CTL branch uses git while
>> main branch uses subsvn which makes it very tedious work to merge all
>> commits manually.
>> However, from my understanding, moving to UTF-32 is the easiest way to fix
>> current problems in CTL. Unless one have time and skills to fix the
>> current bugs and keep scribus using QChar, I tend to agree with Khaled in
>> saving time and effort by switching to UTF-32.
>>
>> Now what we need is testing to see how it goes.
>>
>
>
> Anything I read comparing the various UTF encodings points either towards
> UTF8 and UTF32 being the right thing. UTF8 for reduced size and greater
> flexibility to expand as required, UTF32 for - in theory - fixed sizes.
> UTF16 just seems to be the evil opposite of the advantages of 8 and 32…
>
> Craig
>
>


-- 
Lukas Sommer