[scribus] Paragraph Separator and Newline Both Showing as \r in Python

[Matt Miller]

On Fri, Dec 31, 2021, at 07:11, Gregory Pittman wrote:
> On 12/30/21 20:49, Matt Miller wrote:
>> I'm loading a text frame from a utf-8 encoded text file, and within my Scribus Python code I want to search for the standard newline character, ascii value 10. When I see an ascii 10 as the line separator I want to apply a special paragraph style to the following paragraph. Most paragraphs end with the Unicode paragraph separator character, \u2029, and in those cases the default paragraph style is fine.
>> 
>> My problem is that both these types of characters are matching '\r' when I use re.search in python. also, if I select either line separator character, then do getText(), I get a '\r' no matter what. I've confirmed that my file encoding is utf-8. What am I missing? How can I search for a simple '\n' character?
>
> Hi Matt,
>
> You don't say what OS you're using.

Linux

> Maybe running dos2unix on the text would help.

Well, my ideal workflow is one where the contents of the text file instruct Scribus what paragraph and character styles to use throughout the document. So, I'm careful to put exactly the characters I want into the text file. I've been largely successful, but now I've run into a case where it seems Scribus (or Python) is losing information when I load the file. From inside Scribus I can't distinguish between a Unicode paragraph separator, \u2029, and a simple line feed, \u000A.

I'm able to open my text file from the Python console and dump it out to see that newline is displayed as "\n" and a paragraph separator is displayed as "\u2029." So, I'm suspecting the problem is with Scribus, or how I'm using it. I'm loading the file using insertHtmlText(), but I get the same bad behavior from the GUI when I do "Content | Get Text..." and load the file manually.

I've attached a text file that shows the problem. If you run the following in the scripter console from a document with a text frame named "Text1" you should see the problem:

insertHtmlText('/path/to/line-breaks.txt', 'Text1')
text=getText('Text1')
text

You should then see that both the paragraph separator and the line feed show as '\r' in the string variable. From the .sla I also see that both these characters are stored the same, as <para/>:

<StoryText>
    <DefaultStyle/>
    <ITEXT CH="This line ends with a paragraph separator (2029)."/>
    <para/>
    <ITEXT CH="This line ends with a line feed (000A)."/>
    <para/>
    <ITEXT CH="This line ends with a line separator (2028)."/>
    <breakline/>
    <ITEXT CH="This line ends with a carriage return (000D).This line ends with a vertical tab (000B)."/>
    <ITEXT Unicode="11"/>
    <ITEXT CH="This line ends with a carriage return and a line feed (000D000A)."/>
    <para/>
</StoryText>

However, here's the output I get when I dump the file from the Python console, showing '\n' for the line feed, as it should:

>>> with open('line-breaks.txt', encoding='utf-8') as file:
...     for line in file:
...         line
...
'\ufeffThis line ends with a paragraph separator (2029).\u2029This line ends with a line feed (000A).\n'
'This line ends with a line separator (2028).\u2028This line ends with a carriage return (000D).\n'
'This line ends with a vertical tab (000B).\x0bThis line ends with a carriage return and a line feed (000D000A).\n'
>>>

This test file has six different lines with six different ways to delimit the lines for testing, but, for my current purposes it's just the first two types, \u2029 and \u000A, that I need to distinguish between.

Thanks.

> You might also experiment with the text file outside of Scribus. When I 
> use Kwrite, I can use it to Find/Replace '\n' in files. You might see 
> what this matches versus '\r'.
>
> Greg
>
>
> ___
> Scribus Mailing List: scribus at lists.scribus.net
> Edit your options or unsubscribe:
> http://lists.scribus.net/mailman/listinfo/scribus
> See also:
> http://wiki.scribus.net
> http://forums.scribus.net

-- 

  Matt Miller
  mailto:matt.miller at fastmail.com
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: line-breaks.txt
URL: <http://lists.scribus.net/pipermail/scribus/attachments/20211231/e572aa50/attachment.txt>