[scribus] open source meanderings in creating a large or multiple large reference books.
gpittman at iglou.com
Sat Aug 17 15:55:50 UTC 2013
I thought that perhaps others would have some interest, and maybe even
some advice to me and others about the process (I hesitate to call it a
workflow, since it was a make-it-up-as-I-went process) in turning a
large amount of information into books.
What this is/was.
In October, 2014, CMS (Centers for Medicare and Medicaid Services) has
mandated that health providers must begin using the ICD-10, the 10th
version of the International Classification of Diseases. You can get a
flavor of this at the WHO web site:
Internationally, this is something used simply to keep track of the
occurrences of various diseases and conditions. In the US, it's the
basis of managing the submission of information, so that one can be
paid, not only by CMS, but also private insurance companies. ICD 10 is a
dramatic change from ICD 9, with a massive expansion of codes, not just
with new diseases, but the addition of a lot of circumstances -- check
out the V codes, for example. CMS, being the CMS, wasn't completely
satisfied with the WHO system, so they've modified it, in some areas
quite substantially. Even though the implementation is coming more than
a year from now, I wanted some way to begin to try to absorb these
changes to avoid chaos.
CMS doesn't (yet) have a convenient set of web pages like WHO, but they
do have a downloadable zip file. Here's how this process went.
1. I downloaded the zip file, unzipped. 13MB of pure text.
2. Next, I wanted to create a database from this file, which looks like
a table. I created a Postgresql table with the appropriate column names,
and this was when I found that this text file is just a text file,
consisting of lines of text, separated by spaces. Most fields are single
alphanumeric "words", but the last two are multiple word, variable
3. So I wrote a Python script to turn this text file into fields
separated by tabs, so that I could load this into Postgresql.
4. One of the columns of this table is something I labeled "valid".
These have a value of 0 or 1; if 0, this is a heading, subheading, or
sub-subheading. If I were submitting a code to some payor, I cannot use
these (they are not valid for billing), yet they do give a structure to
the overall listing to help find things.
5. The first thing I tried was to export from Postgresql as HTML output.
This becomes a 27MB file (!), very hard to efficiently use with a
browser, and there are no links. Just the headings is 6MB in HTML.
6. Then I thought perhaps creating an ePub with Sigil would be useful.
Take my advice and don't try to import 27MB into Sigil. Even going
category by category pretty quickly became unworkable. Just the S codes
are >38,000 rows. I did actually manage to do it, with a lot of waiting
between operations, sometimes 30 minutes or more. More tedious was the
creation of headings (editing headings and subheadings into h1, h2, h3
and so on). I was hoping that Sigil might have some scripting capability
for this, but it doesn't, so this was a HIGHLY manual operation. There
is the shortcut of simply clicking on a GUI button to convert a <p></p>
line to <h2></h2>, or the keyboard shortcuts of Ctrl+1, Ctrl+2, etc.
This is how I spent most of a weekend.
7. Sigil will generate TOCs for you. Unfortunately, these automatically
generated listings were MASSIVE, and therefore unusable. I really needed
a TOC for the TOC. So I went to the WHO web page, and using this as a
key, created new headings, setting these as h1 and h2 headers, and
everything else below that, h3 and h4. This, too, was a manual operation
(typing). NOW I had a TOC that seemed useful when I told Sigil to only
use h1 and h2.
8. This was unusable on a tablet. Not a memory issue, but the app simply
choked on such a big ePub. Worked Ok on calibre on my desktop. The next
step was to break the massive ePub into 3 ePubs. This was usable but not
necessarily user-friendly, and even these smaller ePubs are a bit clunky.
9. I tried using calibre to convert the ePubs to PDFs, but this is
simply ugly, with very fuzzy text. Unacceptable.
10. This is point where I brought in Scribus. Rather than begin with
plain text, I wanted to make use of the tedious work I'd already done in
Sigil. The starting point is to make a copy of the ePubs, change the
extension to .zip, then unzip. In a subfolder you have your xhtml files.
These imported nicely into Scribus, where I modified the created styles.
In Scribus, I went with a US Letter, 2-column format to get more
information on a page, which was an issue with the ePubs.
11. The next thing was to improve searchability -- you can't simply be
leafing through such a massive amount of information. For this, I went
back to Postgresql, and generated text files consisting only of the
headings and subheadings. I already knew that using all of these for a
TOC or index wasn't good. I decided to filter out the headings that were
only 3 characters, for example, A00, A01, A02, etc. This sounds like a
job for regexp, so I wrote a Perl script to do that. This creates a list
of no more than 100 items, most of the time less, since there are gaps.
G00 Bacterial meningitis, not elsewhere classified
G03 Meningitis due to other and unspecified causes
G04 Encephalitis, myelitis and encephalomyelitis
G05 Encphlts, myelitis & encephalomyelitis in dis classd elswhr
and so on (these heading come from the "short" table item, an
abbreviated version of the "long" version used in the body of the PDF)
12. I was hoping I could use an automated TOC creation in Scribus, but
it doesn't work that way, so I went with the manual process of making
overlying linking frames. Another decision was to break up the ePub into
20-some separate PDFs, to have the A codes in one, then B codes in
another. Even so, the S codes were just too big for Scribus to manage,
so this became S1 and S2 (which were 9.8 MBor 562 pages and 12 MBor 661
pages PDFs respectively). The body of each PDF has a link back to its
index on each page.
13. Individually, these work pretty well, but what was lacking was an
efficient way to jump from one section to the next. Once again, taking a
cue from the WHO web page, I made a 2-page PDF to link to the individual
sections, and of course links from each section to the main index.
A00-A99 Certain infectious and parasitic diseases
B00-B99 Certain infectious and parasitic diseases, continued
and so on
14. At this point, I'm pretty satisfied with the results. As I had
stated before in a prior post, external links don't work on my tablet. I
sent an email to Adobe, and they say that has not been implemented, but
they have put it on the list for the future.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the scribus