<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font face="Liberation Serif">I thought that perhaps others would
have some interest, and maybe even some advice to me and others
about the process (I hesitate to call it a workflow, since it was
a make-it-up-as-I-went process) in turning a large amount of
information into books.<br>
<br>
What this is/was.<br>
In October, 2014, CMS (Centers for Medicare and Medicaid Services)
has mandated that health providers must begin using the ICD-10,
the 10th version of the International Classification of Diseases.
You can get a flavor of this at the WHO web site:<br>
<br>
<a class="moz-txt-link-freetext" href="http://apps.who.int/classifications/icd10/browse/2010/en">http://apps.who.int/classifications/icd10/browse/2010/en</a><br>
<br>
Internationally, this is something used simply to keep track of
the occurrences of various diseases and conditions. In the US,
it's the basis of managing the submission of information, so that
one can be paid, not only by CMS, but also private insurance
companies. ICD 10 is a dramatic change from ICD 9, with a massive
expansion of codes, not just with new diseases, but the addition
of a lot of circumstances -- check out the V codes, for example.
CMS, being the CMS, wasn't completely satisfied with the WHO
system, so they've modified it, in some areas quite substantially.
Even though the implementation is coming more than a year from
now, I wanted some way to begin to try to absorb these changes to
avoid chaos.<br>
<br>
CMS doesn't (yet) have a convenient set of web pages like WHO, but
they do have a downloadable zip file. Here's how this process went.<br>
<br>
1. I downloaded the zip file, unzipped. 13MB of pure text.<br>
<br>
2. Next, I wanted to create a database from this file, which looks
like a table. I created a Postgresql table with the appropriate
column names, and this was when I found that this text file is
just a text file, consisting of lines of text, separated by
spaces. Most fields are single alphanumeric "words", but the last
two are multiple word, variable length fields.<br>
<br>
3. So I wrote a Python script to turn this text file into fields
separated by tabs, so that I could load this into Postgresql. <br>
<br>
4. One of the columns of this table is something I labeled
"valid". These have a value of 0 or 1; if 0, this is a heading,
subheading, or sub-subheading. If I were submitting a code to some
payor, I cannot use these (they are not valid for billing), yet
they do give a structure to the overall listing to help find
things.<br>
<br>
5. The first thing I tried was to export from Postgresql as HTML
output. This becomes a 27MB file (!), very hard to efficiently use
with a browser, and there are no links. Just the headings is 6MB
in HTML.<br>
<br>
6. Then I thought perhaps creating an ePub with Sigil would be
useful. Take my advice and don't try to import 27MB into Sigil.
Even going category by category pretty quickly became unworkable.
Just the S codes are >38,000 rows. I did actually manage to do
it, with a lot of waiting between operations, sometimes 30 minutes
or more. More tedious was the creation of headings (editing
headings and subheadings into h1, h2, h3 and so on). I was hoping
that Sigil might have some scripting capability for this, but it
doesn't, so this was a HIGHLY manual operation. There is the
shortcut of simply clicking on a GUI button to convert a
<p></p> line to <h2></h2>, or the keyboard
shortcuts of Ctrl+1, Ctrl+2, etc. This is how I spent most of a
weekend.<br>
<br>
7. Sigil will generate TOCs for you. Unfortunately, these
automatically generated listings were MASSIVE, and therefore
unusable. I really needed a TOC for the TOC. So I went to the WHO
web page, and using this as a key, created new headings, setting
these as h1 and h2 headers, and everything else below that, h3 and
h4. This, too, was a manual operation (typing). NOW I had a TOC
that seemed useful when I told Sigil to only use h1 and h2.<br>
<br>
8. This was unusable on a tablet. Not a memory issue, but the app
simply choked on such a big ePub. Worked Ok on calibre on my
desktop. The next step was to break the massive ePub into 3 ePubs.
This was usable but not necessarily user-friendly, and even these
smaller ePubs are a bit clunky. <br>
<br>
9. I tried using calibre to convert the ePubs to PDFs, but this is
simply ugly, with very fuzzy text. Unacceptable.<br>
<br>
10. This is point where I brought in Scribus. Rather than begin
with plain text, I wanted to make use of the tedious work I'd
already done in Sigil. The starting point is to make a copy of the
ePubs, change the extension to .zip, then unzip. In a subfolder
you have your xhtml files. These imported nicely into Scribus,
where I modified the created styles. In Scribus, I went with a US
Letter, 2-column format to get more information on a page, which
was an issue with the ePubs.<br>
<br>
11. The next thing was to improve searchability -- you can't
simply be leafing through such a massive amount of information.
For this, I went back to Postgresql, and generated text files
consisting only of the headings and subheadings. I already knew
that using all of these for a TOC or index wasn't good. I decided
to filter out the headings that were only 3 characters, for
example, A00, A01, A02, etc. This sounds like a job for regexp, so
I wrote a Perl script to do that. This creates a list of no more
than 100 items, most of the time less, since there are gaps.<br>
</font><font face="Liberation Serif"><font face="Liberation Serif">G00
Bacterial meningitis, not elsewhere classified<br>
G03 Meningitis due to other and unspecified causes<br>
G04 Encephalitis, myelitis and encephalomyelitis<br>
G05 Encphlts, myelitis & encephalomyelitis in dis classd
elswhr<br>
and so on (these heading come from the "short" table item, an
abbreviated version of the "long" version used in the body of
the PDF)<br>
<br>
</font>12. I was hoping I could use an automated TOC creation in
Scribus, but it doesn't work that way, so I went with the manual
process of making overlying linking frames. Another decision was
to break up the ePub into 20-some separate PDFs, to have the A
codes in one, then B codes in another. Even so, the S codes were
just too big for Scribus to manage, so this became S1 and S2
(which were 9.8 MBor 562 pages and 12 MBor 661 pages PDFs
respectively). The body of each PDF has a link back to its index
on each page.<br>
<br>
13. Individually, these work pretty well, but what was lacking was
an efficient way to jump from one section to the next. Once again,
taking a cue from the WHO web page, I made a 2-page PDF to link to
the individual sections, and of course links from each section to
the main index.<br>
A00-A99 Certain infectious and parasitic diseases<br>
B00-B99 Certain infectious and parasitic diseases, continued<br>
C00-C99 Neoplasms<br>
D00-D48 Neoplasms<br>
and so on<br>
<br>
14. At this point, I'm pretty satisfied with the results. As I had
stated before in a prior post, external links don't work on my
tablet. I sent an email to Adobe, and they say that has not been
implemented, but they have put it on the list for the future.<br>
<br>
Greg<br>
</font>
</body>
</html>