Return to Question

deleted 1 character in body

Source Link

edited Jul 12, 2014 at 14:17

Paulb

Situation: I have developed ofa working proof of concept, three files:

Source Link

asked Jul 10, 2014 at 9:44

Paulb

XSLT 2.0: Crawl HTML and add links

Background: I have 4 GB of text data dispersed in 250,000 html files. I want to interlink the files with <a> for the reader to click on. I have a 12 MB file of regex patterns to identify the <a> sites.

Situation: I have developed of working proof of concept, three files:

an XML file of regex patterns of where we would want to place a touch-link <a>
A test HTML file
An xslt file to read the regex patterns, and apply them to the HTML file

Concern: I have slow performance when I apply the proof of concept to full production data.

The regex patterns (test-anchor-sites.xml):

<regexes>
    <!-- validated list of HREF and IDs where a reader would want to click to -->
    <regex match="Chapter 1" href="../chapter1.html"/>
    <regex match="Chapter 2" href="../chapter2.html"/>
    <regex match="Chapter 3" href="../chapter3.html"/>
    <regex match="Chapter 4" href="../chapter4.html"/>
    <regex match="laminectomy" href="../chapter1.html" id="#d2e1346"/>
</regexes>

The test HTML:

<!DOCTYPE HTML>
<html>
    <head>
        <title>Set Anchor IDs: Test File</title>
    </head>
    <body>
        <div class="cover">
            <div><b>S</b>pinal <b>S</b>urgery</div>
        </div>
        <div class="intro">
            <div>Degeneration of one or more disc(s) of the spine is called <i>degenerative disc disease</i> (DDD).</div>
            <div>Often, degenerative DDD can be successfully treated without surgery. Chapter 1 describes these <b>non</b>-surgical treatments.</div>
            <div>Chapter 2 describes a Laminectomy, which is a surgical procedure that removes a portion of the vertebral bone called the lamina.</div>
            <div>
                <p>A discectomy is the surgical removal of herniated disc material that presses on a nerve root or the spinal cord. It is covered in Chapter 3 and Chapter 4.</p>
                <p>Open disectomy is done through a large incision, and is described in Chapter 3.</p>
                <p>Microdisectomy is minimally invasive surgery, described in Chapter 4, and is often the most appropriate treatment after conservative treatments fail to provide relief.</p>
                <div>A percutaneous discectomy is a surgical procedure in which the central portion of an intervertebral disc is accessed and removed through a cannula.</div>
            </div>
        </div>
    </body>
</html>

The style sheet to load the regex patterns and apply them to the HTML:

<xsl:stylesheet
    version="2.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    exclude-result-prefixes="xs">
    
    <xsl:output
        method="xhtml"
        html-version="5.0"
        omit-xml-declaration="yes"
        encoding="UTF-8"
        indent="yes" />
    
    <xsl:strip-space elements="*"/>
    
    <xsl:variable name="regexes" 
        select="document('test-anchor-sites.xml')"/>
    <xsl:variable name="regex-matches" 
        select="string-join($regexes//regex/@match, '|')"/>
    
    <xsl:key name="id-lookup" match="regex" use="@match"/>
    
    <!-- start -->
    
    <xsl:template match="/">
        <xsl:apply-templates/>
    </xsl:template>
    
    <xsl:template match="@* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
    </xsl:template>
    
    <xsl:template match="*/text()">
        <xsl:analyze-string select="." regex="{$regex-matches}">
            <xsl:matching-substring>
                <a>
                    <xsl:attribute name="href" select="key('id-lookup',.,$regexes)/@href"/>
                    <xsl:value-of select="."/>
                </a>
            </xsl:matching-substring>
            <xsl:non-matching-substring>
                <xsl:value-of select="."/>
            </xsl:non-matching-substring>
        </xsl:analyze-string>
    </xsl:template>
        
</xsl:stylesheet>

Result: Code runs. Does what is desired. But it takes a very long time. Based on a partial run, I estimate this would run for 24 hours to apply 12 MB of regex patterns that inter-link 4 GB of html.

Is there a more efficient way to do this?

Design notes:

Yes, it has occurred to me: maybe 24 hours is OK. After all, applying thousands of regex patterns to 250K html files is a tall order.
I will place some axis checks in the <xsl:template match="*/text()"> code to refine the text that is crawled, such as: [not(self::toc)] or [ancestor::chapter]. I expect this to trim run-time about 10%, not a big change, but it helps.
Not shown here: an xslt that applies the xslt above to a document collection. I don't think this is the problem. It's code that has worked in another system for a very long time.

html parsing xslt