For work, I extract information from HTML and XML sources to save in databases. The objective is to generate JSON representing the source document's information and its relationships in order to 1) extract key information for database columns, like the title of a court case, and 2) to save the whole resulting JSON structure in a PostgreSQL JSONB column for possible later reference.
Normally this is done piecemeal using Xpaths with the lxml.etree or lxml.html libraries in Python, which our system is written in, but we wanted to try solutions using XSLT 3.0 and Xpath 3.1, which they do not support.
I'm mainly asking for feedback on the XSLT, but for context, this is what's happening:
The solution we hit on is to invoke the saxon-js processor for nodejs from within Python with check_output(), capture the output, and load it as JSON.
This is the input. It's a version of https://pubapps2.usitc.gov/337external/4921 run through Python's lxml.html.clean, parsed, and output again with html.tostring(root, method='xml') in order to make it parsable with XSLT. saxon-js is being run with the -ns:##html5 option to simplify the Xpath namespaces.
<html:div xmlns:html="http://www.w3.org/1999/xhtml">
<!--2.5-->
<html:meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"/>
Investigation Detail
<html:meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<html:link rel="shortcut icon" href="/337external/static/images/favicon.ico" type="image/x-icon"/>
<html:link rel="apple-touch-icon" href="/337external/static/images/apple-touch-icon.png"/>
<html:link rel="apple-touch-icon" sizes="114x114" href="/337external/static/images/apple-touch-icon-retina.png"/>
<!-- <link rel="stylesheet" href="/337external/static/css/main.css" type="text/css">-->
<html:div id="page-wrap">
<html:div id="header">
<html:div id="agencytitle">
<html:h1>United States<html:br/>
International Trade Commission</html:h1>
</html:div>
<html:div id="agencytitle" style="width:inherit;">
<html:h1 style="font-size: xx-large;">337Info</html:h1>
</html:div>
<html:div id="topnav">
<html:ul>
<html:li><html:a href="/337external/">Home</html:a></html:li>
<html:li class="help"><html:a href="/337external/advanced">Advanced Search</html:a></html:li>
<html:li class="help"><html:a href="http://www.usitc.gov/documents/337Info_FAQ.pdf" target="_blank">FAQ</html:a></html:li>
<html:li class="help"><html:a href="/337external/help">Help</html:a></html:li>
<html:li class="help"><html:a href="http://www.usitc.gov/documents/337Info_tutorial.pptx" target="_blank">Tutorial</html:a></html:li>
<html:li class="help"><html:a href="mailto:[email protected]?subject=337Info%20External%20help">Contact Us</html:a></html:li>
<html:li class="help"><html:a href="/337external/disclaimer">Disclaimer</html:a></html:li>
</html:ul>
</html:div>
</html:div>
<html:div id="inside">
<html:div id="detail-main-content">
<html:div id="main-filter-content"><html:h2> Summary Investigation Information</html:h2></html:div>
<html:div id="main-detail-window">
<html:div style="float:right">
<html:p>
<html:span style="font-weight:bolder">Investigation Number:</html:span>
<html:span style="margin-left:.5em;">337-TA-1185</html:span>
</html:p>
<html:p>
<html:span style="font-weight:bolder">Investigation Type:</html:span>
<html:span style="margin-left:.5em;"> Violation</html:span>
</html:p>
<html:p>
<html:span style="font-weight:bolder">Docket Number:</html:span>
<html:span style="margin-left:.5em;"> 3418</html:span>
</html:p>
<html:p>
<html:span style="font-weight:bolder">Investigation Status:</html:span>
<html:span style="margin-left:.5em;"> Terminated</html:span>
</html:p>
</html:div>
<html:div id="titlecontainer">
<html:p><html:span style="font-weight:bolder">Title (In the Matter of Certain):</html:span></html:p>
<html:h2>Certain Smart Thermostats, Smart HVAC Systems, and Components Thereof; Inv. No. 337-TA-1185</html:h2>
</html:div>
</html:div>
<!--Start right-container -->
<html:div id="right-filter">
<html:div id="right-filter-content-detail2"><html:h2><html:img src="/337external/static/images/scheduled.png" width="24" height="20" alt="People Icon"/>Procedural History</html:h2></html:div>
<!-- START of Prcedural history -->
<html:div id="returned-detail-content">
<html:p><html:span class="dateTitle">Complaint Filed</html:span>
<html:span class="Returneddate">10/22/2019</html:span></html:p>
<html:p><html:span class="dateTitle">Date of Institution</html:span>
<html:span class="Returneddate">11/27/2019</html:span></html:p>
<html:p><html:span class="mainTitle">Markman Hearing Dates</html:span></html:p>
<html:p><html:span class="SubTitle">Start </html:span>
<html:span class="Returneddate"/></html:p>
<html:p><html:span class="SubTitle">End </html:span>
<html:span class="Returneddate"/></html:p>
<html:p><html:span class="mainTitle"> Evidentiary Hearing Dates</html:span></html:p>
<html:p><html:span class="SubTitle"> Scheduled Start </html:span>
<html:span class="Returneddate">07/21/2020</html:span></html:p>
<html:p><html:span class="SubTitle"> Scheduled End </html:span>
<html:span class="Returneddate">07/24/2020</html:span></html:p>
<html:p><html:span class="SubTitle"> Actual Start </html:span>
<html:span class="Returneddate">11/16/2020</html:span></html:p>
<html:p><html:span class="SubTitle"> Actual End </html:span>
<html:span class="Returneddate">11/19/2020</html:span></html:p>
<html:p><html:span class="mainTitle"> Target Date</html:span>
<html:span class="Returneddate">08/20/2021</html:span></html:p>
<html:p><html:span class="mainTitle"> Final ID On Violation</html:span></html:p>
<html:p><html:span class="SubTitle"> Due Date</html:span>
<html:span class="Returneddate">04/20/2021</html:span></html:p>
<html:p><html:span class="SubTitle"> Issue Date</html:span>
<html:span class="Returneddate">04/20/2021</html:span></html:p>
<html:p><html:span class="mainTitle"> Non Final (Terminating) ID Issued </html:span>
<html:span class="Returneddate"/></html:p>
<html:p><html:span class="mainTitle"> Final Determination of No Violation</html:span>
<html:span class="Returneddate">07/20/2021</html:span></html:p>
<html:p><html:span class="mainTitle"> Final Determination of Violation</html:span>
<html:span class="Returneddate"/></html:p>
<html:p><html:span class="mainTitle"> Termination Date</html:span>
<html:span class="Returneddate">07/20/2021</html:span></html:p>
</html:div>
<html:div> </html:div>
<!-- END of Prcedural history -->
<!-- START of invUnfairAct -->
<html:div id="returned-detail-content2">
<html:div id="right-filter-content-detailnested"><html:h3>Unfair Act Alleged<html:span class="expand float_right">+</html:span></html:h3></html:div>
<html:table id="investigations" width="100%" class="visible_none" cellpadding="0px" cellspacing="0px">
<html:tbody><html:tr>
<html:th style="width:65%">Type </html:th>
<html:th>Active - Inactive</html:th>
</html:tr>
<html:tr>
<html:td>Patent Infringement</html:td>
<html:td>11/22/2019 - </html:td>
</html:tr>
</html:tbody></html:table>
</html:div>
<html:div> </html:div>
<!-- END of invUnfairAct -->
<!-- START of IP -->
<html:div id="returned-detail-content2">
<html:div id="right-filter-content-detailnested"><html:h3>Patent Number(s) <html:span class="expand float_right">+</html:span></html:h3> </html:div>
<html:table id="investigations" class="visible_none" width="100%" cellpadding="0px" cellspacing="0px">
<html:tbody><html:tr>
<html:th>Number </html:th>
<html:th>Active - Inactive</html:th>
</html:tr>
<html:tr>
<html:td>10,018,371</html:td>
<html:td>11/22/2019 - 07/20/2021</html:td>
</html:tr>
<html:tr>
<html:td>8,131,497</html:td>
<html:td>11/22/2019 - 07/20/2021</html:td>
</html:tr>
<html:tr>
<html:td>8,432,322</html:td>
<html:td>11/22/2019 - 07/20/2021</html:td>
</html:tr>
<html:tr>
<html:td>8,498,753</html:td>
<html:td>11/22/2019 - 12/15/2020</html:td>
</html:tr>
</html:tbody></html:table>
<html:div id="right-filter-content-detailnested"><html:h3>HTS Number(s) <html:span class="expand float_right">+</html:span></html:h3></html:div>
<html:table id="investigations" class="visible_none" width="100%" cellpadding="0" cellspacing="0">
<html:tbody><html:tr>
<html:th>Number </html:th>
<html:th>Category Basket</html:th>
</html:tr>
<html:tr>
<html:td>90321000</html:td>
<html:td>Consumer Electronics Products</html:td>
</html:tr>
<html:tr>
<html:td>90322000</html:td>
<html:td>Consumer Electronics Products</html:td>
</html:tr>
<html:tr>
<html:td>90328960</html:td>
<html:td>Consumer Electronics Products</html:td>
</html:tr>
</html:tbody></html:table>
</html:div>
<html:div> </html:div>
<!-- END of IP -->
<!-- START of TEO -->
<!-- END of TEO -->
<!-- START of Remand -->
<!-- END of Remand -->
</html:div>
<!--End right-container -->
<!--
Start landing-container
-->
<html:div id="detail-left-filter">
<html:div id="left-filter-content"><html:h2><html:img src="/337external/static/images/participants_icon.png" width="24" height="20" alt="People Icon"/> Participant Information</html:h2></html:div>
<html:div id="returned-detail-content3">
<html:div id="left-filter-content">
<html:h3>
<html:div class="float_right">
<html:div id="active">☑ <html:span style="font-size:0.6em; padding-right:5px"> - Active </html:span></html:div>
<html:div id="inactive">☒<html:span style="font-size:0.6em"> - Inactive </html:span></html:div>
</html:div>
Complainant Information
</html:h3>
</html:div>
<html:table id="investigations" width="100%" cellpadding="0" cellspacing="0">
<html:tbody><html:tr>
<html:th>Name - City/State/Country</html:th>
<html:th>Lead Counsel for Service</html:th>
<html:th>Active Inactive Date</html:th>
</html:tr>
<html:tr>
<html:td>
<html:div id="active"> ☑ </html:div>
<html:div id="name">
EcoFactor, Inc. - Palo Alto , CA , United States of America
</html:div>
</html:td>
<html:td>Russ August & Kabat</html:td>
<html:td>11/22/2019 - </html:td>
</html:tr>
</html:tbody></html:table>
<html:div id="left-filter-content">
<html:h3>
<html:div class="float_right">
<html:div id="active">☑ <html:span style="font-size:0.6em; padding-right:5px"> - Active </html:span></html:div>
<html:div id="inactive">☒<html:span style="font-size:0.6em"> - Inactive </html:span></html:div>
</html:div>
Respondent Information
</html:h3>
</html:div>
<html:div id="bbGrid-subgrid">
<html:div class="bbGrid-container">
<html:table id="investigations" class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th class="icon-plus-main" style="width:15px">+</html:th>
<html:th rowspan="2">Name - City/State/Country</html:th>
<html:th rowspan="2">Lead Counsel for Service</html:th>
<html:th rowspan="2">Active Inactive Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td> <html:div id="active"> ☑</html:div> <html:div id="name">Alarm.com Holdings, Inc. - Tysons , VA , United States</html:div>
</html:td>
<html:td>
Foster, Murphy, Altman & Nickel, PC
</html:td>
<html:td>10/22/2019
-
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> No Violation Found 07/20/2021</html:td>
<html:td> 07/20/2021</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td> <html:div id="active"> ☑</html:div> <html:div id="name">Alarm.com Incorporated - Tysons , VA , United States</html:div>
</html:td>
<html:td>
Foster, Murphy, Altman & Nickel, PC
</html:td>
<html:td>11/22/2019
-
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> No Violation Found 07/20/2021</html:td>
<html:td> 07/20/2021</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td><html:div id="inactive">☒</html:div> <html:div id="name">Daikin America, Inc. - Orangeburg , NY , United States</html:div>
</html:td>
<html:td>
Latham & Watkins LLP
</html:td>
<html:td>11/22/2019
-
07/01/2020
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> Settlement 07/01/2020</html:td>
<html:td> 07/01/2020</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td><html:div id="inactive">☒</html:div> <html:div id="name">Daikin Industries, Ltd. - Osaka , , Japan</html:div>
</html:td>
<html:td>
Latham & Watkins LLP
</html:td>
<html:td>11/22/2019
-
07/01/2020
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> Settlement 07/01/2020</html:td>
<html:td> 07/01/2020</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td><html:div id="inactive">☒</html:div> <html:div id="name">Daikin North America LLC - Houston , TX , United States</html:div>
</html:td>
<html:td>
Latham & Watkins LLP
</html:td>
<html:td>11/22/2019
-
07/01/2020
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> Settlement 07/01/2020</html:td>
<html:td> 07/01/2020</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td> <html:div id="active"> ☑</html:div> <html:div id="name">Ecobee Ltd. - Toronto , , Canada</html:div>
</html:td>
<html:td>
Venable LLP
</html:td>
<html:td>11/22/2019
-
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> No Violation Found 07/20/2021</html:td>
<html:td> 07/20/2021</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td> <html:div id="active"> ☑</html:div> <html:div id="name">Ecobee, Inc. - Toronto , , Canada</html:div>
</html:td>
<html:td>
Venable LLP
</html:td>
<html:td>11/22/2019
-
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> No Violation Found 07/20/2021</html:td>
<html:td> 07/20/2021</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td> <html:div id="active"> ☑</html:div> <html:div id="name">Google LLC - Mountain View , CA , United States</html:div>
</html:td>
<html:td>
WHITE & CASE LLP
</html:td>
<html:td>11/22/2019
-
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> No Violation Found 07/20/2021</html:td>
<html:td> 07/20/2021</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td><html:div id="inactive">☒</html:div> <html:div id="name">Schneider Electric SE - Rueil-Malmaison , , France</html:div>
</html:td>
<html:td>
Jenner & Block LLP
</html:td>
<html:td>11/22/2019
-
08/31/2020
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> Settlement 08/31/2020</html:td>
<html:td> 08/31/2020</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td><html:div id="inactive">☒</html:div> <html:div id="name">Schneider Electric USA, Inc. - Andover , MA , United States</html:div>
</html:td>
<html:td>
Jenner & Block LLP
</html:td>
<html:td>11/22/2019
-
08/31/2020
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> Settlement 08/31/2020</html:td>
<html:td> 08/31/2020</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
<html:tr class="bbGrid-row">
<html:td class="bbGrid-subgrid-control">
<html:span class="icon-plus">+</html:span>
</html:td>
<html:td> <html:div id="active"> ☑</html:div> <html:div id="name">Vivant, Inc. - Provo , UT , United States</html:div>
</html:td>
<html:td>
Williams Simons & Landis PLLC
</html:td>
<html:td>11/22/2019
-
</html:td>
</html:tr>
<html:tr class="bbGrid-subgrid-row visible_none">
<html:td/>
<html:td colspan="4">
<html:div class="bbGrid-container">
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Disposition, Date</html:th>
<html:th>Disposition by Unfair Acts, Date</html:th>
<html:th>Remedial Orders Issued, Issue Date, Status, Change Date</html:th>
</html:tr>
</html:thead>
<html:tbody>
<html:tr class="bbGrid-row" bgcolor="white">
<html:td> No Violation Found 07/20/2021</html:td>
<html:td> 07/20/2021</html:td>
<html:td> </html:td>
</html:tr>
</html:tbody>
</html:table>
<html:table class="bbGrid-grid table table-bordered table-condensed" width="100%" cellpadding="0" cellspacing="0">
<html:thead class="bbGrid-grid-head">
<html:tr class="bbGrid-grid-head-holder">
<html:th>Customs Enforcement Desc</html:th>
<html:th>Forum</html:th>
<html:th>Receipt Customs Letter</html:th>
<html:th>Seizure Forfeiture Order</html:th>
<html:th>Documents</html:th>
</html:tr>
</html:thead>
<html:tbody>
</html:tbody>
</html:table>
</html:div>
</html:td>
</html:tr>
</html:tbody>
</html:table>
</html:div></html:div>
</html:div>
<html:div id="left-filter-content">
<html:h2><html:img src="/337external/static/images/participants_icon.png" width="24" height="20" alt="People Icon"/> Agency Participant Information</html:h2>
</html:div>
<html:div id="returned-detail-content3">
<html:div style="width:35%;float:left;">
<html:div id="left-filter-content"><html:h3>Office of Unfair Import Investigations (OUII)</html:h3>
</html:div>
<html:div id="returned-detail-content">
<html:p>
<html:span style="font-weight:bolder">Level of Participation:</html:span>
<html:span style="margin-left:.5em;"> Full </html:span>
</html:p>
<html:br/>
<html:table id="investigations" width="100%" cellpadding="0px" cellspacing="0px">
<html:tbody><html:tr>
<html:th>Name </html:th>
</html:tr>
<html:tr>
<html:td>
<html:div id="active"> ☑ </html:div>
<html:div class="name"> Jeffrey Hsu</html:div>
</html:td>
</html:tr>
<html:tr>
<html:td>
<html:div id="active"> ☑ </html:div>
<html:div class="name"> Paul Gennari</html:div>
</html:td>
</html:tr>
</html:tbody></html:table>
</html:div>
</html:div>
<html:div style="width:30%;float:left;">
<html:div id="left-filter-content"><html:h3>General Counsel (GC)</html:h3>
</html:div>
<html:div id="returned-detail-content">
<html:table id="investigations" width="100%" cellpadding="0px" cellspacing="0px">
<html:tbody><html:tr>
<html:th>Name </html:th>
</html:tr>
<html:tr>
<html:td>
<html:div id="active"> ☑ </html:div>
Michael Liberman
</html:td>
</html:tr>
<html:tr>
<html:td>
<html:div id="active"> ☑ </html:div>
Megan Valentine
</html:td>
</html:tr>
<html:tr>
<html:td>
<html:div id="inactive"> ☒ </html:div>
Houda Morad
</html:td>
</html:tr>
</html:tbody></html:table>
</html:div>
</html:div>
<html:div style="width:35%;float:left;">
<html:div id="left-filter-content"><html:h3>Administrative Law Judge (ALJ) </html:h3>
</html:div>
<html:div id="returned-detail-content">
<html:p>
</html:p><html:table id="investigations" width="100%" cellpadding="0px" cellspacing="0px">
<html:tbody><html:tr>
<html:th>Name </html:th>
</html:tr>
<html:tr>
<html:td>
<html:div id="active"> ☑ </html:div> David Shaw
</html:td>
</html:tr>
</html:tbody></html:table>
</html:div>
</html:div>
<html:div style="float:right;width:100%">
<html:div style="float:right;">
<html:div id="active">☑ <html:span style="font-size:0.6em; padding-right:5px"> - Active </html:span></html:div>
<html:div id="inactive">☒<html:span style="font-size:0.6em"> - Inactive </html:span></html:div>
</html:div>
</html:div>
<html:br clear="all"/>
</html:div>
</html:div>
</html:div>
<!--End landing-container -->
<html:div style="clear: both;"/>
</html:div>
<html:div id="footer">
<html:p> </html:p>
<html:div class="address"> … </html:div>
<html:p> </html:p>
<html:div class="midSection" align="left"> … </html:div>
<html:p/>
<html:p> </html:p>
<html:div class="midSectionRight" align="left"> … </html:div>
<html:p/>
<html:p> </html:p>
<html:div class="right" align="left"> </html:div>
</html:div>
<html:div style="clear: both;"/>
</html:div>
</html:div>
This is my desired output:
{'Office of Unfair Import Investigations (OUII)': {'Level of Participation': 'Full',
'people': [{'name': 'Jeffrey Hsu', 'status': 'active'},
{'name': 'Paul Gennari', 'status': 'active'}]},
'Administrative Law Judge (ALJ) ': {'people': [{'name': 'David Shaw',
'status': 'active'}]},
'General Counsel (GC)': {'people': [{'name': 'Michael Liberman',
'status': 'active'},
{'name': 'Megan Valentine', 'status': 'active'},
{'name': 'Houda Morad', 'status': 'inactive'}]},
'Patent Number(s)': [{'Number': '10,018,371',
'Active - Inactive': '11/22/2019 - 07/20/2021'},
{'Number': '8,131,497', 'Active - Inactive': '11/22/2019 - 07/20/2021'},
{'Number': '8,432,322', 'Active - Inactive': '11/22/2019 - 07/20/2021'},
{'Number': '8,498,753', 'Active - Inactive': '11/22/2019 - 12/15/2020'}],
'Investigation Type': 'Violation',
'Investigation Number': '337-TA-1185',
'Investigation Status': 'Terminated',
'Respondent Information': [{'Active Inactive Date': '10/22/2019 -',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/20/2021',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'No Violation Found 07/20/2021'}],
'Lead Counsel for Service': 'Foster, Murphy, Altman & Nickel, PC',
'Name - City/State/Country': 'Alarm.com Holdings, Inc. - Tysons , VA , United States'},
{'Active Inactive Date': '11/22/2019 -',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/20/2021',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'No Violation Found 07/20/2021'}],
'Lead Counsel for Service': 'Foster, Murphy, Altman & Nickel, PC',
'Name - City/State/Country': 'Alarm.com Incorporated - Tysons , VA , United States'},
{'Active Inactive Date': '11/22/2019 - 07/01/2020',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/01/2020',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'Settlement 07/01/2020'}],
'Lead Counsel for Service': 'Latham & Watkins LLP',
'Name - City/State/Country': 'Daikin America, Inc. - Orangeburg , NY , United States'},
{'Active Inactive Date': '11/22/2019 - 07/01/2020',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/01/2020',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'Settlement 07/01/2020'}],
'Lead Counsel for Service': 'Latham & Watkins LLP',
'Name - City/State/Country': 'Daikin Industries, Ltd. - Osaka , , Japan'},
{'Active Inactive Date': '11/22/2019 - 07/01/2020',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/01/2020',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'Settlement 07/01/2020'}],
'Lead Counsel for Service': 'Latham & Watkins LLP',
'Name - City/State/Country': 'Daikin North America LLC - Houston , TX , United States'},
{'Active Inactive Date': '11/22/2019 -',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/20/2021',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'No Violation Found 07/20/2021'}],
'Lead Counsel for Service': 'Venable LLP',
'Name - City/State/Country': 'Ecobee Ltd. - Toronto , , Canada'},
{'Active Inactive Date': '11/22/2019 -',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/20/2021',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'No Violation Found 07/20/2021'}],
'Lead Counsel for Service': 'Venable LLP',
'Name - City/State/Country': 'Ecobee, Inc. - Toronto , , Canada'},
{'Active Inactive Date': '11/22/2019 -',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/20/2021',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'No Violation Found 07/20/2021'}],
'Lead Counsel for Service': 'WHITE & CASE LLP',
'Name - City/State/Country': 'Google LLC - Mountain View , CA , United States'},
{'Active Inactive Date': '11/22/2019 - 08/31/2020',
'Dispositions': [{'Disposition by Unfair Acts, Date': '08/31/2020',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'Settlement 08/31/2020'}],
'Lead Counsel for Service': 'Jenner & Block LLP',
'Name - City/State/Country': 'Schneider Electric SE - Rueil-Malmaison , , France'},
{'Active Inactive Date': '11/22/2019 - 08/31/2020',
'Dispositions': [{'Disposition by Unfair Acts, Date': '08/31/2020',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'Settlement 08/31/2020'}],
'Lead Counsel for Service': 'Jenner & Block LLP',
'Name - City/State/Country': 'Schneider Electric USA, Inc. - Andover , MA , United States'},
{'Active Inactive Date': '11/22/2019 -',
'Dispositions': [{'Disposition by Unfair Acts, Date': '07/20/2021',
'Remedial Orders Issued, Issue Date, Status, Change Date': '',
'Disposition, Date': 'No Violation Found 07/20/2021'}],
'Lead Counsel for Service': 'Williams Simons & Landis PLLC',
'Name - City/State/Country': 'Vivant, Inc. - Provo , UT , United States'}],
'Docket Number': '3418',
'HTS Number(s)': [{'Category Basket': 'Consumer Electronics Products',
'Number': '90321000'},
{'Category Basket': 'Consumer Electronics Products', 'Number': '90322000'},
{'Category Basket': 'Consumer Electronics Products', 'Number': '90328960'}],
'Procedural History': {'Non Final (Terminating) ID Issued': '',
'Final Determination of No Violation': '07/20/2021',
'Final Determination of Violation': '',
'Termination Date': '07/20/2021',
'Due Date': '04/20/2021',
'Evidentiary Hearing Dates': {'Scheduled Start': '07/21/2020',
'Actual Start': '11/16/2020',
'Actual End': '11/19/2020',
'Scheduled End': '07/24/2020'},
'Complaint Filed': '10/22/2019',
'Issue Date': '04/20/2021',
'Scheduled Start': '07/21/2020',
'Date of Institution': '11/27/2019',
'Actual Start': '11/16/2020',
'Final ID On Violation': {'Due Date': '04/20/2021',
'Issue Date': '04/20/2021'},
'Markman Hearing Dates': {'Start': '', 'End': ''},
'Actual End': '11/19/2020',
'Target Date': '08/20/2021',
'Scheduled End': '07/24/2020',
'Start': '',
'End': ''},
'Unfair Act Alleged': [{'Type': 'Patent Infringement',
'Active - Inactive': '11/22/2019 -'}],
'Complainant Information': [{'Active Inactive Date': '11/22/2019 -',
'Lead Counsel for Service': 'Russ August & Kabat',
'Name - City/State/Country': ''}],
'title': 'Certain Smart Thermostats, Smart HVAC Systems, and Components Thereof; Inv. No. 337-TA-1185'}
This is the current state of my XSLT that produces that JSON.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:map="http://www.w3.org/2005/xpath-functions/map"
version="3.0">
<xsl:output method="json"/>
<xsl:template match='/'>
<xsl:map>
<xsl:map-entry key="'title'" select="string(//div[@id='titlecontainer']/h2)"/>
<xsl:apply-templates/>
</xsl:map>
</xsl:template>
<!-- Overrides default template for text-->
<xsl:template match="text()"/>
<!--Used for title block info, Procedural History, and Office of Unfair Import Investigations (OUII) tables-->
<xsl:template match="p[count(span) = 2]">
<xsl:map-entry key="translate(normalize-space(./span[1]), ':', '')" select="normalize-space(./span[2])"/>
</xsl:template>
<!-- Procedural History-->
<xsl:template match="div[@id='right-filter']/div[@id = 'returned-detail-content']">
<xsl:map-entry key="'Procedural History'">
<xsl:map>
<xsl:for-each-group select=".//p[span[contains(@class, 'SubTitle')]]" group-by="normalize-space(preceding-sibling::p[span[contains(@class, 'mainTitle')]][1])">
<xsl:map-entry key="current-grouping-key()">
<xsl:map>
<xsl:apply-templates select="current-group()"/>
</xsl:map>
</xsl:map-entry>
</xsl:for-each-group>
<xsl:apply-templates/>
</xsl:map>
</xsl:map-entry>
</xsl:template>
<!--Rightmost tables and Complainant Information-->
<xsl:template match="table[preceding-sibling::div[@id = 'right-filter-content-detailnested' or @id='left-filter-content']]">
<xsl:map-entry key="normalize-space(translate(./preceding-sibling::div[1]/h3/text()[normalize-space(.)], '+', ''))">
<xsl:variable name="headers" select=".//th"/>
<xsl:sequence select="array{.//tr[td] ! map:merge(for $i in 1 to count($headers) return map{normalize-space($headers[$i]): normalize-space(./td[$i]/text()[normalize-space(.)])})}"/>
</xsl:map-entry>
</xsl:template>
<!-- Agency Participant Information tables-->
<xsl:template match="div[@id='returned-detail-content3']/div[contains(@style, 'float:left;')]">
<xsl:map>
<xsl:map-entry key="string(./div[@id='left-filter-content']/h3)">
<xsl:map>
<xsl:apply-templates/>
</xsl:map>
</xsl:map-entry>
</xsl:map>
</xsl:template>
<xsl:template match="tbody[tr/th[normalize-space(.) = 'Name']]">
<xsl:variable name="people" as="map(*)*">
<xsl:apply-templates/>
</xsl:variable>
<xsl:map-entry key="'people'" select="array{$people}"/>
</xsl:template>
<!--OUII rows-->
<xsl:template match="div[contains(@style, 'float:left;')]//td[div[contains(@class, 'name')]]">
<xsl:map>
<xsl:map-entry key="'name'" select="normalize-space(./div[contains(@class, 'name')])"/>
<xsl:map-entry key="'status'" select="string(./div[1]/@id)"/>
</xsl:map>
</xsl:template>
<!--Other rows-->
<xsl:template match="div[contains(@style, 'float:left;')]//td[count(div)=1 and div[@id='active' or @id='inactive']]">
<xsl:map>
<xsl:map-entry key="'name'" select="normalize-space(./text()[normalize-space(.)])"/>
<xsl:map-entry key="'status'" select="string(./div[1]/@id)"/>
</xsl:map>
</xsl:template>
<!-- Respondent Information-->
<xsl:template match="div[@id='bbGrid-subgrid']">
<xsl:map-entry key="normalize-space(./preceding-sibling::div[1]/h3/text()[normalize-space()])">
<xsl:variable name="rows" as="map(*)*">
<xsl:apply-templates select="./div/table/tbody/tr[not(.//table)]"/>
</xsl:variable>
<xsl:sequence select="array{$rows}"/>
</xsl:map-entry>
</xsl:template>
<xsl:template match="tr[contains(@class, 'bbGrid-row')]">
<xsl:map>
<xsl:variable name="headers" select="./ancestor::table[1]/thead/tr/th"/>
<xsl:for-each select="td[not(contains(span/@class, 'icon-plus'))]">
<xsl:variable name="index" select="count(preceding-sibling::td) + 1"/>
<xsl:map-entry key="string($headers[$index])" select="if (./div[2]) then normalize-space(./div[2]) else normalize-space(.)"/>
</xsl:for-each>
<xsl:if test="following-sibling::tr[1][td/div/table]">
<xsl:map-entry key="'Dispositions'">
<xsl:variable name="dispositions" as="map(*)*">
<xsl:apply-templates select="following-sibling::tr[1]"/>
</xsl:variable>
<xsl:sequence select="array{$dispositions}"/>
</xsl:map-entry>
</xsl:if>
</xsl:map>
</xsl:template>
<xsl:template match="td/div[1][contains(@id, 'active')]">
<xsl:map-entry key="'status'" select="string(@id)"/>
</xsl:template>
</xsl:stylesheet>
We're fairly happy with the approach so far. I mainly wanted to ask for feedback on the XSLT, as I am returning to the language after a long absence. My premium is on maintainability and comprehensibility for other programmers.
I've tried a few different approaches to see how I like them. For instance, the <!--Rightmost tables and Complainant Information--> template is very compact and produces the desired output but is maybe not very comprehensible.
Comments on how to improve the approach of applying XSLT 3.0 to real-world HTML from Python are also welcome.
