1
\$\begingroup\$

I want the function to take as an input json file containing html_body with its corresponding url and return list of tuples containing headers and their corresponding url (so could be tuple with one item as a list of the headers).

So far I have been able to do so, but I'm missing some headers and more over, I'm sure that this code can be improved.

Commented out possible extensions for <h2> headers extraction, and a different regex pattern I tried.

A taste of the source:

[["<body class=\" catalog-category-view categorypath-sale-html category-sale\">\n<script type=\"text/javascript\">\n//<![CDATA[\nif (typeof(Varien.searchForm) !== 'undefined') {\n    Varien.searchForm.prototype._selectAutocompleteItem = function(element) {\n        var link = element.down();\n        if (link && link.tagName == 'A') {\n            setLocation(link.href);\n        } else {\n            if (element.title){\n                this.field.value = element.title;\n            }\n            this.form.submit();\n        }\n    };\n    Varien.searchForm.prototype.initAutocomplete = function(url, destinationElement) {\n        new Ajax.Autocompleter(\n            this.field,\n            destinationElement,\n            url,\n            {\n                paramName: this.field.name,\n                method: 'get',\n                minChars: 2,\n                frequency: .3,\n                updateElement: this._selectAutocompleteItem.bind(this),\n                onShow : function(element, update) {\n                    if(!update.style.position || update.style.position=='absolute') {\n                        update.style.position = 'absolute';\n                        Position.clone(element, update, {\n                            setHeight: false,\n                            offsetTop: element.offsetHeight\n                        });\n                    }\n                    Effect.Appear(update,{duration:0});\n                }\n\n            }\n        );\n    };\n    Autocompleter.Base.prototype.markPrevious = function() {\n        if (this.index > 0) {\n            this.index--;\n        } else {\n            this.index = this.entryCount - 1;\n        }\n        var entry = this.getEntry(this.index);\n        if (entry.select('a').length === 0) {\n            this.markPrevious(); // Ignore items that don't have link\n        }\n    };\n    Autocompleter.Base.prototype.markNext = function() {\n        if (this.index < this.entryCount - 1) {\n

Code:

from pandas import DataFrame, Series
from tqdm import tqdm
import html5lib
import re
from bs4 import BeautifulSoup

def get_headers_from_json(local_path):
"""
The function takes a json file with html_body and returns a list of headers.
It parses the headers, based on tags starting with 'h'.
It also parses urls and return them in a tuple: (headers, url)
"""
data = json.loads(open(local_path).read())
pattern = "h1" # | h2 | h3"   
# pattern = re.compile("^(?!.*(href)).*^h", re.IGNORECASE)
headers_urls = []
all_headers = []
for x in tqdm(data):
    soup = BeautifulSoup(x[0], 'html.parser')
    html_headers = soup.find_all(re.compile(pattern))
    all_headers.append(html_headers)
    url = x[1]
    tuple_title_url = all_headers, url
    headers_urls.append(tuple_title_url)
return headers_urls

PS - I checked the following links but couldn't find a great solution: HTML tag encoder HTML parsing algorithm for extracting <a> tags Scraping HTML using Beautiful Soup

Improve this question
\$\endgroup\$

1 Answer 1

Reset to default
1
\$\begingroup\$

Currently, you are matching a single h1 tag only, but, from what I understand, you want to catch all the "header" elements. There are multiple ways to do that.

One is with a regular expression:

headers = soup.find_all(re.compile(r"^h"))

Or, with a lambda function:

headers = soup.find_all(lambda tag: tag and tag.name.startswith("h"))

Or, with a list of explicitly specified tags:

headers = soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5'])

Note that in order to get the header texts, you would use .get_text() method:

[header.get_text() for header in headers]

Other notes:

  • if every item returned from tqdm(data) consists of two items, you can unpack them in a for loop:

    for html, url in tqdm(data):

    Then, you can use html instead of x[0] and url instead of x[1].

  • if you are going to go with a regular expression approach to match the headers, you should pre-compile the expression - define it as a constant:

    HEADER_PATTERN = re.compile(r"^h")

  • you can use json.load() to read JSON from a file directly, replace:

    data = json.loads(open(local_path).read())

    with:

    with open(local_path) as input_file:
    data = json.load(input_file)

If performance matters:

  • you can use a faster JSON parser: ujson instead of json
  • use lxml instead of html.parser as a BeautifulSoup underlying parser
  • you can parse header elements only with SoupStrainer ignoring all other elements - this should boost HTML parsing performance as well

Improved code (with some of the suggestions applied):

import re

from bs4 import BeautifulSoup, SoupStrainer
from tqdm import tqdm
import ujson


HEADER_PATTERN = re.compile(r"^h")


def get_headers_from_json(local_path):
    """
    The function takes a json file with html_body and returns a list of headers.
    It parses the headers, based on tags starting with 'h'.
    It also parses urls and return them in a tuple: (headers, url)
    """
    with open(local_path) as input_file:
        data = ujson.load(input_file)

    parse_only = SoupStrainer(HEADER_PATTERN)
    return [
        ([header.get_text() for header in BeautifulSoup(html, 'lxml', parse_only=parse_only)(HEADER_PATTERN)], url)
        for html, url in tqdm(data)
    ]
Improve this answer
\$\endgroup\$
3
  • \$\begingroup\$ Thank you @alecxe , a couple of clarifications: 1. Wouldn't ur pattern get "href" as well? I suggest instead: r"^h[0-9]$" 2. My code works fine without ".get_text" and I noticed that your's doesn't include it as well. 3. What are the benefits of turning the one line to read the JSON file into 2 lines? 4. Reg UJSON see: link Cheers, \$\endgroup\$
    – oba2311
    Jun 26, 2017 at 15:04
  • \$\begingroup\$ @oba2311 Thanks for the comment. 1. Yours regex is a bit more explicit, yeah, but href would not be matched since the expression is applied to tag names only. 2. .get_text() would only be needed if you need the text of the nodes (excluding the opening and closing tags). \$\endgroup\$
    – alecxe
    Jun 26, 2017 at 15:06
  • \$\begingroup\$ @oba2311 ah, did not see your edit. 3. not a big deal, it is simply a bit clearer. Plus, we are following the best-practices and safely closing the file with a context manager. 4. I would suggest to measure performance of different json parsers on your specific use-case - your machine, your json input etc. Thanks. \$\endgroup\$
    – alecxe
    Jun 26, 2017 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.