This is an excerpt of my html code

<div class="row">
    <div class="col-md-3">
        <div class="left_menu">
            <!-- <h2></h2> -->
            <ul class="w_list">
                <li><a href="">Search</a></li>
    <div class="col-md-9">
        <div class="page_title">Search</div>
        <table class="table table-bordered table-responsive">
                <tr class="blue">
                    <td colspan="2"><strong>Product Search</strong></td>
                    <td>Product Name</td>
                    <td><a class="product_link" href="#" onclick="search_product(2,53)">Galaxy Note 8</a></td>
                    <td><a class="product_link" href="#" onclick="search_product(3,53)">Redmi 4</a></td>
                    <td><a class="product_link" href="#" onclick="search_product(4,42)">Washing Machine 44L</a></td>
                    <td><a class="product_link" href="#" onclick="search_product(5,53)">Air Conditioner</a></td>
                    <td><a class="product_link" href="#" onclick="search_product(6,42)">Air Cooler</a></td>

Below is a method from my Django application

def get():
    arr = []
    soup = bs4.BeautifulSoup(open(os.path.join(settings.BASE_DIR, 'sample.html')), 'html.parser')
    for a in soup.find_all(lambda tag: tag.name == "a" and "class" in tag.attrs and "product_link" in tag['class'] and len(tag['class']) > 0):
            if re.match('search_product', a['onclick']):
                obj = {"name": a.get_text(strp=True), "next": a['onclick']}

The actual file has around 300K of those repeated <tr> tags, each corresponding to a product. The code above takes about 8 minutes to extract the required data on my high-end laptop. How can I improve this code so that it can run faster? Are there any other Python libraries which can do the job faster?

  • 3
    \$\begingroup\$ Please provide more context for your question. What does this code accomplish? That should be stated in the title. Your method doesn’t return anything — is that real code? See How to Ask, and edit your question. \$\endgroup\$ – 200_success Sep 23 '17 at 15:54

There are multiple things to do to make the HTML parsing faster:

  • switch to lxml from html.parser (lxml is the fastest - requires lxml to be installed, of course):

    with open(os.path.join(settings.BASE_DIR, 'sample.html')) as f:
        soup = bs4.BeautifulSoup(f, 'lxml')
  • use more concise, readable and faster CSS selectors to filter out the desired elements:

    for a in soup.select("a.product_link"):
  • you can also use a SoupStrainer to limit the parsing scope:

    parse_only = bs4.SoupStrainer("a", class_="product_link", onclick=True)
    soup = bs4.BeautifulSoup(f, 'lxml', parse_only=parse_only)
  • and, since we've limited the parsing scope to the product links - see if you still need that regular expression onclick check at all

Note that you still have a lot of overhead there because of the synchronous nature of your execution logic - you are currently processing products one by one sequentially. Look into asynchronous solutions using, for instance, Scrapy, grequests, asyncio etc.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.