This is an excerpt of my html code

<div class="row">
    <div class="col-md-3">
        <div class="left_menu">
            <!-- <h2></h2> -->
            <ul class="w_list">
                <li><a href="">Search</a></li>
    <div class="col-md-9">
        <div class="page_title">Search</div>
        <table class="table table-bordered table-responsive">
                <tr class="blue">
                    <td colspan="2"><strong>Product Search</strong></td>
                    <td>Product Name</td>
                    <td><a class="product_link" href="#" onclick="search_product(2,53)">Galaxy Note 8</a></td>
                    <td><a class="product_link" href="#" onclick="search_product(3,53)">Redmi 4</a></td>
                    <td><a class="product_link" href="#" onclick="search_product(4,42)">Washing Machine 44L</a></td>
                    <td><a class="product_link" href="#" onclick="search_product(5,53)">Air Conditioner</a></td>
                    <td><a class="product_link" href="#" onclick="search_product(6,42)">Air Cooler</a></td>

Below is a method from my Django application

def get():
    arr = []
    soup = bs4.BeautifulSoup(open(os.path.join(settings.BASE_DIR, 'sample.html')), 'html.parser')
    for a in soup.find_all(lambda tag: tag.name == "a" and "class" in tag.attrs and "product_link" in tag['class'] and len(tag['class']) > 0):
            if re.match('search_product', a['onclick']):
                obj = {"name": a.get_text(strp=True), "next": a['onclick']}

The actual file has around 300K of those repeated <tr> tags, each corresponding to a product. The code above takes about 8 minutes to extract the required data on my high-end laptop. How can I improve this code so that it can run faster? Are there any other Python libraries which can do the job faster?


closed as unclear what you're asking by Jamal Sep 23 '17 at 17:23

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    \$\begingroup\$ Please provide more context for your question. What does this code accomplish? That should be stated in the title. Your method doesn’t return anything — is that real code? See How to Ask, and edit your question. \$\endgroup\$ – 200_success Sep 23 '17 at 15:54

There are multiple things to do to make the HTML parsing faster:

  • switch to lxml from html.parser (lxml is the fastest - requires lxml to be installed, of course):

    with open(os.path.join(settings.BASE_DIR, 'sample.html')) as f:
        soup = bs4.BeautifulSoup(f, 'lxml')
  • use more concise, readable and faster CSS selectors to filter out the desired elements:

    for a in soup.select("a.product_link"):
  • you can also use a SoupStrainer to limit the parsing scope:

    parse_only = bs4.SoupStrainer("a", class_="product_link", onclick=True)
    soup = bs4.BeautifulSoup(f, 'lxml', parse_only=parse_only)
  • and, since we've limited the parsing scope to the product links - see if you still need that regular expression onclick check at all

Note that you still have a lot of overhead there because of the synchronous nature of your execution logic - you are currently processing products one by one sequentially. Look into asynchronous solutions using, for instance, Scrapy, grequests, asyncio etc.


Not the answer you're looking for? Browse other questions tagged or ask your own question.