In the last couple of days I've tried to play around with some scraping using XPATHs and Python to consolidate my knowledge (since it's been some time). I did this on a simple website which doesn't provide an API.


Get all the information about every product on crlaurence without using Scrapy (which I know would have a huge impact on the speed but would take too much time to learn at the moment), and save the result in a JSON file.


import json
import re
import requests

from time import sleep

from lxml import html

URL = 'http://www.crlaurence.com'
SITEMAP_URL = f'{URL}/apps/contentloader/default.aspx?content=www.crlaurence.com/adv/sitemap/sitemap_us.html'

SITEMAP_URLS_XPATH = '//div[@class="sitemapContent"]//a/@href'
CATEGORIES_XPATH = '//div[@class="divCell"]/a/@href'

PRODUCT_CATALOG_NUMBER_XPATH = '//div[@id="ctl00_ContentPlaceHolder1_lblMoreDetails"]//b/text()'
PRODUCT_TITLE_XPATH = '//span[@class="lblProductDesc"]//text()'
PRODUCT_SPECIFICATION_XPATH = '//div[@id="ctl00_ContentPlaceHolder1_lblBulletList"]//text()'
PRODUCT_DESCRIPTION_XPATH = '//div[@id="ctl00_ContentPlaceHolder1_lblProdDetails"]/h1//text()'
PRODUCT_ADDITIONAL_INFO_XPATH = '//div[@id="ctl00_ContentPlaceHolder1_lblMoreDescription"]/text()'
PRODUCT_IMPORTANT_NOTES_XPATH = '//div[@id="ctl00_ContentPlaceHolder1_lblNotes"]//text()'
PRODUCT_OTHER_PRODUCTS_XPATH = '//table[@class="table_color"]/tr'
PRODUCT_DETAILS_XPATH = '//div[@id="ctl00_ContentPlaceHolder1_lblMoreDetails"]/table//tr//text()'
PRODUCT_RELATED_ITEMS_XPATH = '//div[@class="divRelatedTxt"]//text()'

class Product:
    def __init__(self, tree, url):
        self.tree = tree
        self.url = url

    def get_image(self):
        product_id = re.match(r'.*ProductID=(\d+).*', self.url).group(1)
        if product_id:
            return f'{URL}/crlapps/showline/largerimage.aspx?productid={product_id}'
        return 'Could not get image URL.'

    def get_catalog_number(self):
        catalog_number = self.tree.xpath(PRODUCT_CATALOG_NUMBER_XPATH)
        if catalog_number:
            return catalog_number[0]
        return 'Could not get product number.'

    def get_product_name(self):
        product_name = self.tree.xpath(PRODUCT_TITLE_XPATH)
        if product_name:
            return product_name[0]
        return 'Could not get product name.'

    def get_product_specification(self):
        specifications = self.tree.xpath(PRODUCT_SPECIFICATION_XPATH)
        if specifications:
            return specifications
        return 'Could not get product specification.'

    def get_product_description(self):
        description = self.tree.xpath(PRODUCT_DESCRIPTION_XPATH)
        if description:
            return ' '.join(description)
        return 'Could not get product description.'

    def get_small_images(self):
        images = []
        for xpath in PRODUCT_SMALL_IMAGES_XPATH:
            image = self.tree.xpath(xpath)
            if image:
        if images:
            return images
        return 'Could not get product images.'

    def get_small_images_1(self):
        images = []
        for xpath in PRODUCT_SMALL_IMAGES_1_XPATH:
            image = self.tree.xpath(xpath)
            if image:
        if images:
            return images
        return 'Could not get product images.'

    def get_product_additional_info(self):
        info = self.tree.xpath(PRODUCT_ADDITIONAL_INFO_XPATH)
        if info:
            return info[0]
        return 'Could not get product additional info'

    def get_product_important_notes(self):
        notes = self.tree.xpath(PRODUCT_IMPORTANT_NOTES_XPATH)
        if notes:
            return ''.join(notes)
        return 'Could not get product important notes'

    def get_product_details(self):
        details = self.tree.xpath(PRODUCT_DETAILS_XPATH)
        if details:
            result = []
            for k, v in zip(details[::2], details[1::2]):
                    'key': k,
                    'value': v
            return result
        return 'Could not get product details.'

    def get_product_related_items(self):
        rel_items = self.tree.xpath(PRODUCT_RELATED_ITEMS_XPATH)
        if rel_items:
            result = []
            for cn, n in zip(rel_items[::2], rel_items[1::2]):
                    'catalogNumber': cn.strip(),
                    'name': n
            return result
        return 'Could not get product related items.'

    def to_json(self):
        return {
            'catalogNumber': self.get_catalog_number(),
            'Name': self.get_product_name(),
            'specification': self.get_product_specification(),
            'description': self.get_product_description(),
            'logos': self.get_small_images(),
            'logos1': self.get_small_images_1(),
            'additionalInfo': self.get_product_additional_info(),
            'importantNotes': self.get_product_important_notes(),
            'image': self.get_image(),
            'details': self.get_product_details(),
            'relatedItems': self.get_product_related_items()

def retry(url):
        return requests.get(url).text
    except Exception as e:
        print('Retrying product page in {} seconds because: {}'.format(wait, e))
        return retry(url)

def get_sitemap_url():
    raw_html_page = retry(SITEMAP_URL)
    tree = html.fromstring(raw_html_page)
    for sitemap_url in tree.xpath(SITEMAP_URLS_XPATH):
        if 'GroupID' in sitemap_url:
            yield f'{URL}{sitemap_url}'

def get_product_page(url):
    raw_html_page = retry(url)
    tree = html.fromstring(raw_html_page)

    for category_link in tree.xpath(CATEGORIES_XPATH):
        yield from get_product_page(f'{URL}/crlapps/showline/{category_link}')

    for category_link in tree.xpath(CATEGORIES_XPATH):
        link = f'{URL}/crlapps/showline/{category_link}'
        if 'ProductID' in link:
            yield link

def main():
    for sitemap_url in get_sitemap_url():
        for product_link in get_product_page(sitemap_url):
            raw_html_page = retry(product_link)
            tree = html.fromstring(raw_html_page)
            product = Product(tree, product_link).to_json()

            with open('crlaurence.json') as result_json_file:
                existing_data = json.load(result_json_file)


            with open('crlaurence.json', 'w') as result_json_file:
                json.dump(existing_data, result_json_file, indent=4)

if __name__ == '__main__':


  • while I'm aware of PEP8 I'd like the reviews to be mostly focused on improving the speed of the program (although any suggestion is welcome!). The program takes more than 20 hours to run and I'm wondering if there're any ways to improve on that.

You should probably start with a profiler whenever you have unexplained performance problems, that should quickly point you to the functions or areas that take most of the runtime.

That said, I immediately noticed the json.load and json.dump are called more than once. That seems to be at least a first candidate for optimisation. Either start keeping everything in RAM until you're ready to write it to disk, or perhaps write it to disk while you're still scraping (if you need any postprocessing: do it all at once when all data has been collected).

The program also doesn't run without a previously set up JSON output file, that's definitely worth fixing.

Yeah, so after a few seconds I've got like 100kb already. If this runs for 20 hours, the amount of time spent on parsing and dumping this data over and over and over is gonna be quite a bit - and it's going get slower the further on this is running, so while at the start the impact measured might not be that much, it's just gonna increase.

There's no logging, so of course it's also hard to see what the program is doing. Consider putting in maybe the URLs, or a dot every 100 pages, or whatever thing to make it easier to spot progress (or the lack thereof).

Come to think of it, unless you've ruled out that possibility, the scraping might get throttled by the website.

Edit: That reminds me, each page is fetched sequentially and there'll be a lot of waiting for network I/O. The next obvious thing would be to do the fetching concurrently. Have a couple of worker threads / processes and fetch multiple pages at the same time, getting new work items from a shared queue or so, then write results to disk in another thread.

I'm gonna suggest using cProfile here, but just take a look at the reference too.

In particular, try this:

python3 -m cProfile -o profile laurence.py

Wait a moment, then abort the run. Next, inspect the output:

python3 -m pstats profile

Use sort and stats, like sort time and stats 10 or so to get an overview.

And it really looks like the I/O is the biggest reason from the very small sample.

Lastly, parsing via lxml into a full DOM tree might be slow too, plus evaluation of the XPath queries (convenient as they might be). You could always explore a SAX or similar streaming parser. In Python that's for example html.parser - not sure how it looks like from the compatibility perspective of course.

| improve this answer | |
  • 2
    \$\begingroup\$ Nice answer! Worth adding though that lxml is generally considered to be the fastest among existing Python parsers, though the parsing speed depends on multiple factors. \$\endgroup\$ – Grajdeanu Alex. Mar 8 '19 at 8:34
  • \$\begingroup\$ Yeah, I'd focus on the network latencies first, it's just one of the easy things to check in case of a scraper. It also looks like the memory consumption could be reduced, but all of those will probably be secondary. \$\endgroup\$ – ferada Mar 8 '19 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.