5
\$\begingroup\$

I'm trying to find out why my web-scraping code with BeautifulSoup (BS) is slower than my code without BS. I would think that BS code would be faster than the other code - so, maybe I'm doing something wrong?

With BS

from bs4 import BeautifulSoup
import requests
import pandas as pd
import time

# start timer
start = time.time()

# control parameters: 
# dates
dateFrom = '2016-01-01'
dateTo = '2016-07-31'

# url
url = 'http://utilitytool.casc.eu/CascUtilityWebService.asmx/GetNetPositionDataForAPeriod?dateFrom=' + dateFrom + '&dateTo=' + dateTo
# /control parameters:
page = requests.get(url)

soup = BeautifulSoup(page.content)

# Extract data from soup
calendardate = [i.text for i in soup.findAll('calendardate')]
calendarhour = [i.text for i in soup.findAll('calendarhour')]
be = [i.text for i in soup.findAll('be')]
nl = [i.text for i in soup.findAll('nl')]
deat = [i.text for i in soup.findAll('deat')]
fr = [i.text for i in soup.findAll('fr')]

# lose the useless string in date list
calendardate = [w.replace('T00:00:00', '') for w in calendardate ]

# convert hour column to int
calendarhour = [int(i) for i in calendarhour]

# Python operates with hours: 0-23 and not with 1-24
datetime = [x-1 if x - 1 > 9 else '0' + str(x-1) for x in calendarhour]

# create DateTime list
datetime = ["%s %s:00:00" % t for t in zip(calendardate, datetime)]

# Create Pandas Df
df = pd.DataFrame({
        'datetime': datetime,
        'be': be,
        'nl': nl,
        'deat': deat,
        'fr': fr
    },
    columns = ['datetime', 'be', 'nl', 'deat', 'fr'])


# end time
end = time.time()
print('\nTime elapsed', round(end - start, 3), 's')

Without BS

import pandas as pd
from datetime import datetime
import time
import urllib.request
import re

# start timer
start = time.time()

# control parameters: 
# dates
dateFrom = '2016-01-01'
dateTo = '2016-07-31'

# url
url = 'http://utilitytool.casc.eu/CascUtilityWebService.asmx/GetNetPositionDataForAPeriod?dateFrom=' + dateFrom + '&dateTo=' + dateTo
# /control parameters: 

# request the url
req = urllib.request.Request(url)
resp = urllib.request.urlopen(req)
respData = resp.read()

# find all paragraphs
# all_paragraphs = soup.find_all('netpositiondata')
' must add lists of different years'
# save data between paragraphs
CalendarDate = re.findall(r'<CalendarDate>(.*?)</CalendarDate>', str(respData))
CalendarHour = re.findall(r'<CalendarHour>(.*?)</CalendarHour>', str(respData))
BE = re.findall(r'<BE>(.*?)</BE>', str(respData))
NL = re.findall(r'<NL>(.*?)</NL>', str(respData))
DEAT = re.findall(r'<DEAT>(.*?)</DEAT>', str(respData))
FR = re.findall(r'<FR>(.*?)</FR>', str(respData))

# lose the useless string in date list
CalendarDate = [w.replace('T00:00:00', '') for w in CalendarDate]
# convert hour column to int
CalendarHour = [int(i) for i in CalendarHour]
# convert strings to floats
BE = [float(i) for i in BE]
NL = [float(i) for i in NL]
DEAT = [float(i) for i in DEAT]
FR = [float(i) for i in FR]

DateTime = [x-1 if x - 1 > 9 else '0' + str(x-1) for x in CalendarHour]

# create DateTime list
DateTime = ["%s %s:00:00" % t for t in zip(CalendarDate, DateTime)]

# create pandas df from lists
df = pd.DataFrame()
df['CalendarDate'] = CalendarDate
df['CalendarHour'] = CalendarHour
df['DateTime'] = DateTime
df['BE'] = BE
df['NL'] = NL
df['DEAT'] = DEAT
df['FR'] = FR

# end time
end = time.time()
print('\nTime elapsed', round(end - start, 3), 's')

# convert DateTime column from string to DateTime format
df['DateTime'] = [datetime.strptime(i, '%Y-%m-%d %H:%M:%S') for i in df['DateTime']]

The document looks like this:

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfNetPositionData xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://tempuri.org/">
  <NetPositionData>
    <CalendarDate>2016-01-01T00:00:00</CalendarDate>
    <CalendarHour>1</CalendarHour>
    <BE>90.4000000000</BE>
    <NL>-779.8000000000</NL>
    <DEAT>4874.6000000000</DEAT>
    <FR>-4185.2000000000</FR>
  </NetPositionData>
  <NetPositionData>
    <CalendarDate>2016-01-01T00:00:00</CalendarDate>
    <CalendarHour>2</CalendarHour>
    <BE>257.0000000000</BE>
    <NL>-1166.6000000000</NL>
    <DEAT>4347.4000000000</DEAT>
    <FR>-3437.8000000000</FR>
  </NetPositionData>
  <!-- … and so on for ~5000 entries -->
</ArrayOfNetPositionData>
\$\endgroup\$
2
  • 1
    \$\begingroup\$ Since you have XML data and not HTML, why didn't you use the lxml parser with beautifulsoup? \$\endgroup\$ Dec 20 '16 at 8:14
  • \$\begingroup\$ tried using that also; same result \$\endgroup\$
    – nick
    Dec 20 '16 at 10:06
3
\$\begingroup\$

Regardless of the parsing method, and considering that you are using pandas dataframes in the end, I would simplify some of its usage.

pandas has its own objects dealing with dates and times, and they are pretty smart: pd.Timestamp and pd.Timedelta. You can get your whole datetime manipulation done using:

calendardate = (<method using either bs or re>)
calendarhour = (<method using either bs or re>)

calendartimes = [
    pd.Timestamp(date) + pd.Timedelta('{}h'.format(int(time)-1))
    for date, time in zip(calendardate, calendarhour)
]

Also, these calendartimes may end up being more than a column of data, you can convert them to an index using pd.to_datetime:

df = pd.DataFrame({
    'BE': be,
    'NL': nl,
    'DEAT': deat,
    'FR': fr
}, index=pd.to_datetime(calendartimes))

(Oh and, when using a dictionary to feed into pd.DataFrame, you don't need to specify the columns, the keys of the dictionary will be used).


Now there are a few other things to consider when looking at the code.

First off, get consistent. Between the two versions, naming conventions are not the same so it is harder to talk about the variables at play. The first code is way better at this as it follows PEP 8 more closely.

Consistency would also be to use the same libraries to retrieve the required page. And, as far as timing is concerned, retrieving the page should not be timed as it highly depends on your bandwidth at the moment of download. Including that time in your measurements of the parsing bits is not fair.

Talking about timing, you should put your code into functions so it is easier to reuse, test and time. Wrapping your tests under if __name__ == '__main__': would also be good practice:

import re
from bs4 import BeautifulSoup
import requests
import pandas as pd


DATA_URL = 'http://utilitytool.casc.eu/' \
           'CascUtilityWebService.asmx/GetNetPositionDataForAPeriod'


def download_data(from_date, to_date, url=DATA_URL):
    date_format = '%Y-%m-%d'
    parameters = {
        'dateFrom': from_date.strftime(date_format),
        'dateTo': to_date.strftime(date_format),
    }

    page = requests.get(url, params=parameters)
    page.raise_for_status()
    return page.content


def parse_page_using_bs(content):
    soup = BeautifulSoup(content, 'xml')

    # Extract data from soup
    calendardate = (i.text for i in soup.findAll('calendardate'))
    calendarhour = (i.text for i in soup.findAll('calendarhour'))
    be = (i.text for i in soup.findAll('be'))
    nl = (i.text for i in soup.findAll('nl'))
    deat = (i.text for i in soup.findAll('deat'))
    fr = (i.text for i in soup.findAll('fr'))

    # Merge date and hours in a single datetime
    calendartimes = [
        pd.Timestamp(date) + pd.Timedelta('{}h'.format(int(time)-1))
        for date, time in zip(calendardate, calendarhour)
    ]

    # Create Pandas Df
    return pd.DataFrame({
            'BE': be,
            'NL': nl,
            'DEAT': deat,
            'FR': fr
    }, index=pd.to_datetime(calendartimes))


def parse_page_using_re(content):
    data = str(content)
    calendardate = re.findall(r'<CalendarDate>(.*?)</CalendarDate>', data)
    calendarhour = re.findall(r'<CalendarHour>(.*?)</CalendarHour>', data)
    be = re.findall(r'<BE>(.*?)</BE>', data)
    nl = re.findall(r'<NL>(.*?)</NL>', data)
    deat = re.findall(r'<DEAT>(.*?)</DEAT>', data)
    fr = re.findall(r'<FR>(.*?)</FR>', data)

    # Merge date and hours in a single datetime
    calendartimes = [
        pd.Timestamp(date) + pd.Timedelta('{}h'.format(int(time)-1))
        for date, time in zip(calendardate, calendarhour)
    ]

    # convert strings to floats
    be = [float(i) for i in be]
    nl = [float(i) for i in nl]
    deat = [float(i) for i in deat]
    fr = [float(i) for i in fr]

    # create pandas df from lists
    df = pd.DataFrame()
    df['BE'] = be
    df['NL'] = nl
    df['DEAT'] = deat
    df['FR'] = fr
    df.index = pd.to_datetime(calendartimes)
    return df


if __name__ == '__main__':
    import time
    import datetime

    page = download_data(datetime.date(2016, 1, 1), datetime.date(2016, 7, 31))

    start = time.time()
    parse_page_using_bs(page)
    end = time.time()
    print('\nTime elapsed for Beautifulsoup', round(end - start, 3), 's')

    start = time.time()
    parse_page_using_re(page)
    end = time.time()
    print('\nTime elapsed for re', round(end - start, 3), 's')

This also enables you to use better timing tools, like timeit:

if __name__ == '__main__':
    from timeit import timeit
    from datetime import date

    page = download_data(date(2016, 1, 1), date(2016, 7, 31))
    for function in ['parse_page_using_bs', 'parse_page_using_re']:
        setup = 'from __main__ import {} as parse, page'.format(function)
        print(function, ':', timeit('parse(page)', setup=setup, number=10))

Now, as you’re asking about performance, building intermediate lists using BeautifulSoup.find_all or re.findall only for transformation purposes is not the best thing you can do. Better use generators. It's easier with re using re.finditer as almost a drop-in replacement for re.findall. But with BeautifulSoup, you'll need a bit of manual labor:

def find_iter(soup, tag):
    content = soup.find(tag)
    while content is not None:
        yield content
        content = content.find_next(tag)

This will allow you the following rewrite:

import re
from bs4 import BeautifulSoup
import requests
import pandas as pd


DATA_URL = 'http://utilitytool.casc.eu/' \
           'CascUtilityWebService.asmx/GetNetPositionDataForAPeriod'


def download_data(from_date, to_date, url=DATA_URL):
    date_format = '%Y-%m-%d'
    parameters = {
        'dateFrom': from_date.strftime(date_format),
        'dateTo': to_date.strftime(date_format),
    }

    page = requests.get(url, params=parameters)
    page.raise_for_status()
    return page.content


def find_iter(soup, tag):
    content = soup.find(tag)
    while content is not None:
        yield content
        content = content.find_next(tag)


def parse_page_using_bs(content):
    soup = BeautifulSoup(content, 'xml')

    # Extract data from soup
    calendardate = (i.text for i in find_iter(soup, 'calendardate'))
    calendarhour = (i.text for i in find_iter(soup, 'calendarhour'))
    be = (i.text for i in find_iter(soup, 'be'))
    nl = (i.text for i in find_iter(soup, 'nl'))
    deat = (i.text for i in find_iter(soup, 'deat'))
    fr = (i.text for i in find_iter(soup, 'fr'))

    # Merge date and hours in a single datetime
    calendartimes = [
        pd.Timestamp(date) + pd.Timedelta('{}h'.format(int(time)-1))
        for date, time in zip(calendardate, calendarhour)
    ]

    # Create Pandas Df
    return pd.DataFrame({
            'BE': [float(x) for x in be],
            'NL': [float(x) for x in nl],
            'DEAT': [float(x) for x in deat],
            'FR': [float(x) for x in fr],
    }, index=pd.to_datetime(calendartimes))


def parse_page_using_re(content):
    data = str(content)
    calendardate = re.finditer(r'<CalendarDate>(.*?)</CalendarDate>', data)
    calendarhour = re.finditer(r'<CalendarHour>(.*?)</CalendarHour>', data)
    be = re.finditer(r'<BE>(.*?)</BE>', data)
    nl = re.finditer(r'<NL>(.*?)</NL>', data)
    deat = re.finditer(r'<DEAT>(.*?)</DEAT>', data)
    fr = re.finditer(r'<FR>(.*?)</FR>', data)

    # Merge date and hours in a single datetime
    calendartimes = [
        pd.Timestamp(date.group(1)) +
        pd.Timedelta('{}h'.format(int(time.group(1))-1))
        for date, time in zip(calendardate, calendarhour)
    ]

    # create pandas df from lists
    df = pd.DataFrame()
    df['BE'] = [float(i.group(1)) for i in be]
    df['NL'] = [float(i.group(1)) for i in nl]
    df['DEAT'] = [float(i.group(1)) for i in deat]
    df['FR'] = [float(i.group(1)) for i in fr]
    df.index = pd.to_datetime(calendartimes)
    return df


if __name__ == '__main__':
    from timeit import timeit
    from datetime import date

    page = download_data(date(2016, 1, 1), date(2016, 7, 31))
    for function in ['parse_page_using_bs', 'parse_page_using_re']:
        setup = 'from __main__ import {} as parse, page'.format(function)
        print(function, ':', timeit('parse(page)', setup=setup, number=10))

The last improvement that can be made is about how you parse the data using BeautifulSoup. Instead of asking it to skim through each and every tags it holds, you can take advantage of what you know about your data structure, namely, each tag of interest are within a <NetPositionData> tag. So you can only ask for these, iterating around siblings (which is more efficient than "any" tag) and retrieving data from there. With that, you can easily build a list of tuples from your data that pandas can convert to a DataFrame just as easily than a dictionary:

def find_iter(soup, tag):
    content = soup.find(tag)
    while content is not None:
        yield content
        content = content.find_next_sibling()


def parse_page_using_bs(content):
    soup = BeautifulSoup(content, 'xml')

    # Extract data from soup
    net_positions = [
        (pd.Timestamp(tag.calendardate.text) +
         pd.Timedelta('{}h'.format(int(tag.calendarhour.text)-1)),
         float(tag.be.text),
         float(tag.nl.text),
         float(tag.deat.text),
         float(tag.fr.text))
        for tag in find_iter(soup, 'netpositiondata')
    ]

    # Create Pandas Df
    df = pd.DataFrame(
        net_positions,
        columns=['DateTime', 'BE', 'NL', 'DEAT', 'FR'])

    # Optionally turn DateTime into an index
    df.index = pd.to_datetime(df.DateTime)
    return df
\$\endgroup\$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.