9
\$\begingroup\$

I am working on a simple web crawler that returns the start time and date for an event listed on a webpage. The webpage can be in two different formats and there are multiple other dates listed on the page. The part of the webpage I am scanning looks like this:

...
<h4>Time</h4>
    <p> 
        Starts: <time itemprop="startDate" datetime="2017-03-13T17:30:00ADT">Monday March 13, 2017 - 05:30 PM</time>
        <br />
        Ends: <time itemprop="endDate" datetime="2017-03-13T19:00:00ADT">Monday March 13, 2017 - 07:00 PM</time>
    </p>

<h4>Additional Dates/Times</h4>
    <p>
        Starts: <time>Monday January 30, 2017 - 05:30 PM</time>
        <br />
        Ends: <time>Monday January 30, 2017 - 07:00 PM</time>
    </p><div class="eventitemrepetition eventItemRepetition"><p>

<p><h4>Location</h4>
...

or like this:

...
<h4>Time</h4>
<p>
    <time itemprop="startDate" datetime="2017-03-17T13:30:00ADT">Friday March 17, 2017 - 01:30 PM</time>
</p>

<p><h4>Location</h4>
....

After using BeautifulSoup to find the links I am interested in I pass the link to these methods which find the text between 'Time' and 'Location'. Next it searches for a tag in that code block '">" which I know only appears before the date text and scans until the next closing tag, returning a date string.

MAX_CHARS = 140

def get_date(link):
    date_text = ""
    event_html = urllib2.urlopen(link['href']).read()
    start = find_date_location(event_html)
    # after the datetime closing tag the date begins until another tag opens
    for x in range(start, start + MAX_CHARS):
        if event_html[x] == '<':
            break
        date_text += event_html[x]

    return date_text


def find_date_location(html):
    date_starts = html.find('<h4>Time</h4>')
    date_ends = html.find('<h4>Location</h4>')
    for x in range(date_starts, date_ends):
        if html[x] + html[x+1] == '">':
            return x+2

    raise ValueError('Date not found in HTML within time range')

This would return "Monday March 13, 2017 - 05:30 PM" in the first case and "Friday March 17, 2017 - 01:30 PM" in the second case. My method seems really hacky. Any tips on how I could do this better?

\$\endgroup\$
5
\$\begingroup\$

Your current approach is very fragile and might easily break if the markup changes even slightly - imagine, for example, opening and closing h4 being on separate lines while still being a valid HTML element.

I would use a proper HTML parser like BeautifulSoup instead (you mentioned you've already tried it). In order to locate the start and end dates, we may use the itemprop attribute:

from bs4 import BeautifulSoup


def get_even_date_range(html):
    soup = BeautifulSoup(html, 'html.parser')

    start_date = soup.find("time", itemprop="startDate")
    end_date = soup.find("time", itemprop="endDate")

    return (start_date.get_text() if start_date else None,
            end_date.get_text() if end_date else None)

Here, the get_even_date_range() function would return a tuple with start and end dates as items. It would return None if a date is not found. For the first sample input HTML, it would return:

('Monday March 13, 2017 - 05:30 PM', 'Monday March 13, 2017 - 07:00 PM')

And, for the second:

('Friday March 17, 2017 - 01:30 PM', None)

You can then go further and convert the date strings to datetimes using the datetime.strptime() and the %A %B %d, %Y - %H:%M %p format:

from datetime import datetime
from bs4 import BeautifulSoup


DATE_FORMAT = "%A %B %d, %Y - %H:%M %p"


def get_date(date_element):
    return datetime.strptime(date_element.get_text(), DATE_FORMAT) if date_element else None


def get_even_date_range(html):
    soup = BeautifulSoup(html, 'html.parser')

    start_date = soup.find("time", itemprop="startDate")
    end_date = soup.find("time", itemprop="endDate")

    return get_date(start_date), get_date(end_date)

Note that I've also moved the repetitive date retrieval logic to a separate reusable get_date() function.

\$\endgroup\$
  • \$\begingroup\$ Awesome, I thought BeautifulSoup must have some way to do this. I just couldn't figure out how. \$\endgroup\$ – BruceM Mar 14 '17 at 23:59
  • 1
    \$\begingroup\$ @BruceM and this is just one way to get to these elements. BeautifulSoup has one of the best APIs I've ever seen in Python libraries, you can get to your desired elements in so many different ways. Go over the documentation and the BeautifulSoup questions on SO to get better understanding about how the library is used. Thanks! \$\endgroup\$ – alecxe Mar 15 '17 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.