Scan a webpage to find the start time and date for an event

Question

I am working on a simple web crawler that returns the start time and date for an event listed on a webpage. The webpage can be in two different formats and there are multiple other dates listed on the page. The part of the webpage I am scanning looks like this:

...
<h4>Time</h4>
    <p> 
        Starts: <time itemprop="startDate" datetime="2017-03-13T17:30:00ADT">Monday March 13, 2017 - 05:30 PM</time>
        <br />
        Ends: <time itemprop="endDate" datetime="2017-03-13T19:00:00ADT">Monday March 13, 2017 - 07:00 PM</time>
    </p>

<h4>Additional Dates/Times</h4>
    <p>
        Starts: <time>Monday January 30, 2017 - 05:30 PM</time>
        <br />
        Ends: <time>Monday January 30, 2017 - 07:00 PM</time>
    </p><div class="eventitemrepetition eventItemRepetition"><p>

<p><h4>Location</h4>
...

or like this:

...
<h4>Time</h4>
<p>
    <time itemprop="startDate" datetime="2017-03-17T13:30:00ADT">Friday March 17, 2017 - 01:30 PM</time>
</p>

<p><h4>Location</h4>
....

After using BeautifulSoup to find the links I am interested in I pass the link to these methods which find the text between 'Time' and 'Location'. Next it searches for a tag in that code block '">" which I know only appears before the date text and scans until the next closing tag, returning a date string.

MAX_CHARS = 140

def get_date(link):
    date_text = ""
    event_html = urllib2.urlopen(link['href']).read()
    start = find_date_location(event_html)
    # after the datetime closing tag the date begins until another tag opens
    for x in range(start, start + MAX_CHARS):
        if event_html[x] == '<':
            break
        date_text += event_html[x]

    return date_text


def find_date_location(html):
    date_starts = html.find('<h4>Time</h4>')
    date_ends = html.find('<h4>Location</h4>')
    for x in range(date_starts, date_ends):
        if html[x] + html[x+1] == '">':
            return x+2

    raise ValueError('Date not found in HTML within time range')

This would return "Monday March 13, 2017 - 05:30 PM" in the first case and "Friday March 17, 2017 - 01:30 PM" in the second case. My method seems really hacky. Any tips on how I could do this better?

Answer 1

Your current approach is very fragile and might easily break if the markup changes even slightly - imagine, for example, opening and closing h4 being on separate lines while still being a valid HTML element.

I would use a proper HTML parser like BeautifulSoup instead (you mentioned you've already tried it). In order to locate the start and end dates, we may use the itemprop attribute:

from bs4 import BeautifulSoup


def get_even_date_range(html):
    soup = BeautifulSoup(html, 'html.parser')

    start_date = soup.find("time", itemprop="startDate")
    end_date = soup.find("time", itemprop="endDate")

    return (start_date.get_text() if start_date else None,
            end_date.get_text() if end_date else None)

Here, the get_even_date_range() function would return a tuple with start and end dates as items. It would return None if a date is not found. For the first sample input HTML, it would return:

('Monday March 13, 2017 - 05:30 PM', 'Monday March 13, 2017 - 07:00 PM')

And, for the second:

('Friday March 17, 2017 - 01:30 PM', None)

---

You can then go further and convert the date strings to datetimes using the datetime.strptime() and the %A %B %d, %Y - %H:%M %p format:

from datetime import datetime
from bs4 import BeautifulSoup


DATE_FORMAT = "%A %B %d, %Y - %H:%M %p"


def get_date(date_element):
    return datetime.strptime(date_element.get_text(), DATE_FORMAT) if date_element else None


def get_even_date_range(html):
    soup = BeautifulSoup(html, 'html.parser')

    start_date = soup.find("time", itemprop="startDate")
    end_date = soup.find("time", itemprop="endDate")

    return get_date(start_date), get_date(end_date)

Note that I've also moved the repetitive date retrieval logic to a separate reusable get_date() function.